What's New in MB-System Version 5.4

The version 5.4.x releases of MB-System includes a number of changes and improvements relative to the earlier version 5 releases. The most significant change is the addition of an autoconf-based build system including a configure script. A detailed listing of bug fixes and other changes is found in the ChangeLog file. The most prominent particulars follow:

• New autoconf based build system.
• The MB-System distribution now comes with a configure script that can be used to build the software without editing any files.
• The configure script generates Makefiles that support installing the executables, libraries, header files, and man pages in locations outside the MB-System source directories (e.g. in /usr/local).
• The configure script generated Makefiles build shared libraries.
• The older "install_makefiles" build system is still included in the distribution, and can be used to build MB-System.

• Support for several new data formats.
• SEA Swathplus interferometric sonar SXP data format is supported as format 222 (MBF_SWPLSSXP).
• 3DatDepth lidar data format is supported as format 231 (MBF_3DDEPTHP).
• Electronic Navigation Ltd. WASSP multibeam sonar data format supported as format 241 (MBF_WASSPENL).
• NGDC MGD77T data format for tab delimited underway marine geophysical data is supported as format 174 (MBF_MGD77TAB).

• New programs:
• MBsvpselect: This program chooses and implements the best available sound speed model for each swath file in a survey according to user specified criteria.

What Was New in MB-System Version 5.3

The version 5.3.x releases of MB-System included a number of changes and improvements relative to the earlier version 5 releases. Two of these changes, a new "fast bathymetry" or fbt file format, and the implementation of file locking, will generate files that older versions of MB-System cannot read. Consequently, we recommend that any MB-System users that work together on common datasets upgrade simultaneously to release 5.3.1887 or higher. The particulars follow:

• Changes to the "fbt" or "fast bathymetry" files in the MB-System processing environment.
• We updated the definition of swath format MBF_LDEOIH (format 71), which is used for the "fast bathymetry" or *.fbt files in MB-System processing environments.
• The previous form of *.fbt files had a serious limitation in that bathymetry from multibeams operated near the seafloor in deep water did not represent the full numerical resolution of those data.
• The updated format allows for depth and distance resolution to 0.001 m even in the deep ocean.
• Old *.fbt files are read transparently, but newly written files will not be readable by older versions of MB-System..

• File locking by data editing tools MBedit, MBeditvia, MBnavedit, MBclean, and the processing program MBprocess.
• A file locking mechanism has been implemented to allow multiple users to work on the same projects without interfering. The intent is for this mechansim to work on heterogeneous networks, which means that the data can be on non-Posix filesystems mounted on multiple computers running different operating systems.
• The file locking is implemented in a crude fashion involving the creation and deletion of "lock files". These are text files with a ".lck" suffix that are created in parallel with the associated "raw" swath files. If a lock file exists, the swath file has been locked by a program and cannot be opened for editing or processing by any other program. When a program completes its work on or with a swath file, it removes the lock by deleting the file.
• The lock files include text indicating the program that generated the lock, the time the lock was created, the purpose (e.g. bathymetry editing), which user generated the lock, and what machine that user was logged into.
• File locking is implemented for MBedit, MBeditviz, MBnavedit, and MBprocess.
• An example of the contents of a lock file is:
       # File /Volumes/MappingAUVOps2011/20110525m1/20110525_202216.mb88
       # Locked by user <caress> on cpu <diebold.shore.mbari.org> at <Fri May 27 09:45:42 2011>
       Locking Program: MBedit
       Locking User: caress
       Locking CPU: diebold.shore.mbari.org
       Locking Time: Fri May 27 09:45:42 2011
       Locking Purpose ID: 2
       Locking Purpose Description: Edit Bathymetry
  
• The primary negative consequence of file locking is the potential creation of orphaned lock files if a locking program crashes or is interrupted. The mbdatalist tool can now be used to detect lock files (-S option) in any of the files references through a datalist, or to remove any lock files (-Y option).

• MBedit and MBnavedit can now operate on datalists.
• Previously, these two editing tools opened one file at a time, specified from a file opening dialog accessed by clicking the <File> button.
• Now, <File> is a pull down menu with two options: <Open> and <File Selection List>. The first brings up the same file opening dialog as before. The second brings up a list dialog showing all of the files available for editing.
• If a user opens a single file for editing, that file will be added to an internal list of files available for editing, and then loaded.
• If a user opens a datalist, then all of the files referenced through the recursive datalist structure will be added to the internal list of files available for editing, and the first file will be loaded.
• The user can, by selecting the <File->File Selection List> menu item, display the internal list of available files. Selecting a file in this list will cause that file to be loaded for editing. If another file was already loaded, it will be closed out gracefully before the new file is loaded.
• The list of available files also indicates which files have been previously edited (so that ".esf" or ".nve" files exist) and which are currently locked by other programs and users.

• Pseudo-Parallel Processing With MBprocess
• Updating the processing of one or more surveys with MBprocess can be time consuming because each swath file must be processed in turn.
• Since the processing of each file depends only on the parameter and ancilliary files parallel to that file, the use of MBprocess is inherently parallelizable.
• The file locking mechanism described above allows users to simultaneously run multiple instances of MBprocess on the same datalist structure.
• Each MBprocess instance will parse through the datalist structure and attempt to process each swath file in turn.
• Swath files will be skipped if they are up-to-date, or if they are locked.
• The MBprocess instances can be on different computers, as long as the same filesystems are mounted.
• The benefit of this pseudo-parallel processing tends to be limited by the network throughput. For instance, at MBARI the swath data are served on CIFS or Samba filesystems over Gigabet ethernet. We find that processing runs go faster with up to six MBprocess instances running on up to three different computers, but that adding more than six processes causes the entire run to take longer.

• New data formats:
  • HYSWEEP HSX multibeam data format (format id 201)

What Was New in MB-System Versions 5.1 and 5.2

The version 5.1.x and 5.2.z releases of MB-System included a number of changes and improvements relative to the version 4 releases. The most significant changes included:

• A new approach to managing data processing.
• Many tools - one output file. In previous versions of MB-System, each processing program read an input swath data file and produced an output swath data file. This "serial" processing scheme generally produced a large number of intermediate data files. MB-System version 5.0 features the integration of the editing and analysis tools with a single program, mbprocess, that outputs processed data files. The new "parallel" processing scheme covers bathymetry data processing, but does not yet incorporate the sidescan processing capabilities. All of the old tools and capabilities are still part of the distribution.
• Recursive datalists. The lists of data files used by gridding and plotting programs can now be recursive, making it simpler to manage data from many different surveys.
• Automatic format identification. MB-System programs will now attempt to automatically identify the swath data format based on the filename suffix.
• Extended inf files. Users can generate inf files by directing the output of mbinfo to a file named by adding an ".inf" suffix to the swath data file name. Several programs can parse inf files, if they exist, to quickly obtain data locations or ranges. This feature speeds operations such as gridding, mosaicing, and automated plotting.
  
  
• New command line tools.
• MBprocess. This new tool performs a variety of processing tasks and produces a single output processed swath data file. The program mbprocess can apply bathymetry edits from mbedit and mbclean, navigation edits from mbnavedit, sound velocity profile changes from mbvelocitytool, and a variety of other corrections.
• MBset. This new tool allows users to create and modify the parameter files used to control the operation of mbprocess.
• MBdatalist. This new tool allows users to list the files referenced by a recursive datalist structure. It can also be used to create the ancillary ".inf", ".fbt", and ".fnv" files for all of the data files referenced in a recursive datalist structure.
• MBsvplist. This new tool lists water sound velocity profiles embedded in swath data files, creating secondary files that can be read into MBvelocitytool.
• MBareaclean. This new tool identifies and flags artifacts in swath sonar bathymetry data within a specified area of interest. The area is divided into a grid with square cells or bins, and the data are grouped according to these bins. Once all of data are read, statistical tests are applied to the soundings within each bin.
• MBotps. This new tool uses the Oregon State Tidal Prediction Software (OTPSnc) package to calculate open ocean tidal models for bathymetry correction.
• MBextractsegy. This new tool extracts subbottom profiler data from swath files to SEGY format files.
• MBsegyinfo. This new tool extracts SEGY data file information and statistics.
• MBsegylist. This new tool produces arbitrary ascii tables from SEGY data files.
• MBsegygrid. This new tool grids seismic and subbottom data from SEGY data files.
• MBsegypsd. This new tool calculates sonograms from SEGY data files.
  
  
• Improved bathymetry and navigation editors.
• MBedit and MBnavedit now swallow data files whole rather than reading in limited size buffers.
• MBedit now outputs beam edit events rather than an entire swath file. The edits are applied by MBprocess.
• MBnavedit now outputs the edited navigation rather than an entire swath file. The edited navigation is merged using MBprocess.
• Both editors show the position of the currently displayed data within the entire data file.
• MBnavedit has two navigation modeling modes relevant to swath data collected using poorly navigated ROVs and towfishes. One mode applies a dead reckoning model with interactively set drifts, and the other involves inverting for an optimally smooth navigation by penalizing speed and acceleration.
  
• MBnavadjust. This new tool allows users to adjust poorly navigated surveys by matching features in overlapping swathes. It is particularly useful for processing surveys conducted from submerged platforms.

• New Visualization Based Tools
• MBgrdviz is a GMT grid 2D/3D visualization utility. MBgrdviz also allows the display of sonar navigation, sites, and routes, and interactive survey planning.
• MBeditviz is an interactive 3D visualization bathymetry editor and patch test tool.

• Support for Projected Coordinate Systems
• MB-System now incorporates the source code for the PROJ.4 Cartographic Projections library, providing support for (apparently) all commonly used geodetic coordinate systems. PROJ.4 was developed by Gerald Evenden (then of the USGS), and was obtained from the www.remotesensing.org website.
• A large number of commonly used projected coordinate systems (e.g. UTM) are defined in a file (mbsystem/share/projections.dat) distributed with MB-System. These include all of the standard UTM zones, all of the standard state plate coordinate systems, and most of the European Petroleum Survey Group (EPSG) coordinate systems (also including UTM).
• MB-System can now handle swath data that is navigated in a supported projected coordinate system. In particular, data files that are navigated with UTM eastings and northings instead of longitude and latitude can now be plotted and processed with MB-System.
• The programs mbgrid and mbmosaic can now output grids and mosaics in any of the projected coordinate systems specified in mbsystem/share/projections.dat.
• The TIFF images generated with mbm_grdtiff and mbgrdtiff now fully conform to the GeoTIFF standard, providing that the source grids or mosaics were generated using mbgrid or mbmosaic in either Geographic coordinates, UTM coordinates, or any of the EPSG coordinate systems specified in the projections.dat file. This means, for instance, that GeoTIFF images generated with mbgrdtiff will be properly georeferenced when they are imported into ESRI ArcGIS or other GIS packages.
  
  
• Restructuring the code.
• All of the C code now conforms to the ANSI C standard.
• The underlying input/output library (MBIO) has been substantially rewritten. The structure has been streamlined, simplifying both future development and support of the existing code. The MBIO API has been greatly modified.

• Handling of old Simrad multibeam data.
• Vendor format data from the old Simrad multibeams (pre-1997 sonars) are now supported by a single format id (51) rather than a separate format id for each sonar model. The old format id's are automatically aliased to 51, so existing shellscripts will continue to work.
• MB-System no longer supports beam flagging in format 51 data. The use of mbedit and mbclean on format 51 data will cause the flagged beams to be irrevocably nulled. Previous versions of MB-System used the highest bit in the depth values to represent beam flags because no Simrad data seemed to use that bit. We have not obtained data with depth values using the full bit-range, conflicting with the old beam flagging scheme. We recommend that old Simrad data be translated to the extended processing format (57) which contains proper beam flags and supports all processing functions. Format 57 is also used for processing data from the current Simrad multibeam sonars.
• Sidescan data from old Simrad multibeams (pre-1997 sonars) are now handled in the same manner as data from the newer sonars (e.g. EM3000, EM3000, EM120). The raw samples in the vendor data format are binned, averaged, and interpolated into a 1024 pixel sidescan swath. This binned sidescan is not saved in the vendor format, so (as above) it is recommended that the data be translated to an extended format (57) that stores both bathymetry beam flags and processed sidescan.

• Streamlining of MB-System Default Parameters.
• Prior to version 5.0, the MB-System defaults set by mbdefaults included the format id, a control for ping averaging, longitude and latitude bounds for windowing by area, and begin and end times for windowing in time. These values are no longer set in the .mbio_defaults file or controlled by mbdefaults. As noted above, the format id is automatically identified from the filename when possible. When filenames do not match one of the recognized structures, users must specify the format using the relevant programs -Fformat option. The controls for ping averaging and windowing in time and space are rarely used, and must now be explicitly set in command line arguments.

• New Data Formats
• Furuno HS10 multibeam bathymetry is supported as format 171.
• SeaBeam 2120 multibeam data in the L3 Communications XSE format are supported as format 94 (already used to support Elac Bottomchart MkII XSE data).
• Raw STN Atlas multibeam data generated by the upgraded Hydrosweep DS2 multibeam on the R/V Ewing are supported by read-only format 182. Processing is supported using the augmented read-write format 183.
• The IFREMER netCDF multibeam archiving data format is supported as format 75. Similarly, the IFREMER netCDF navigation archiving data format is supported as format 167.
• The STN Atlas processing data format SURF is supported as format 181. At present, SURF is supported as a read-only format. This allows plotting and gridding of the SURF data, but not processing. Writing or translating the SURF data to allow processing will be supported in a later version.
• The Hawaii Mapping Research Group's MR1 format is supported as format 64. This format is used to disseminate data from both the HMRG interferometric sonars (e.g. MR1) and the WHOI DSL 120 deep-towed inteferometric sonar. This format has been supported by including the code for the HMRG library libmr1pr in the MB-System library. Thanks to Roger Davis and HMRG for making the code available under the GPL.
• The Reson 7k format produced by the 7000 series Reson SeaBat multibeams and the Reson 6046 datalogger is now supported as format 88. This format can incoporate sidescan sonar and subbottom profiler data as well as the multibeam data.
• Third generation Simrad multibeams (EM122, EM302, EM710) are supported by formats 58 and 59.
• Imagenex and Odom DeltaT multibeams are supported by formats 191 and 192.

Back to MB-System Home Page...