velosearaptor

velosearaptor

Library of python modules for reading and processing raw RDI Teledyne ADCP data. The code interfaces the UH package pycurrents and its Multiread for efficient reading of raw ADCP data.

velosearaptor.io provides convenience functions for reading raw data either into an xarray.Dataset or into the output structure format provided by the UH software package.

velosearaptor.adcp is a collection of functions that are useful to quickly analyze raw ADCP data.

velosearaptor.madcp contains functions for processing moored ADCP data. Many thanks to Eric Fiering for sharing his code on moored ADCP data processing mcm_avg.py that much of this is based on.

Installation

The project is managed with uv:

git clone https://github.com/modscripps/velosearaptor.git
cd velosearaptor
uv sync

This creates a virtual environment in .venv/ with velosearaptor installed in editable mode. Run things inside it with uv run, e.g. uv run pytest, or see make help for the available development tasks.

pycurrents

velosearaptor depends on pycurrents, which is not on PyPI. uv sync installs it automatically from that GitHub snapshot of the UH repository. Since pycurrents builds a number of Cython/C extensions, a C compiler is required (on macOS run xcode-select --install).

To develop against a local pycurrents checkout, overlay it into the environment rather than editing pyproject.toml:

uv pip install -e ../pycurrents --no-deps

Installing magdec

magdec computes magnetic declination and is optional; it is only needed when lon/lat are provided and no declination is passed in explicitly.

Requirements: a C compiler and make.

./install_magdec.sh

This clones and builds geomag into geomag/ at the repository root, where velosearaptor finds it at runtime. Because this is a local build rather than a system install, it only works from a source checkout. To remove, delete the geomag/ directory.

For a system-wide install, cd geomag and do something like sudo make install; magdec is then picked up from your PATH instead.

History

v0.4.0 (unreleased)

New Features

  • A transducer-relative vertical frame for both averaging methods (#129). velosearaptor.madcp.ProcessADCP.average_ensembles and velosearaptor.madcp.ProcessADCP.burst_average_ensembles accept vertical_frame="transducer" and publish on the instrument's own bins, the distance from the transducer to the center of each bin, which is the axis velosearaptor.madcp.ProcessADCP.process_pings publishes as z. The output attribute vertical_frame records which coordinate a file carries, on all three methods. vertical_frame="depth" remains the default for the two averaging methods, every ping is still interpolated onto the universal grid built from dgridparams before anything is averaged, and that product is unchanged. Skipping the interpolation raises coverage. On the bundled 24606000.000 through burst_average_ensembles with tgridparams={"burst_average": True}, the depth-gridded product carries a finite u on 92.00% of its 19 by 365 cells and the bin-axis product on 98.60% of the same 19 by 365, with all 19 bins carrying velocity somewhere in the record. The transducer moves 1.42 m within a burst at the median and 2.37 m at the maximum, against a 16 m cell, too little for knockdown to account for the 6.6 point difference. Where the rest of it comes from was not measured. The candidates are the interp1 edge treatment and the alignment of the 24-level default depth grid against a 19-bin profile, which has_velocity then trims back to 19 levels. On the bin axis ngood and the four nbad_* counts are ping counts per bin and pg is the percentage derived from ngood, with nothing interpolated between the flag a criterion raised and the count, so the four rejection counts partition the rejected pings exactly at every published cell. The depth frame cannot do this. On its burst path 91 of 6471 cells read pg >= pg_limit with a NaN u, because interp1 widens a screened bin's mask onto the neighboring grid cells while the interpolated pg still reads above the limit. The bin depth of each averaging interval is written alongside z as a derived two-dimensional coordinate depth(z, time), equal to xducer_depth plus z for a downlooker and xducer_depth minus z for an uplooker. Its comment carries the caveat that xducer_depth comes from gsw.z_from_p while the depth gridding path of this package uses seawater.depth2. The two disagree by a few centimeters, so depth is exactly the reconstruction from the published xducer_depth and differs by that much from the axis a depth-gridded run of the same file would produce. dtop, dbot and d_interval are not written in this frame, because nothing read them, and dgridparams governs nothing here. The processing log gains a vertical_frame line on every path and stops listing dgridparams wherever the frame is transducer, which includes process_pings. has_velocity still drops levels carrying no velocity anywhere in the record, in this frame as in the other, so z is a subset of arange(NCells) * CellSize + Bin1Dist and index k of z is in general not bin k. z is published explicitly and maskbins is a file attribute, so nothing is renumbered silently, and tests/test_transducer_frame.py pins the drop. maskbins and interpolate_bin were bin-referenced already and are unchanged, and pg_limit still screens the burst path. process_pings now takes its range vector from self.tsdat.dep. It read the loop variable ens.dep before, which the source called a fudge, and which crashed when the reader could not return the last chunk and left ens as None. pycurrents rebuilds that vector from file-level attributes on every read, so every ensemble returns the same values and no published number moves. The pinned-output manifest was regenerated twice on this branch. The first regeneration added one dataset attribute, vertical_frame, to all four pinned configurations. The second changed two z comment attributes on the two process_pings configurations. Neither moved a checksum, a dtype, a shape or a dimension. The madcp module notes gain a Vertical Frame section describing both coordinates, what the frame changes beyond the axis, and what it leaves alone.
  • None switches an editing criterion off, for every entry of editparams (#131). maskbins=None and pg_limit=None already meant that, and velosearaptor.madcp.ProcessADCP._add_meta_data_to_ds already wrote "none" for a None value of max_e, max_e_deviation and min_correlation, while the three criteria raised TypeError on it. min_correlation=None disables the correlation test. max_e_deviation=None disables the sigma branch of the error velocity test, so the threshold is the fixed max_e. max_e=None removes the cap and leaves max_e_deviation * std(e). Both None switch the test off and max_e_applied records inf, distinct from the NaN of a threshold that could not be estimated. YAML null reaches the criteria as None. The defaults are unchanged and the pinned output passes with every checksum untouched. The max_e_applied comment now says where the standard deviation is taken on each path, within the interval on the two averaging methods and over the whole record on velosearaptor.madcp.ProcessADCP.process_pings, which the method docstring said and the file attribute did not. Measured for the decision, on the bundled files: the sigma branch binds on 365 of 365 bursts of 24606000.000 and removes 2.0 to 3.4% of candidate pings per depth level with 8.7% at two levels near the far end, and on binmap_16670013.000 through process_pings(binmap=True) the absolute 0.2 binds for the whole record and the test removes 1.3 to 2.5% in the near half rising to 7 to 13% in the far quarter. The rejection follows the instrument's signal-to-noise profile, which is what an error velocity test is for, so the default stays at 2 sigma. The madcp module notes gain a Quality Control section listing the four criteria in the order they are applied, what each parameter controls, where pg_limit applies, and what the file records.

Documentation

  • Add the v0.3.0 entries for the QC fixes of August 2026, PR85 through PR95, which were released in that tag and left out of its history. They are filed under v0.3.0 where they belong and this line records that a released section was edited.

Internal Changes

  • Apply the QC mask once, from the flags, after the rotation into earth coordinates (PR130). Editing no longer writes to xyze. velosearaptor.madcp.ProcessADCP._edit_masks and _edit are now _qc_flags and _qc, which raise the flags of PR123 and PR125 and write nothing to any velocity array. velosearaptor.madcp.ProcessADCP._to_enu applies ens.valid to the rotated velocities, once, through the new module-level _apply_qc, so an ensemble leaves it masked as it did before, which tests/test_qc_flags.py asserts on every call of the four pinned configurations and under ibad. velosearaptor.madcp.ProcessADCP.process_pings calls the same function once more after its loop with the record-wide error velocity flag, because on that path the flags have two lifecycles. This is the last step of #30 and closes the flag architecture proposed there in 2022: each criterion stored as its own flag, the combined mask applied at one site. Output is byte-identical on the four pinned configurations and tests/data/pinned_output.json passes untouched, and six further runs with maskbins and ibad across all three processing methods, 116 variables in all, are identical to main. Three properties of the code downstream make the move bitwise safe. Both rotation kernels of rdi_xyz_enu compute a cell of enu from the same three products in the same order, and a value under a mask never reaches the output, so which kernel the masked/unmasked dispatch picks does not change a published number. interp1 fills masked input with 1e100 and masks every output above 0.999e100, so which cells are masked decides the result and what sits under the mask does not. MaskedArray.mean and MaskedArray.std fill masked cells with 0, sum the full array and divide by the unmasked count, so the same mask gives the same element set, the same summation order and the same result. The remaining question is whether one boolean per cell can stand in for a mask carried per component, and the components differ only under ibad. Measured on both bundled files, before and after editing, all four components carry one mask on every path when ibad is unset. With ibad set, Transform.beam_to_xyz masks e over the whole ensemble, all 28800 cells of the first 720 by 40 ensemble of tests/data/binmap_16670013.000, because a three-beam solution has no error velocity, while u, v and w share one mask. Masking all four components from ~ens.valid therefore adds the edited cells on u, v and w, and adds nothing to e under ibad, where nothing is left to add. editparams keeps its name. The QCparams rename proposed in #30 was considered and declined: the constructor keyword, velosearaptor.madcp.ProcessADCP.parse_editparams, which both example notebooks call, the proc.editparams attribute users can set directly, and the editparams: key of the YAML parameter file at the global and the per-instrument level would all change for a cosmetic gain, and a rename without a compatibility shim makes an existing .yml file fall back to default editing parameters silently, because the reader initializes a missing key to None. No published number, dtype or variable attribute changes, and every renamed method is private, so no public name changes either.
  • Publish per-criterion QC rejection counts on all three processing paths (PR126), the step of #30 that puts the flags of PR123 on the output Dataset. nbad_nodata, nbad_cor, nbad_maskbins and nbad_max_e count, at each published cell, how many pings the no-data, correlation, user-declared-bin and error-velocity criteria rejected. Each rejected ping is attributed to exactly one of the four, by the fixed precedence nbad_nodata, then nbad_cor, then nbad_maskbins, then nbad_max_e, because the raw criteria overlap. Without that rule the correlation test also fires on almost every cell that already carries no beam data at all, measured at 100.00% of them under binmapping and 99.63% without, 29.26% of everything the correlation test reports under binmapping. On velosearaptor.madcp.ProcessADCP.process_pings and velosearaptor.madcp.ProcessADCP.burst_average_ensembles the four counts partition the rejected pings exactly on the instrument's own bins, measured on the bundled files at 11011 of 11011 rejected for the burst path. process_pings publishes on those same instrument bins, so its published values sum exactly too. burst_average_ensembles instead publishes on the universal depth grid, where each of the four counts is interpolated and floored independently, and floor(a) + floor(b) can fall short of floor(a + b). Measured on the bundled file, the published values sum exactly on 25.7% of cells and fall short by up to 2 on the rest. velosearaptor.madcp.ProcessADCP.average_ensembles partitions its own bins the same way, 20701 of 20701, but publishes on the universal depth grid too, where a cell fed by two instrument bins rejected for different reasons counts in both, so the published sum can exceed the rejected count, measured at a factor of 1.074, 20862 counts over 19422 invalid on-profile grid-cell-pings, while still attributing 100.00% of invalid on-profile cells to at least one criterion. Computing the same ratio as npings - ngood from the published file instead gives 1.036, because ngood counts a ping that never reached that grid depth as neither good nor rejected. The four counts are int8 and unmasked on process_pings, where every cell is one ping. A bin declared bad through maskbins carries no velocity for the whole record, so its level leaves the published depth axis and nbad_maskbins reads 0 on this path. Adding the four counts costs 4 bytes per cell, 28 to 32, against 60 had they been float64. On the two averaging paths they are float64 and NaN wherever the depth lies outside the instrument's profile, matching pg, and ngood on the burst path. No existing published number changes. The pinned-output manifest was regenerated at each step of this branch and reviewed in the diff, most recently for the burst comment above, which moved four comment attributes on burst_average_ensembles and no checksum, dtype, dimension or shape. A further six off-pin runs with maskbins and ibad confirmed by hand against main across all three processing methods. Of the 92 pre-existing variables those six configurations publish, 82 are byte-identical and the other 10, ngood and pg wherever they appear, differ only in a new ancillary_variables reference to the four counts.
  • Compute percent good from the QC flags on all three processing paths (PR125). pg was a side effect. velosearaptor.madcp.ProcessADCP._edit wrote np.ma.masked into xyze at three points, that mask survived the rotation into earth coordinates and the depth gridding, and each path counted whatever was left in the velocity array. It is now counted from the per-criterion flags of PR123, which is step 3 of #30 and what allows those mask writes to be removed later. The three editing flags do not account for the mask on their own, because cells arrive at _edit already masked. Binmapping cannot fill every cell of every beam, and the instrument rejects beams by itself. Measured on the bundled files, 1.71% of cells arrive masked on velosearaptor.madcp.ProcessADCP.average_ensembles and on velosearaptor.madcp.ProcessADCP.process_pings without binmapping, 16.29% with binmapping, which is 29.3% of every invalid cell there, and 0.03% on velosearaptor.madcp.ProcessADCP.burst_average_ensembles. Two flags carry that: flag_no_data_beam, the mask on the beam velocities, and flag_no_data, its reduction over the beams that enter the velocity solution. ens.valid is the complement of flag_no_data, flag_cor, flag_maskbins and flag_max_e, and equals the cell mask on xyze after editing on every call of all four pinned configurations, which tests/test_qc_flags.py asserts. The reduction from beam space to cell space is now one function, _cell_flag, shared by the correlation criterion and the new one, so softening the whole-cell rejection for three-beam solutions (#18) is a single change. It honors ibad for the same reason the correlation reduction did (#90). Transform.beam_to_xyz fills the excluded beam from the other three before transforming, so that beam's mask never reaches xyze, and reducing over all four beams instead disagrees with the mask on 86 cells of the first ensemble of tests/data/binmap_16670013.000 at ibad=0. Each path keeps the grid it counts on, settled in PR124. process_pings assembles validity per ens_size chunk and applies the record-wide error velocity flag after the loop, on the instrument bins. burst_average_ensembles sums validity over the pings of a burst, also on the instrument bins. average_ensembles carries validity as a sixth column of the single interp1 call in _regrid_enu_amp, so counting on the depth grid costs no extra interpolation, and the velocities and amplitudes that call returns are bitwise unchanged. On process_pings the validity array starts False, because a chunk the reader cannot return skips the loop body without raising any flag and a True default would publish it as 100% good. None of the four pinned configurations contains such a chunk, so tests/test_qc_flags.py makes the reader fail one. Nothing is written to the output dataset and no published number changes. tests/data/pinned_output.json passes untouched, and six further runs with maskbins and ibad across all three methods, 92 variables in all, are identical to main.
  • Say why percent good is counted on a different grid on each averaging path (PR124). The per-path pg comments of PR118 each said what that path counts, and none said why the paths differ, which is what was left of #82. The criterion is whether the transducer moves appreciably within the averaging window. velosearaptor.madcp.ProcessADCP.burst_average_ensembles counts on instrument bins before depth gridding, because every ping in a burst is assigned the burst mean transducer depth, so the pings share one bin grid by construction and the count is exact in bin space. Measured on the bundled 24606000.000, the transducer depth spread within a burst is 0.000 m. velosearaptor.madcp.ProcessADCP.average_ensembles counts on the gridded pings, because over its longer intervals the transducer moves. Measured on the bundled binmap_16670013.000, the spread within a 30 minute interval is 4.661 m median, against a 4 m bin size and a 4 m grid interval, so the pings in one interval share no common bin grid to count on. velosearaptor.madcp.ProcessADCP.process_pings averages nothing, so the ordering question does not arise there. The two averaging comments now name each other, so the difference reads as a choice. The order is settled and is not a candidate for unification. pg_limit screens velocities on the burst path, and at a threshold of 80 the choice of order would change the screening verdict on 11% of cells, which reaches published velocities. Nothing about how pg is computed changes on any path. tests/test_provenance_attrs.py asserts that the criterion is stated, so the sentence cannot be dropped later.
  • Regenerate the pinned-output manifest and drop the release claim from its name (PR124). tests/data/pinned_output_v0.3.0.json is now tests/data/pinned_output.json, and it holds the last reviewed output. The pg comment change above is the first intended output change since the pin of PR122, and regenerating from the v0.3.0 tag reproduces the old comment, so the pin either tracks the branch or accumulates a list of intended deviations from the release. Tracking the branch keeps one source of truth. Every regeneration is reviewed in the diff of the manifest and gets an entry here saying what moved, and git log tests/data/pinned_output.json lists them. As a tripwire against a change nobody intended this is as strong as pinning a release. What it gives up is the question a user with a release installed would ask, whether the numbers of that version still hold, and answering that now means checking the tag out and regenerating there. This regeneration moved four variable attributes, the pg comment on each of the four pinned configurations, and no checksum, dtype, dimension or shape.
  • Build a per-criterion QC flag alongside each editing mask (PR123). Editing rejects a cell for one of three reasons and the output records only that it was rejected. Each criterion now also produces a boolean flag, carried on the ensemble next to the mask it already produces, as the first step of #30. flag_cor_beam holds the correlation test per beam, flag_cor reduces it over the beams that enter the velocity solution, flag_maskbins marks the bins declared bad through maskbins, and flag_max_e marks error velocities above the applied threshold. Nothing is written to the output dataset and no published number changes, which the pinned-output test of PR122 holds for the four default configurations and a further six runs with maskbins, ibad and binmap confirmed by hand across all three processing methods. The four criteria become module-level functions taking their parameters explicitly, so a criterion can be evaluated without constructing a velosearaptor.madcp.ProcessADCP, and velosearaptor.madcp.ProcessADCP._edit_masks computes both of its flags before writing anything to xyze, so the beam-space criteria no longer read the array they mask. The three flags do not share a lifecycle. The two averaging methods raise all three per ensemble, while velosearaptor.madcp.ProcessADCP.process_pings raises the two beam-space flags per chunk and the error velocity flag once over the whole record after the loop. editparams keeps its name and the three public method names are untouched.
  • Pin the full output of all four processing configurations against v0.3.0 (PR122). tests/test_pinned_output.py compares every coordinate, data variable, variable attribute and dataset attribute of the finished Dataset against a reference recorded from the v0.3.0 tag, for velosearaptor.madcp.ProcessADCP.average_ensembles, velosearaptor.madcp.ProcessADCP.process_pings with and without binmapping, and velosearaptor.madcp.ProcessADCP.burst_average_ensembles on the bundled raw files. Values are compared by sha256 over the filled arrays, so the comparison is bitwise. Every other test in the suite asserts on the one quantity its issue was about, so a restructuring of the three processing loops can pass all of them while changing velocities. Replacing the float64 accumulation in the average_ensembles mean with float32 leaves all 167 pre-existing tests green and moves u, v, w and e at the 1e-7 relative level. The reference is a 56 kB JSON manifest taken from the in-memory Dataset. A stored netCDF would be 4.9 MB, would grow the repository on every intentional output change, and would pin the netCDF encoding of whichever xarray version wrote it alongside the numbers. The manifest records dtypes, shapes, attribute types and per-variable minimum, maximum and mean, so a failure names the variable that moved and puts a magnitude on it. proc time and velosearaptor_version are excluded, being the two attributes that differ between runs of identical code. Magnitudes below 1e-12 are folded onto zero before anything is compared. e and w are sums and differences of beam pairs and cancel exactly for some cells, and whether the cancellation lands on zero or on a residue depends on the order the host's numpy adds the terms in, so without the fold those two variables disagreed between CI runs of identical code while their minimum, maximum, mean and invalid count matched to the last digit. On the per-ping path e carries residues down to 4.3e-19 and the smallest surviving magnitude in any pinned variable is 3.0e-7, so the threshold sits in twelve empty orders of magnitude. It folds values onto zero and quantizes nothing else, so a one-ulp change anywhere else still moves the checksum. Regenerate the reference with uv run python tests/test_pinned_output.py --write.

v0.3.0 (2026 August)

Everything since v0.2.0 in March 2022: the move from gadcp to velosearaptor, the switch to a src/ layout with uv and ruff, and a systematic pass over the QC, editing and output-metadata code. Several of the bugs fixed in that pass changed results silently, so files processed with an earlier version may need reprocessing. The checklist below says when.

Upgrading

Reprocess if you:

  • supplied t0/t1 as strings for a window whose calendar year differs from that of the first ping, the normal case for a mooring spanning New Year. The requested window was silently off by a year (PR106).
  • set pressure_scale_factor to anything other than 1 and processed continuous data with velosearaptor.madcp.ProcessADCP.average_ensembles. The factor was applied twice on the low-pass pressure path, which sets xducer_depth and the depth grid (PR111).
  • set maskbins, or ran velosearaptor.madcp.ProcessADCP.process_pings with a non-default ens_size. The adaptive error velocity threshold was estimated over the wrong samples (PR116).
  • processed with binmap=True and use amp. Amplitude was biased low on 4.3% of finite cells in the bundled binmapped test file, by a median of 22 counts. Velocity is unchanged (PR105).
  • averaged over intervals longer than 32767 pings, a 12 h average at a 1 s ping rate, where npings and ngood either raised OverflowError or wrapped silently (PR104).
  • use ngood from velosearaptor.madcp.ProcessADCP.burst_average_ensembles, where depth bins the instrument never sampled read 0 rather than missing (PR115).
  • ran a record with a non-uniform ping interval through the low-pass pressure path, where the documented 30 minute cutoff never bound. Continuous records are unaffected (PR108).
  • have a record with non-monotonic time stamps, where the repair was classified from the first backward jump alone and applied to the whole record (PR109).
  • processed with binmap=True and ibad set. The excluded beam was silently re-included when the beam-to-instrument transform was recomputed after binmapping (PR85).
  • use u_error, v_error or w_error. They were the standard deviation over the number of pings in the interval, including pings that editing had removed, which understates the standard error by a factor of sqrt(pg/100), about 2.2 at 20 percent good (PR86).
  • set ibad and rely on pg or on the editing. The correlation test rejected a cell when the excluded beam alone fell below min_correlation, so an instrument with a dead beam paid for it in the editing as well as in the three-beam solution (PR91).
  • use pg from velosearaptor.madcp.ProcessADCP.process_pings, which read 0 everywhere (PR89), or from velosearaptor.madcp.ProcessADCP.average_ensembles with more than 327 good pings per interval, where it wrapped negative (PR95).
  • select depth levels by the published range on either averaging method. Levels that carried amplitude and no velocity at all were kept in the file (PR88).

Update code if you:

New Features

  • Install magdec via shell script.
  • Add an example notebook.
  • Read serial number from binary file and compare with SN in meta data (PR13). By Jesse Cusack.
  • Add bin mapping for cases with large pitch and roll. This adds velosearaptor.madcp.ProcessADCP.process_pings (PR17). By Jesse Cusack.
  • Add position to moored ADCP meta data (PR21).
  • Allow for external input of pressure time series in velosearaptor.madcp.ProcessADCP (PR12).
  • Improve default depth grid to also work well with mooring knockdowns (PR44).
  • Optionally read processing parameters from .yml-file (PR26).
  • Allow for string format when supplying start and end time in .yml parameter file (PR63).
  • Add CF-compliant meta data to output dataset (PR26).
  • Add ngood, the number of pings that entered the mean in each bin, to the output of both averaging methods, and scale the velocity standard errors by it (PR86). u_error, v_error and w_error were the standard deviation over npings, the total count for the interval, which includes the pings editing removed. In bins with low percent good that understates the error by sqrt(pg/100), about a factor of 2.2 at pg = 20, and the bias is largest where the uncertainty matters most. ngood is the count pg is derived from, and the standard error is NaN where it is zero. velosearaptor.madcp.ProcessADCP._ave2nc divides by sqrt(ngood) and falls back to npings when ngood is absent.
  • Record the applied QC parameters in the output (PR87). The effective error velocity threshold, min(max_e, max_e_deviation * std(e)), is computed per averaging interval from that interval's data and was discarded after use, so the editing actually performed could not be reconstructed from the file. It is now the variable max_e_applied on the time axis on all three processing paths, NaN where it could not be computed, and pg_limit and maskbins join max_e, max_e_deviation and min_correlation in the attributes. Part of #83. The question of whether the adaptive threshold should exist at all continues in #131.
  • Add a depth_offset argument to velosearaptor.madcp.ProcessADCP for an ADCP whose pressure sensor carries a constant bias (PR117). The offset in meters is added to the published depth coordinate of the two averaging methods and to xducer_depth on all three paths. Positive means the instrument was deeper than the pressure record says. All gridding and interpolation run in uncorrected pressure-derived depth, so u, v, w and pg come out bit-identical to an unoffset run and the published axis is the unoffset axis translated by a constant. That is what shifting the finished file by hand gives. Supplying a corrected external pressure series does not reproduce it, because depth(p) is nonlinear, so a constant shift in dbar is a varying shift in meters, and re-deriving per-ping depths re-samples the interpolation onto the grid. Three consequences follow from applying the offset at the output only. dtop and dbot in dgridparams are read in the uncorrected frame, so requesting dtop=100 with depth_offset=5 publishes an axis starting at 105, and the corrected extent is logged when the grid is built. The offset does not apply to z, the transducer-relative distance published by velosearaptor.madcp.ProcessADCP.process_pings, where only xducer_depth moves. pressure keeps the unmodified measurement, and when the offset is nonzero the equivalent corrected pressure is stored alongside it as pressure_corrected so that recomputing depth from the file reproduces the published axis. The applied offset is recorded in the depth_offset dataset attribute and named in the comment on depth and xducer_depth. Files processed without the argument are unchanged.

Breaking Changes

  • Transfer repository from gunnarvoet to modscripps and rename from gadcp to velosearaptor. Legacy code still exists at https://github.com/gunnarvoet/gadcp.
  • Change processed dataset coordinate z to depth (PR48).
  • Change vel_std variables in output dataset to vel_error by dividing the standard deviation of each average by the square root of the number of pings (PR26).
  • Low-pass filter (inherently noisy) pressure before ensemble-averaging continuous ping data (PR61). By Gunnar Voet.
  • Rename the vertical coordinate of velosearaptor.madcp.ProcessADCP.process_pings output from depth to z. Any script indexing ds.depth on that path breaks, and the values it was reading were never water depth. For an uplooker the axis also runs the opposite way, so previously published per-ping files are inverted relative to water depth as well as offset from it: on tests/data/binmap_16670013.000 (mean xducer_depth 154.5 m) depth = 5.15 was 149 m of water depth and depth = 161.15 was above the sea surface. Reprocess, or flip and offset by hand (PR103).
  • ngood from velosearaptor.madcp.ProcessADCP.burst_average_ensembles is now float rather than int32, and reads NaN for depth bins outside the instrument's profile where it previously read 0 (PR115). Code that selects ngood == 0 to find bins where no ping survived editing now gets only those, which is the point of the change; code that assumes an integer dtype needs updating. Every other value is unchanged.
  • The pg_limit dataset attribute reads "not applied" on files produced by velosearaptor.madcp.ProcessADCP.average_ensembles and velosearaptor.madcp.ProcessADCP.process_pings (PR118). Only velosearaptor.madcp.ProcessADCP.burst_average_ensembles screens percent good against it, as the class docstring has always said, and files from the other two paths recorded a threshold that had never been applied to them. Code doing float(ds.attrs["pg_limit"]) on such a file now raises where it previously returned that threshold. The burst path still records the number, or "none" when the screening is switched off.

Bug Fixes

  • Fix conda/pip environment.
  • Read correct instrument orientation when a majority of the time series has been recorded outside the water (PR44).
  • Read paths provided via pathlib.PosixPath objects (PR55). By Gunnar Voet.
  • Fix xarray warning in groupby (PR64). By Gunnar Voet.
  • Replace scipy.stats.mode with np.unique for dominant period calculation (PR66).
  • Fix non-monotonic ADCP time vectors by interpolating isolated bad pings, truncating segment overlaps, or raising on ambiguous cases (PR70).
  • Raise a clear error in ProcessADCP._parse_sysconfig when no pressure record exceeds 15 dbar. The existing guard could never fire, so this case surfaced as a confusing IndexError.
  • Stamp the processing date in the log header in UTC rather than local time.
  • velosearaptor.madcp.ProcessADCP.process_pings now publishes its vertical axis as z, the distance from the transducer to the center of each bin, carrying the z attributes from velosearaptor.io.cf_conventions. The axis was never water depth on that path. The two averaging methods do grid onto water depth and keep publishing depth. Bin depth is recoverable per ping as xducer_depth plus z for a downlooker and xducer_depth minus z for an uplooker (PR103).
  • Pass ibad when the beam-to-instrument transform is recomputed after binmapping (PR85). velosearaptor.madcp.ProcessADCP.process_pings(binmap=True) called _calculate_xyze without it, so a beam excluded at construction, and honored on the non-binmapped path, was silently re-included in the binmapped product.
  • Drop depth levels from the averaged product by velocity coverage (PR88). velosearaptor.madcp.ProcessADCP._ave2nc dropped levels where every variable was missing. Editing masks velocities only, so amp can be finite in a level that carries no finite velocity sample, and files advertised depth levels with no velocity in them. A level is now kept when u is finite anywhere in time.
  • Compute percent good in velosearaptor.madcp.ProcessADCP.process_pings (PR89). pg was allocated as zeros and the computation was commented out, so the single-ping product read pg = 0 wherever there was data and any percent-good filter above zero removed the whole dataset. For single-ping data percent good is binary, 100 where the ping survived editing and 0 where it was edited out.
  • Exclude the ibad beams from the correlation test (PR91, #90). velosearaptor.madcp.ProcessADCP._edit rejected a cell when any of the four beams fell below min_correlation, including a beam excluded through ibad that never enters the velocity solution. On a WH150 with ibad=2 at min_correlation=64, the four-beam test rejected 14.7% of in-range samples where the three good beams alone rejected 6.4%. The all-beams behavior is unchanged when ibad is None.
  • Record maskbins faithfully when it is given as a list of bin numbers (PR93). The attribute added in PR87 was written with np.flatnonzero, which is correct for the documented boolean form and returns positions for an integer list, so maskbins=[9, 10, 11, 12, 13, 14] masked bins 9 to 14 and recorded [0, 1, 2, 3, 4, 5], and [0, 1] recorded [1] because bin 0 is falsy. The editing was always right. An empty selection is recorded as "none", since netCDF cannot hold an empty array.
  • Widen ngood before computing percent good in velosearaptor.madcp.ProcessADCP.average_ensembles (PR95, #94). 100 * ngood // npings was evaluated in int16, which wraps once ngood exceeds 327, so the bins with the most good pings came out with negative pg and a downstream pg filter discarded exactly the best data. A 10 s ping rate at a 1 h interval is 360 pings. velosearaptor.madcp.ProcessADCP.burst_average_ensembles was not affected.
  • Fix four crashes on documented argument paths that no test exercised (PR104). velosearaptor.madcp.ProcessADCP.process_pings(start=..., stop=...) raised UnboundLocalError. Both arguments are ping indices, which the docstring now states along with the fact that they mean something different from average_ensembles(start, stop), and they are now range checked, an empty range included. burst_average=True on continuous data now raises a clear "this data is not burst sampled" error rather than computing -2147483648 pings per burst and failing later in an unrelated reduction. npings and ngood are stored as int32, so averaging intervals longer than 32767 pings (a 12 h average at a 1 s ping rate) no longer raise OverflowError or wrap silently. This widens the dtype of both variables in the output file. interpolate_bin within two bins of either profile end no longer indexes negative bins or runs off the end. The neighbour window is clipped, and the first and last bin, which have no neighbour on one side, raise a clear error. An interpolated bin keeps its own pg and ngood, so the interpolation stays visible in the output.
  • Binmapping no longer writes unmasked NaN into vel, amp and cor (PR105). Cells outside the mapped beam range are masked instead. amp and cor are integer typed, so the NaN written into them was destroyed by the cast and became 0, and those zeros were averaged over the four beams into the published amplitude. On the bundled binmapped test file that biased 4.3% of cells low by a median of 22 counts, so anyone who processed with binmap=True should reprocess for amplitude. Velocity is unchanged at the default min_correlation of 64, because the uint8(nan) cast happened to reject exactly the cells the mask now rejects. That the correlation test rejected them at all rested on uint8(nan) evaluating to 0, which is undefined behavior, so the rejection is now explicit rather than incidental.
  • Convert t0/t1 given as strings against the yearbase of the raw file rather than against the year of the requested time itself (PR106). A requested window in a later calendar year than the first ping was silently off by a year, which is the normal case for a mooring spanning New Year. velosearaptor.madcp.ProcessADCP.parse_tgridparams now also raises when the requested window does not overlap the data at all, and when it is empty or reversed, naming the requested window and the file's actual range in calendar time. A window that reaches beyond the data on one side still passes quietly.
  • Fix external pressure NaN patching, which raised TypeError: only integer scalar arrays can be converted to a scalar index whenever the external pressure record had more than one contiguous NaN run (PR107). That is the leading-plus-trailing case the code was written for. Leading and trailing NaN runs recorded on deck are now both patched with atmospheric pressure. A NaN gap between good data, an end run recorded in the water, or an external record that is NaN across the whole ADCP time range now raises rather than flowing into the interpolator as NaN.
  • Compute the pressure low-pass cutoff on wall-clock time rather than on the modal ping interval (PR108). scipy.signal.filtfilt treats every ping as equally spaced, so on burst-sampled data the documented 30 minute cap was expressed in a fictitious time base and never bound. A 20 dbar knockdown came back as 8 dbar in a synthetic test. fs now comes from the mean sampling interval over the record, and a warning is raised when the ping pattern is detectably non-uniform. Continuous records are unaffected. A record whose mean ping interval puts the cutoff at or below the Nyquist period now raises with a clear message, where it previously failed inside scipy.signal.butter.
  • velosearaptor.madcp.ProcessADCP._ensure_monotonic_dday now classifies and repairs each backward or repeated time step on its own (PR109). It detects when the outlier is the ping before a backward jump, a forward clock spike, rather than always rewriting the ping after it, handles repeated timestamps explicitly, and verifies that the result is strictly increasing, raising ValueError if the repair did not achieve it. Previously the classification looked only at the first backward jump and applied that verdict to the whole record, and nothing checked the outcome. Repairs are also carried back into the raw time base in tsdat.dday, which the pressure low-pass filter reads and which previously kept the original bad timestamps.
  • Apply pressure_scale_factor exactly once on the low-pass pressure path (PR111). velosearaptor.madcp.ProcessADCP._lowpassfilter_pressure multiplied the pressure record by the factor a second time, after _scale_pycurrents_pressure had already applied it when the record was built, so the low-passed pressure came out scaled by the square of the factor while the raw record was scaled once. The low pass is the default pressure source for continuous data in average_ensembles, and dat.pressure sets xducer_depth and the depth grid, so anyone who set pressure_scale_factor to anything other than the default of 1 on that path should reprocess. The raw, burst-average and external-pressure paths were always correct. At the default of 1 nothing changes.
  • velosearaptor.madcp.ProcessADCP.burst_average_ensembles now reports depth bins outside the instrument's profile as missing rather than as zero percent good (PR115). Percent good is computed on instrument-relative bins and then interpolated onto the universal depth grid. Off-grid depths came back masked from that interpolation and were cast to 0, raising a RuntimeWarning once per burst and leaving a cell the instrument never sampled indistinguishable from one where every ping was rejected by editing. pg and ngood now carry NaN there. The published pg values are unchanged, because _ave2nc already masked pg wherever amp is NaN and the amplitude regridding uses the same depth vector as this interpolation, so the amp mask happened to cover exactly the same cells. Nothing masked ngood, so there the wrong value reached the file: on the bundled burst-sampled test file, 365 of its 366 zero cells were unsampled depth bins and one was a real cell with no surviving ping.
  • Compute the adaptive error velocity threshold over the samples it is meant to describe (PR116). velosearaptor.madcp.ProcessADCP._edit applied maskbins after taking the standard deviation of the error velocity, so bins the user had declared bad set the threshold that decides which of the kept bins survive. When those bins are the noisy ones their contribution pushes max_e_deviation times sigma past max_e and the adaptive criterion switches off altogether: on tests/data/binmap_16670013.000 with bins 30-39 masked, the applied threshold was the fixed 0.2 and is now 0.1634. Separately, velosearaptor.madcp.ProcessADCP.process_pings called _edit once per ens_size chunk, so the chunk size was the window the standard deviation was estimated over, even though ens_size is documented as a memory knob. The same file gave 60644 finite velocity cells and a record-mean u of 0.03501 at the default ens_size of 50000, against 58768 and 0.03441 at 250. The correlation and maskbins masks are still applied per chunk and the error velocity test now runs once over the whole record, so the two runs agree. Results are unchanged for the two averaging methods unless maskbins is set, and unchanged for process_pings at the default ens_size. Anyone who set maskbins, or who ran process_pings with a non-default ens_size, should reprocess.
  • Stop pointing at an npings variable that is not in the file (PR118). velosearaptor.io.cf_conventions declares ancillary_variables: "npings" on twelve entries and velosearaptor.madcp.ProcessADCP.process_pings does not write npings, so nine variables in every single-ping file named something absent. A CF checker failed on it, and a reader looking for the sample count behind pg or the error estimates found nothing. Names the file does not carry are now dropped from ancillary_variables when the dataset is built, so a variable added later cannot reintroduce the reference. The two averaging methods do write npings and keep the reference.

Documentation

  • Consolidate readme and history files.
  • Add button with link to source code on GitHub (PR43).
  • Describe the z coordinate of the raw output in its attributes (#52). z is the distance from the transducer to each bin, not water depth, and keeps its name for that reason. The averaging methods regrid onto water depth and publish it as depth, while velosearaptor.madcp.ProcessADCP.process_pings keeps the transducer-relative axis and publishes it as z.
  • Record how the file was produced in its attributes (PR118). Processing settings that change the numbers went to a log file that does not travel with the netCDF, and driftparams was not logged at all. Every file now carries the velosearaptor version, which of the three methods produced it, the pressure source and pressure_scale_factor, whether a clock correction was applied and at what drift rate, ibad, and the requested time window in calendar time. Settings are recorded only on the paths where they governed the run. The depth grid and the averaging interval appear on the two averaging methods, dt_hours on velosearaptor.madcp.ProcessADCP.average_ensembles alone since a burst takes its interval from the ping pattern, binmap on velosearaptor.madcp.ProcessADCP.process_pings, and interpolate_bin on velosearaptor.madcp.ProcessADCP.burst_average_ensembles. ens_size is left out because it no longer affects results.
  • Say which of its three meanings pg carries (PR118, #82). It is binary on the single-ping path, computed after per-ping interpolation onto the universal grid in velosearaptor.madcp.ProcessADCP.average_ensembles, where interp1 widens an edited bin and mooring knockdown is absorbed into the number, and computed on instrument-relative bins before interpolation in velosearaptor.madcp.ProcessADCP.burst_average_ensembles. Each file now gets the comment for the path that made it, including the note that the output pg is a velosearaptor-defined quantity sharing a name with RDI's own four PercentGood fields, which velosearaptor.io.read_raw_rdi reads and no processing step uses.

Internal Changes

  • Remove gvpy dependency (PR27).
  • Auto-detect sonar type instead of hardcoding Workhorse (PR65).
  • Speed up bin-averaging (PR67).
  • Move to a modern pyproject.toml / uv setup with a src/ layout (#69). Removes setup.py, setup.cfg, requirements.txt, and environment.yml; tests move to a top-level tests/ directory.
  • Install pycurrents from its git snapshot instead of the retired Mercurial repository, which fixes the documentation build (#68).
  • Point the magdec install at the new geomag git repository and drop the conda requirement (#71).
  • Replace isort/black/flake8 with ruff, and clean up the findings so make check passes. Linting, format checking, and the test suite now all run in CI on pull requests.
  • Remove velosearaptor.yoyo. Every method referenced the gv (gvpy) and plt names that were dropped when the gvpy dependency was removed in PR27, so the module raised NameError on any call.

v0.2.0 (2022 March)

This release brings a major refactoring of the velosearaptor.madcp module with lots of breaking changes. The module now allows for improved ensemble averages for burst sampling schemes with better control of the editing parameters.

New Features

Breaking Changes

Bug Fixes

  • Correct time stamp calculation for burst averages.
  • For burst sampling schemes average pressure before gridding to depth.

Documentation

Internal Changes

  • Improved the pip/conda requirements to automatically install the pycurrents package.

v0.0.1 (2020-04-28)

  • Moved all ADCP-related functions from gvpy to this module.
 1"""
 2.. include:: ../../README.md
 3
 4---
 5
 6.. include:: ../../HISTORY.md
 7
 8"""
 9
10__all__ = ["adcp", "io", "madcp", "tools"]
11
12__author__ = "velosearaptor Developers"
13__email__ = "gvoet@ucsd.edu"
14# A `.devN` suffix marks development between releases. Keep this identical to
15# `version` in pyproject.toml. `madcp.ProcessADCP._add_meta_data_to_ds` writes
16# this string into every output file as the `velosearaptor_version` attribute,
17# so a file produced from an untagged commit has to stay distinguishable from
18# one produced from the release. Drop the suffix in both files in the release
19# commit, then reopen the next `.dev0` after tagging.
20__version__ = "0.4.0.dev0"
21
22from . import adcp, io, madcp, tools