$Header$ -*-text-*-

netCDF Operators NCO version 5.3.3 split the uprights

http://nco.sf.net (Homepage, Mailing lists, Help)
http://github.com/nco/nco (Source Code, Issues, Releases)

What's new?
Version 5.3.3 adds IEEE floating-point exception testing to ncks.
All operators include corner-case quantization fixes for strange
numbers (-0.0, NaN) that are also now in the netCDF-C Library.
ncclimo now only tries to create regional timeseries statistics
on variables that are single-level horizontal lat/lon/ncol fields.
ncremap supports interpolation of pure-pressure to hybrid
sigma-pressure coordinates with distinct source and destination
surface pressure fields. And ncrename adds some words of caution.
Skip this release if these changes do not interest you. 

Did you realize that the first public release of ncks was May 2, 1995?
https://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/1995/msg00069.html
Happy 30th birthday ncks!!!

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. The new ncks --fpe option reports the status of all IEEE Floating
Point Exceptions (FPEs) after execution. This can be useful to verify
whether input datasets contain strange numbers that could be
arithmetically processed, and to ensure the integrity of NCO
arithmetic. 

% ncks --fpe -O -7 -L 1 -C -v ppc_zro_ngt_nan_flt --qnt_alg=gbr \
       --qnt default=3 --cmp='shf|zst' in.nc out.nc
ncks: Successfully set FE_INVALID|FE_DIVBYZERO|FE_OVERFLOW exception flags
ncks: FE_DIVBYZERO is not raised
ncks: FE_INEXACT is raised
ncks: FE_INVALID is not raised
ncks: FE_OVERFLOW is not raised
ncks: FE_UNDERFLOW is not raised

http://nco.sf.net/nco.html#fpe
http://nco.sf.net/nco.html#ncks

B. All operators can now quantize -0.0 (a legal number) without
triggering any IEEE Floating Point Exceptions (FPEs). Other weird
numbers, like NaN, already avoided FPEs. Moreover, the necessary
changes to avoid FPEs have been merged upstream into the quantization
algorithms in libnetcdf version 4.10-development. Thanks to Jeff
Whitaker (NOAA) for bringing this to our attention, to M. Garcia
for the original bug report, and to Ward Fisher for merging.
http://nco.sf.net/nco.html#qnt

C. When no explicit variable extraction list is specified in
timeseries mode, ncclimo automatically creates timeseries for all
input variables with at least two dimension. However, when regional
statistics are also requested with --rgn_stt, these statistics are
only meaningful for spatial timeseries, i.e., single level fields with
latitude, longitude, and time. Under these circumstances ncclimo now
only creates regional statistics for single level spatial timeseries.
Previously it would try (and fail) to generate regional statistics 
for multi-level or otherwise inappropriate rank >=2 variables.

ls caseid*.nc | ncclimo --split --rgn_avg -s 2013 -e 2014 -o foo

http://nco.sf.net/nco.html#ncclimo

D. Previous versions of NCO could only interpolate ECMWF/IFS
data to hybrid sigma-pressure coordinates when the input
surface pressure (lnsp) was for the output as well. Now NCO
can interpolate to a distinct output surface pressure that
is storeed in the vertical template file (a so-called "fat"
template that includes the surface pressure, rather than a
"skinny" template that does not).

ncremap --ps_nm=lnsp --vrt_fl=vrt.nc in.nc out.nc

http://nco.sf.net/nco.html#vrt
http://nco.sf.net/nco.html#ncremap

E. ncks --chk_map now prints the minimum and maximum angular
and horizontal resolution of the source and destination grids.
The angular resolution is estimated as the square-root of the area
(in sr) of the smallest/largest gridcells, and is expressed in
degrees. The spatial resolution is estimated 
This is the angle subtended by a spherical cap with the same area
as the gridcell.

% ncks --chk_map map_ne30pg2_to_cmip6_180x360_ncoaave.20231201.nc
...
area_a min, ~dx: 4.9093515660647374e-04 sr, ~141.14 km, ~1.27 degrees in grid A cell [14462,-45.7267,+177.69]
area_a max, ~dx: 6.8523331840188574e-04 sr, ~166.75 km, ~1.50 degrees in grid A cell [1858,+0.749936,+359.25]
...
area_b min, ~dx: 2.6580860638549116e-06 sr, ~10.39 km, ~0.09 degrees in grid B cell [60,-89.5,+59.5]
area_b max, ~dx: 3.0460968486221404e-04 sr, ~111.18 km, ~1.50 degrees in grid B cell [32272,-0.5,+231.5]
...

This feature can be useful in assessing the uniformity of grids, and
http://nco.sf.net/nco.html#chk_map
http://nco.sf.net/nco.html#ncks

F. ncrename now WARNs users attempting to rename coordinates in
netCDF4 files. The WARNING, which prints only when debugging level
> 0, the input file is in netCDF4 format, and the user tries to
rename a variable or dimension, is self-explanatory:

zender@spectral:~$ ncrename -D 1 -O -v lat,latitude in.nc out.nc
ncrename: WARNING Renaming variables, dimensions, and attributes in any netCDF3-format file (i.e., classic, 64-bit, and CDF5) ALWAYS works. Moreover, renaming groups and attributes in any netCDF4-format file always works. However, these re-assuring notes precede a discomfitting message: A longstanding problem in the netCDF library (it is not an NCO issue per se) sometimes crashes ncrename or, even worse, silently corrupts data in the output file when input file is netCDF4-format. These outcomes are only known to occur when attempting to rename variables to/from the same names as dimensions (i.e., coordinate variables), and or dimensions to/from the same names as variables. A long thread describing the status of this issue since 2017 is at https://github.com/Unidata/netcdf-c/issues/597. A chronology of the issue dating back to 2007 is at https://nco.sourceforge.net/nco.html#bug_nc4_rename.
ncrename HINT: Do not expect this issue to be fixed in the near future :)
ncrename HINT: There is a straightforward two- or three-step workaround that ALWAYS works: 1. Translate the file to any netCDF3 format (e.g., with "ncks -5 in.nc out.nc"). 2. Perform the renaming. 3. (Optional, only necessary if you wish to recover the netCDF4 features like compression) Translate back to a netCDF4-format (e.g., with "ncks -7 in.nc out.nc").
ncrename: In total renamed 0 attributes, 0 dimensions, 0 groups, and 1 variable
zender@spectral:~$ ncrename -O -v lat,latitude in.nc out.nc
zender@spectral:~$ ncrename -D 1 -O -a lat@units,meaning in.nc out.nc
ncrename: In total renamed 1 attribute, 0 dimensions, 0 groups, and 0 variables
zender@spectral:~$ ncrename -D 1 -O -v lat,latitude in3.nc out.nc
zender@spectral:~$ 

The second and third and fourth examples show no WARNING output
because the debugging level is the default (0) in the second, and
because the user is not renaming a variable or dimension in the
third, and because the input file is netCDF3-format in the fourth.
The decision was made to turn-on this WARNING only in a non-default
debug level to avoid annoying users with existing workflows that
expect no text output from ncrename. This compromise is intended
to prevent widespread panic and mass hysteria. Thanks to Toby Searle
(UKMO) for prompting this change.

BUG FIXES:
   
A. Previous versions of NCO could omit storing the attributes
for the frc_column and frc_landunit variables created by the S1D
feature. This is fixed. There is no workaround. The solution is
to upgrade
http://nco.sf.net/nco.html#s1d

Full release statement at http://nco.sf.net/ANNOUNCE
    
KNOWN PROBLEMS DUE TO NCO:

This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems. 
These problems occur with NCO 5.2.7 built/tested under
MacOS 15.3.1 with netCDF 4.9.4-development on HDF5 1.14.3
and with Linux FC40 with netCDF 4.9.2 on HDF5 1.14.3.

A. NOT YET FIXED (NCO problem)
   Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments

   Demonstration:
   ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)
   ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions

   Demonstration:
   ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

   20140826: Verified problem still exists
   20140619: Problem reported by rmla
   Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
   Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
   Specifying strided hyperslab on large netCDF4 datasets leads
   to slowdown or failure with recent netCDF versions.

   Demonstration with NCO <= 4.4.5:
   time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
   Demonstration with NCL:
   time ncl < ~/nco/data/ncl.ncl   
   20140718: Problem reported by Parker Norton
   20140826: Verified problem still exists
   20140930: Finish NCO workaround for problem
   20190201: Possibly this problem was fixed in netCDF 4.6.2 by https://github.com/Unidata/netcdf-c/pull/1001
   Cause: Slow algorithm in nc_var_gets()?
   Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
   Workaround #2: Convert file to netCDF3 first, then use stride
   Workaround #3: Compile NCO with netCDF >= 4.6.2

B. NOT YET FIXED (netCDF4 library bug)
   Simultaneously renaming multiple dimensions in netCDF4 file can corrupt output

   Demonstration:
   ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc # Completes but produces unreadable file foo.nc
   ncks -v one ~/foo.nc

   20150922: Confirmed problem reported by Isabelle Dast, reported to Unidata
   20150924: Unidata confirmed problem
   20160212: Verified problem still exists in netCDF library
   20160512: Ditto
   20161028: Verified problem still exists with netCDF 4.4.1
   20170323: Verified problem still exists with netCDF 4.4.2-development
   20170323: https://github.com/Unidata/netcdf-c/issues/381
   20171102: Verified problem still exists with netCDF 4.5.1-development
   20171107: https://github.com/Unidata/netcdf-c/issues/597
   20190202: Progress has recently been made in netCDF 4.6.3-development
   More details: http://nco.sf.net/nco.html#ncrename_crd

C. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable

   20130724: Verified problem still exists. 
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
   NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

D. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
   ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails 

"Sticky" reminders:

A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., 
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

B. Pre-built executables for many OS's at:
   http://nco.sf.net#bnr