/ Announce / Developer / Discussion / Help / Manual / Project / Source / [ACME] [ANTLR] [CF] [DIWG] [GSL] [netCDF] [OPeNDAP] [SWAMP] [UDUnits]


Bienvenue sur le netCDF Operator (NCO) site

Current stable NCO version is 4.6.3 released

The NCO toolkit manipulates and analyzes data stored in netCDF-accessible formats, including DAP, HDF4, and HDF5. It exploits the geophysical expressivity of many CF (Climate & Forecast) metadata conventions, the flexible description of physical dimensions translated by UDUnits, the network transparency of OPeNDAP, the storage features (e.g., compression, chunking, groups) of HDF (the Hierarchical Data Format), and many powerful mathematical and statistical algorithms of GSL (the GNU Scientific Library). NCO is fast, powerful, and free.


Recent Releases & Milestones


What is NCO?

The netCDF Operators (NCO) comprise about a dozen standalone, command-line programs that take netCDF, HDF, and/or DAP files as input, then operate (e.g., derive new data, compute statistics, print, hyperslab, manipulate metadata) and output the results to screen or files in text, binary, or netCDF formats. NCO aids analysis of gridded scientific data. The shell-command style of NCO allows users to manipulate and analyze files interactively, or with expressive scripts that avoid some overhead of higher-level programming environments.

Traditional geoscience data analysis requires users to work with numerous flat (data in one level or namespace) files. In that paradigm instruments or models produce, and then repositories archive and distribute, and then researchers request and analyze, collections of flat files. NCO works well with that paradigm, yet it also embodies the necessary algorithms to transition geoscience data analysis from relying solely on traditional (or “flat”) datasets to allowing newer hierarchical (or “nested”) datasets.

The next logical step is to support and enable combining all datastreams that meet user-specified criteria into a single or small number of files that hold all the science-relevant data organized in hierarchical structures. NCO (and no other software to our knowledge) can do this now. We call the resulting data storage, distribution, and analysis paradigm Group-Oriented Data Analysis and Distribution (GODAD). GODAD lets the scientific question organize the data, not the ad hoc granularity of all relevant datasets. The User Guide illustrates GODAD techniques for climate data analysis:

Note that the “averagers” (ncra and ncwa) are misnamed because they perform many non-linear statistics as well, e.g., total, minimum, RMS. Moreover, ncap2 implements a powerful domain language which handles arbitrarily complex algebra, calculus, and statistics (using GSL). The operators are as general as netCDF itself: there are no restrictions on the contents of input file(s). NCO's internal routines are completely dynamic and impose no limit on the number or sizes of dimensions, variables, and files. NCO is designed to be used both interactively and with large batch jobs. The default operator behavior is often sufficient for everyday needs, and there are numerous command line (i.e., run-time) options, for special cases.


How to Contribute: Volunteer, Endorse, or Donate

NCO has always (since 1995) been GPL'd Open Source. SourceForge.net started hosting NCO in March, 2000. This facilitated collaboration, code contributions, and support.

In March, 2015, NCO development moved to GitHub.com. We continue to use the SourceForge discussion fora for historical continuity (seventeen years of searchable Q&A). No matter what your programming background there is a task you can help with. From re-organizing the TODO list itself, to improving this cheesy webpage, to documentation, to designing and implementing new features and interfaces, we need your help! The project homepage contains mailing lists, discussion forums, and instructions to make contributing to NCO easy.

Many users feel unable to volunteer their time. An equally effective contribution in the long-run would be your endorsement, which may make the difference between a declined and an accepted proposal. An endorsement can be a few sentences that describes how NCO benefits your work or research. E-mail your endorsement to Charlie “my surname is zender” Zender with Subject: “NCO Proposal Endorsement”. This information is useful advocating for more NCO support. “What future proposals?” you ask, “Aren't you already funded?” Yes, in 2012 NASA funded us to implement netCDF4 groups and HDF support, and in 2014 NASA funded us to improve SLD handling. These funds are/were primarily for development and maintainance of specific features. To realize our grander ambition, i.e., to shift geoscience data analysis from a flat- to a hierarchical-paradigm (GODAD), will require a sustained effort and software ecosystem that understands and implement hierarchical dataset concepts. And it's hard to sell a federal agency on the wisdom of investing in a paradigm shift! Other more prosaic tasks that need work are, for example, I/O parallelization (!!!), user-defined and non-atomic types, more CF conventions, cloud-services, JSON back-end, and user-defined ncap2 functions. If you send an endorsement, please include (at least) your Name, Title, and Institutional affiliation.

Lastly, as of June, 2003, if you're more strapped for time than money and want to contribute something back, consider a monetary donation. This may incentivize us to tackle your favorite TODO items.
Inspired by President Obama's plan to bring more transparency to government investment, these homepage donation counters track the influence of your monetary donations on NCO development:
Donations received between 20030624 and 20141126: US$144.55. Thank you, donors!
NCO features “incentivized” by these donations: More emoticons in the documentation :)

NSF EarthCube Project

The National Science Foundation Grant NSF ICER-1541031 funded the Unidata-led EarthCube Project, “EarthCube IA: Collaborative Proposal: Advancing netCDF-CF for the Geoscience Community” from 20150901–20170831 as part of the Integrative and Collaborative Education and Research (ICER) program. This URL, http://nco.sf.net#prp_e3, points to the most up-to-date information on the EarthCube proposal.

UCI's primary role is to help extend CF to cover hierarchical data structures, aka groups. Groups are the Group-Oriented Data Analysis and Distribution (GODAD) paradigm we are developing for geoscience data analysis. We will convene workshops for interested stakeholders in 2016 and 2017. Be on the lookout for announcements!


DOE ACME Project

“Lightweight Climate Analysis Tools for ACME” is a US Department of Energy Cooperative Agreement (CA) DE-SC0012998 from 20141215–20171214 that funds our contribution to the Accelerated Climate Modeling for Energy (ACME) project, a part of DOE's Earth System Modeling (ESM) program. The ACME project provides the resources to implement support for parallel regridding and workflows in NCO accessible through UV-CDAT. Spatially intelligent, parallelized analysis tools are a key component of the Group-Oriented Data Analysis and Distribution (GODAD) paradigm we are developing for geoscience data analysis.


NASA ACCESS Project

The National Aeronautics and Space Administration (NASA) Cooperative Agreement NNX14AH55A funds our project, “Easy Access to and Analysis of NASA and Model Swath-like Data” from 20140701–20160630 as part of the Advancing Collaborative Connections for Earth System Science (ACCESS) program. This URL, http://nco.sf.net#prp_axs, points to the most up-to-date information on the ACCESS 2013 project.

This ACCESS project provides resources to implement support in NCO for Swath-like data (SLD), i.e., dataset with non-rectangular and/or time-varying spatial grids in which one or more coordinates are multi-dimensional. It is often challenging and time-consuming to work with SLD, including all NASA Level 2 satellite-retrieved data, non-rectangular subsets of Level 3 data, and model data (e.g., CMIP5) that are increasingly stored on non-rectangular grids. Spatially intelligent software tools is a key component of the Group-Oriented Data Analysis and Distribution (GODAD) paradigm we are developing for geoscience data analysis.

We are currently recruiting a programmer (aka software engineer) or postdoc based at UCI for at least two years, to accomplish our ACCESS objectives. As described in the proposal, this person will be responsible for incorporating geospatial features and parallelism into NCO. See the ads for more details. (PDF, TXT).


Publications and Presentations


Release Highlights

Stable releases receive unique tags and their tarballs are created and stored here at GitHub. Identical copies of those tarballs are also stored here on SourceForge for historical continuity. You may retrieve the source of tagged versions directly with, e.g., git clone -b 4.6.3 http://github.com/nco/nco.git nco-4.6.3.

Pre-built Executables

Pre-built binary executables are available for many platforms. Our source tarballs are always up-to-date, and work on our development systems (Fedora, Ubuntu, and Mac OS X). We also attempt to provide (theoretically) platform-independent sources in the most common Linux package formats (Debian and RPM). Below are one-line installation instructions and links to these and to packages for other platforms created by volunteers. Anyone willing to perform regular regression testing and porting of NCO to other platforms is welcome. Previous versions of these executables are still available by searching the directory index here.

AIX on IBM mainframes

Anaconda

Debian and Ubuntu GNU/Linux

Fedora, RedHat Enterprise Linux (RHEL), and Community ENTerprise Operating System (CentOS) GNU/Linux

Gentoo GNU/Linux

Mac OS X/Darwin

Microsoft Windows (native build, compiled with Visual Studio 2015, use this if unsure)

These native Windows executables should be stand-alone, i.e., not require users to have any additional software. This is a new feature as of 20120615, please send us feedback. To build NCO from source yourself using MSVC or Qt, please see the NCO Qt/MSVC build page.

Microsoft Windows (running Cygwin environment, compiled with GNU-toolchain)

Before using, first install curl (in the "Web" category of Cygwin Setup), and set export UDUNITS2_XML_PATH='/usr/local/share/udunits/udunits2.xml' (or wherever udunits2.xml is installed). Thanks to Mark Hadfield and Pedro Vicente for creating Cygwin tarballs. Thanks to Cygnus Solutions and RedHat Inc. for developing and supporting Cygwin over the years.

Python Bindings Source and Documentation

Documentation and Users Guide

The NCO Users Guide is available for reading in these formats:

nco.texi is the most up-to-date. Files nco.dvi, nco.ps, and nco.pdf contain all the mathematical formulae (typeset with TeX) missing from the screen-oriented formats. The screen-oriented formats—nco.html, nco.info, nco.txt, and nco.xml—contain all the documentation except the highly mathematical sections. Wenshan Wang of UCI contributed a Quick Reference Card (last updated ) suitable for printing, framing, and/or carrying on your person at all times.

Other documentation:


FAQ: Frequently Asked Questions

These questions show up almost as frequently as my mother. But they are more predictable:


Help/Support/Contacts:

If you have support questions or comments please contact us via the Project Forums (rather than personal e-mail) so other users can benefit from and contribute to our exchange. Let us know how NCO is working for you—we'd like to hear. Have you read the documentation and browsed the forums to see if your question/comment has been reported before? Please read the Guide's suggestions for productive Help Requests and Bug Reports.


ANNOUNCE/ChangeLog/README/TODO

Files containing useful information about the current distribution:


Source Code

The simplest way to acquire the source is to download the compressed tarball:

The best way to acquire the source and occasionally update to the latest features is with Git. The browsable Repository contains up-to-the-minute sources and is the easiest way to stay synchronized with NCO features. Updating NCO source requires some familiarity with development tools, especially Git and Make. You may retrieve any NCO distribution you wish from GitHub. Usually you wish to retrieve a recent tagged (i.e., released) version. This command retrieves the entire NCO repository (< 20 MB) and then checks out NCO version 4.6.3:

git clone https://github.com/nco/nco.git;cd nco;git checkout 4.6.3

These commands retrieve the current (“bleeding edge”) development version of NCO into a local directory named nco:

git clone https://github.com/nco/nco.git ~/nco

or

git clone git@github.com:/nco/nco.git

, Track changes to the development version using

cd nco;git pull

One difference between running a "tagged" release (e.g., 4.6.3) and the development version is that the tagged release operators will print a valid version number (e.g., 4.6.3) when asked to do so with the -r flag (e.g., ncks -r). The development version simply places today's date in place of the version. Once the autotools builds are working more robustly, the confusion over versions should largely disappear.

Developer NCO Source Documentation

Automated source documentation, created by the Doxygen tool is available. Some developers find this documentation helpful, as it can clarify code and data relationships in the code.

The Doxygen documentation is infrequently (i.e., never since Daniel left) updated.


Compilation Requirements

Still having trouble building NCO from source? Read these Build Hints

Using NCO at UCI, NCAR, and other High Performance Computing Centers (HPCCs)

HPCCs unfortunately do not utilize modern package systems like RPMs or .debs, or do so on old OSs with no access to newer RPMs and .debs. Institution-supported executables are usually available with module load nco. These stable releases are often many versions (up to two years) old. Thanks to funding from external grants,

DOE, NCAR, and UCI HPCC users may find more recent pre-built NCO executableses the personal directories shown below. These are usually built from a recent tagged-version of NCO (e.g., 4.6.X-alphaY) not from the “bleeding-edge” of master which is usually untagged. One way to use these pre-built executables is to prepend them to your executable and library search paths, e.g., export PATH="~zender/bin:${PATH}", export LD_LIBRARY_PATH="~zender/lib:${LD_LIBRARY_PATH}"


Known Problems from 2013 (version 4.2.4) Onwards

  • Known Problems through 2012 (version 4.2.3)

  • People:

    Current Developers (please contact us via the project forums not via email):

    Alumni Developers:


    Get NCO netCDF Operators at SourceForge.net. Fast, secure and Free Open Source software downloads
    / Announce / Developer / Discussion / Help / Manual / Project / Source / [ACME] [ANTLR] [CF] [DIWG] [GSL] [netCDF] [OPeNDAP] [SWAMP] [UDUnits]