IOOS Metadata Profile Version 1.2

Revision History

Notes/Caveats

1. The IOOS Metadata Profile is a compound profile that builds off of the NOAA NCEI NetCDF Templates v2.0, meaning the full IOOS Metadata Profile is a combination of the NCEI Templates plus IOOS-specific guidance included in this document, such as:
   • attributes that are IOOS-specific (i.e. additions to the NCEI Templates)
   • attributes where variations exist between the IOOS guidance and the NCEI Templates (e.g. platform_vocabulary, where NCEI recommendations are specifically disallowed)
   • attributes with a different role in the NCEI Templates; for example, the attribute _FillValue is required by the NCEI Template; however, in the IOOS Profile it is listed as recommended only; conversely the opposite is possible as well
   • attributes with otherwise modified or further qualified meanings/definitions
2. The NCEI Templates in turn build off of the ACDD and CF conventions. Some attributes in the IOOS Profile originate from ACDD and CF (consult the Convention field in the table to determine the origin of each attribute). Links to each are below:
   • NOAA NCEI NetCDF Templates 2.0
   • Attribute Convention for Data Discovery 1.3
   • Climate and Forecast Conventions (CF) 1.7
3. In the IOOS Profile, attributes can be either required or recommended:
   • all required attributes must have meaningful values assigned to them in accordance with the rules prescribed by the corresponding Convention or Template.
   • each and all of the recommended attributes may be omitted; however, it is highly desirable that these attributes are included into the netCDF metadata AND have meaningful values assigned to them.
   • consult the Role field value in the table to determine whether each attribute is required or recommended

4. The IOOS Profile is geared towards datasets containing in situ observations encoded in multidimensional, self-describing formats such as netCDF. Some aspects of the profile are specific to the CF Discrete Sampling Geometries guidance for these types of data, in particular the Platform section. For datasets for which the DSG do not pertain (e.g. gridded model outputs or other gridded products), attributes in these sections can be omitted and still be compliant with the profile. To avoid over-complicating the documentation, these Platform-related attributes are listed as ‘required’ rather than ‘required, if applicable’ because they are required for DSG-compliant datasets that must include a sampling platform.

5. For in situ observation datasets, the IOOS Profile allows only one ‘platform’ per dataset. Please see the corresponding Platform section of the profile below for more information.

6. The U.S. IOOS National Glider Data Assembly Center currently uses a slightly different netCDF File Format (V2); work is in progress to harmonize the NGDAC File Format and IOOS Metadata Profile.

Gold Standard Example Datasets

IOOS provides a collection of “Gold Standard” example datasets in ERDDAP to demonstrate implementation of this Metadata Profile. The Gold Standard datasets can be used as templates for data providers to generate their own compliant datasets in ERDDAP, and include a fully-deployable ERDDAP instance that includes both the example data and configuration files. Consult the links below for more information:

• IOOS Metadata Profile “Gold Standard” Example Datasets
• IOOS “ERDDAP Gold Standard” GitHub Repository

In addition to the Gold Standard datasets, there are some in-line examples included below that highlight specific attribute use cases that comply with the profile.

IOOS Metadata Profile Attributes

Dataset Description

The attributes listed below are a collection of high-level attributes recommended largely by the parent conventions of the IOOS Metadata Profile (CF, ACDD). Their purpose is to describe the dataset in human-readable terms, define specific conventions used by other attributes (standard_name_vocabulary), or qualify other attributes (naming_authority). A high-quality dataset will generally include meaningful values for all of these attributes, even though they are not all required according to the profile.

Example

Taken from the Morro Bay BS1 MET Gold Standard Example dataset.

NC_GLOBAL {
    Conventions               CF-1.6, ACDD-1.3, IOOS-1.2
    featureType               TimeSeries
    id                        morro-bay-bs1-met
    infoUrl                   https://data.cencoos.org/#metadata/57163/station
    license                   The data may be used and redistributed for free but is not intended for legal use, since it may contain inaccuracies.  Neither the data Contributor, ERD, NOAA, nor the United States Government, nor any of their employees or contractors, makes any warranty, express or implied, including warranties of merchantability and fitness for a particular purpose, or assumes any legal liability for the accuracy, completeness, or usefulness, of this information.
    naming_authority          edu.calpoly.marine
    standard_name_vocabulary  CF Standard Name Table v72
    summary                   Timeseries data from 'Morro Bay - BS1 MET' (morro-bay-bs1-met)
    title                     Morro Bay - BS1 MET
}

Attribution

The attributes listed in the table below allow for consistent attribution of datasets within IOOS’ national products. Data providers are encouraged to follow these attribute guidelines exactly to ensure datasets appear with proper attribution.

Consult the Gold Standard Example Datasets for good examples to start from.

Example

Taken from the Morro Bay BS1 MET Gold Standard Example dataset.

NC_GLOBAL {
    creator_country          USA
    creator_email            marineops at calpoly.edu
    creator_institution      California Polytechnic State University, Center for Coastal Marine Sciences
    creator_institution_url  http://www.marine.calpoly.edu/
    creator_name             California Polytechnic State University, Center for Coastal Marine Sciences
    creator_sector           academic
    creator_type             institution
    creator_url              http://www.marine.calpoly.edu/
    institution              California Polytechnic State University, Center for Coastal Marine Sciences
}

NC_GLOBAL {
    contributor_email            cencoos_communications@mbari.org,feedback@axiomdatascience.com
    contributor_name             Central & Northern California Ocean Observing System (CeNCOOS),Axiom Data Science
    contributor_role             contributor,processor
    contributor_url              http://cencoos.org/,https://www.axiomdatascience.com
    contributor_role_vocabulary  https://vocab.nerc.ac.uk/collection/G04/current/
}

NC_GLOBAL {
    publisher_country      USA
    publisher_email        marineops at calpoly.edu
    publisher_institution  California Polytechnic State University, Center for Coastal Marine Sciences
    publisher_name         California Polytechnic State University, Center for Coastal Marine Sciences
    publisher_type         institution
    publisher_url          http://www.marine.calpoly.edu/
}

Variables

A collection of variable attributes that should be applied to all geophysical or other measured parameter variable contained in the dataset. This is mostly a re-listing and description of how to use CF convention attributes, with the addition of a few IOOS-specific attributes for variable precision/accuracy and standard name identification.

Note: the table below uses geophysical_variable as an alias intending to represent any variable containing geophysical data.

Example

Taken from the Morro Bay BS1 MET Gold Standard Example dataset.

Attributes {
    air_temperature {
        _FillValue        -9999.0
        missing_value     -9999.0
        long_name         Air Temperature
        platform          station
        standard_name     air_temperature
        units             degree_Celsius
        standard_name_url http://vocab.nerc.ac.uk/collection/P07/current/CFSN0023/
    }
}

Platform

The method for specifying platform metadata for in situ measurements has historically been a source of confusion. CF Discrete Sampling Geometries (DSG) guidelines represent observing platforms as ‘featureTypes’ (e.g. ‘timeSeries’, ‘profile’, ‘trajectory’), and allow multiple platforms to be included in a single file by way of a coordinate variable that uniquely identifies each feature by ID. For example, each buoy in a timeSeries feature dataset with multiple buoys can be differentiated by storing its identifier in this coordinate variable, which varies by the instance (or ‘station’ dimension in the case of timeSeries) according to the number of timeSeries features in the dataset. This feature ID coordinate variable is specified by the cf_role attribute - more info available in the table below.

More information about CF DSG and associated requirements is available in CF Documentation - Chapter 9.

Guidelines for Platforms in Datasets:

One platform per dataset: The IOOS Metadata Profile restricts datasets to contain only a single physical observing platform per dataset (e.g. buoy, glider, station, or anything that might be assigned a WMO ID). This simplifies the dataset structure significantly, and it allows the use of several global attributes to specify platform metadata that might otherwise be found in platform container variable(s) (e.g. platform_id and platform_name, as well as the OceanSITES-derived wmo_platform_code). It also simplifies the creation of aggregated datasets at the ERDDAP level to create groups of individual platform datasets.

Notes:

Data providers are encouraged to carefully review this section and upstream standards, and especially to review the Gold Standard example datasets to ensure compliance with platform guidance in this profile.

Aggregated ERDDAP datasets: ERDDAP provides the ability to create aggregated datasets from collections of individual datasets. While this is encouraged as a way to streamline access for end users to related datasets, IOOS will not harvest aggregated datasets into upstream national products, nor will NDBC harvest them for GTS publication. Therefore, you should set gts_ingest=false and ioos_ingest=false flags on any ERDDAP aggregated datasets.

Gridded Datasets: As mentioned in the Notes/Caveats section, the Platform rules do not apply to datasets that do not adhere to the CF Discrete Sampling Geometries guidelines (e.g. gridded model outputs or other gridded products). Attributes in this section can be omitted for such datasets and still be compliant with this profile.

DSG FeatureType Guidance:

Recommendations on the specific CF DSG featureTypes to use for different platform instrumentation scenarios is as follows:

TimeSeries - single station: A DSG TimeSeries feature represents measurements taken at a single location (lat, lon, altitude) over time. This can be distinguished from TimeSeries - multiple station by having a ‘station dimension’ of only 1, indicating only a single feature, or station, per dataset.

TimeSeries - multiple station: A DSG TimeSeries featureType dataset with a ‘station dimension’ of greater than 1. TimeSeries - multiple station represents a platform with sensors at different heights as a vertical collection of features where the altitude (depth or height) of each feature reflects the vertical position of each sensor on the platform. Latitude/Longitude positions must be the same for each feature.

TimeSeriesProfile - single station: A DSG TimeSeriesProfile feature represents a platform with multiple sensors as a vertical profile with a altitude (depth or height) position for each sensor. This can also be used for profiling sensors like ADCPs. This metadata profile restricts TimeSeriesProfile datasets to include a ‘station dimension’ of only 1, indicating a single timeSeries feature (or ‘station’) per dataset.

single-sensor-per-dataset: individual ERDDAP datasets of either TimeSeries or TimeSeriesProfile type that are associated via matching platform_id and wmo_platform_code global attributes, breaking a platform’s data apart into individual DSG datasets. Some IOOS RAs use this pattern already, and provide one dataset for sensor package (e.g., CTD, ADCP, met, etc). The code used to ingest sensor datasets for GTS by NDBC or for Sensor Map by IOOS will perform this aggregation, so data providers need not create aggregations in ERDDAP. Each individual dataset intended for the GTS must follow the Requirements for NDBC/GTS Ingest individually.

Note that because neither platform_id or wmo_platform_code attributes are required in this profile, any dataset without them will be considered to be specifying data for only one, distinct platform, and aggregation with other datasets by downstream processes at NDBC or IOOS is not guaranteed.

TrajectoryProfile: A DSG TrajectoryProfile feature represents a series of individual profiles taken along a path across the Earth’s surface. Similar to ‘TimeSeriesProfile - single station’ this metadata profile restricts TrajectoryProfile to include a ‘trajectory dimension’ of only 1, indicating a single trajectory feature (glider, ship, or other platform) per dataset.

Platform Attribution:

Quality Control/QARTOD

Guidance for implementing QARTOD quality control flag variables in a standardized fashion. QARTOD flag variables are associated with data variables using the CF “Ancillary Variables” approach. More information on this is available in CF Chapter 3.4.

This profile requires that all variables containing results from QARTOD tests be referenced by ancillary variables, and those variables use a valid CF Standard Name to reflect the test performed (see qartod_variable:standard_name in the table below for a list of suitable standard names).

Notes:

• See the Requirements for NDBC/GTS Ingest section below for important guidelines to properly specify a dataset’s QARTOD “Aggregate/Rollup” flag variable for GTS ingest by NDBC.
• The table below uses geophysical_variable as an alias intending to represent any variable containing geophysical data, and qartod_variable as an alias for an ancillary QC flag variable.

Example

Adapted from: Morro Bay BS1 MET Gold-Standard Example dataset.

  air_temperature {
    String standard_name "air_temperature";
    String long_name "Air Temperature";
    String units "degree_Celsius";
    String ancillary_variables "air_temperature_qc_agg air_temperature_gross_range air_temperature_flat_line";
  }
  air_temperature_qc_agg {
    String standard_name "aggregate_quality_flag";
    String long_name "Air Temperature QARTOD Aggregate Quality Flag";
    String flag_values 1, 2, 3, 4, 9;
    String flag_meanings "PASS NOT_EVALUATED SUSPECT FAIL MISSING";
    String references "http://services.cormp.org/quality.php";
    Int32 _FillValue 2;
  }
  air_temperature_gross_range {
    String standard_name "gross_range_test_quality_flag";
    String long_name "Air Temperature QARTOD Gross Range Test Quality Flag";
    String flag_values 1, 2, 3, 4, 9;
    String flag_meanings "PASS NOT_EVALUATED SUSPECT FAIL MISSING";
    String references "http://services.cormp.org/quality.php";
    Int32 _FillValue 2;
  }
  air_temperature_flat_line {
    String standard_name "flat_line_test_quality_flag";
    String long_name "Air Temperature QARTOD Flat Line Test Quality Flag";
    String flag_values 1, 2, 3, 4, 9;
    String flag_meanings "PASS NOT_EVALUATED SUSPECT FAIL MISSING";
    String references "http://services.cormp.org/quality.php";
    Int32 _FillValue 2;
  }

NDBC/GTS Ingest

A collection of variables that relate to requirements for IOOS datasets to be harvested by NDBC for ingestion and for delivery to the GTS. A description of the specifics of the NDBC/GTS ingest process used by IOOS is available in the Requirements for NDBC/GTS Ingest section below.

Example

Taken from PacIOOS’s AWS-HIMB ERDDAP Dataset.

NC_GLOBAL {
    gts_ingest "true";
}

air_temperature {
    String gts_ingest "true";
}

sea_water_temperature {
    String gts_ingest "true";
}

air_temperature_raw {
    String gts_ingest "false";
}

sea_water_temperature_raw {
    String gts_ingest "false";
}

IOOS Ingest

The ioos_ingest flag is intended to flag datasets that, in the rare case, should not be harvested by IOOS national products (such as the IOOS Catalog and Sensor Map). This allows data providers and ERDDAP operators the ability to exclude datasets that aren’t allowed for these public products on an individual basis. For example, an IOOS RA might host data owned by a private industry group who places sharing restrictions on their datasets. Note: RAs who are Certified RICEs must adhere to the NOAA data sharing policy and strive to make data publicly available, regardless of funding relationships with data providers. Therefore, this flag should be used for rare exceptions to this rule.

Example

Coming soon…

Instrument

The IOOS Metadata Profile generally follows the NCEI Templates guidance on usage of the instrument attribute and associated instrument_variables. The NCEI Templates define two different usages of the instrument attribute in a dataset:

1) as a global variable containing a vocabulary-constrained string describing the instrument type, or

2) as a variable attribute instrument that references an 'instrument_variable' by name that contains additional metadata, if needed.

The IOOS Metadata Profile defines specific attributes that can be attached to a dataset’s instrument_variables:

• attributes that describe the instrument details (e.g. calibration_date, make_model)

• attributes that allow compliance with the IOOS Convention for Asset Identification by further qualifying the resulting Asset Identifier for measured variables (e.g. component, discriminant)
  
  

Example

Usage of component with an instrument_variable:

Attributes {
    sea_water_temperature {
      instrument      temperature_sensor
    }
    temperature_sensor {
      component      nortek_adp_514
    }

}

Usage of discriminant with an instrument_variable:

Attributes {
    sea_water_temperature_top {
      instrument      temperature_sensor_top
    }
    sea_water_temperature_bottom {
      instrument      temperature_sensor_bottom
    }
    temperature_sensor_top {
      component      nortek_adp_514
      discriminant   top
    }
    temperature_sensor_bottom {
      component      nortek_adp_514
      discriminant   bottom
    }

}

Requirements for IOOS Dataset NDBC/GTS Ingest

In partnership with IOOS, NOAA NDBC ingests nonfederal IOOS partner data and delivers a subset of those variables (mainly meteorological and physical oceanographic) through the NWS system and on to the WMO Global Telecommunication System (GTS). NDBC also publishes all the data they harvest to their web products.

NDBC will leverage IOOS RA ERDDAP servers as the central access points for nonfederal IOOS partner data. This is a departure from the previous data delivery method, where providers and RAs packaged XML files and posted them to an NDBC FTP server. It is critical that RAs work with their regional data providers to ensure their data are available in the RA ERDDAPs, so that NDBC can continue to access those datasets.

In order to allow NDBC to query and filter the correct subset of datasets in an ERDDAP server and process them consistently and predictably, data providers must ensure the following dataset attribution requirements are met.

Refer to the Platform, NDBC/GTS Ingest, and Quality Control/QARTOD tables above for detailed descriptions of the attributes shown below.

For NDBC to pull data for a dataset to the GTS:

1. The dataset should be in ERDDAP
2. The dataset should meet the Single Platform requirement described in the Platform section
3. The dataset should meet requirements for defining the vertical coordinate variable set forth in the Requirements for Vertical Coordinate Variable section
4. The dataset should have a global attribute called wmo_platform_code, with the numeric WMO ID or alphanumeric NWS ID as the value (see wmo_platform_code in the Platform section for details on specific ID requirements)
5. The dataset should have a global attribute called gts_ingest with a value of true
6. Any variables the RA wants to push to NDBC should have an attribute called gts_ingest with value of true
7. The variable should have a standard_name attribute with a value that’s a valid CF Standard Name. Consult the spreadsheet in the XML to CF Standard Name Mappings section below for Standard Name mappings for specific variables.
8. The variable should include an ancillary variable representing the QARTOD aggregate flag as defined in the Requirements for the QARTOD Aggregate/Rollup Flag section
9. The variable should have a units attribute, with a value that’s a valid unit (that is, the units are convertible to the CF canonical unit using the udunits library)
   
   

Note: it is not a requirement for a variable’s QC flag ancillary variables to include a gts_ingest flag.

XML to CF Standard Name Mappings: Mappings between the elements, or tags, of the XML file format used for GTS dissemination and the corresponding CF Standard Names to use in ERDDAP variables are found in the spreadsheet below. Note that the file includes additional attribution guidance for certain variables (e.g. salinity) and information about default values that NDBC uses for some elements in derived XML files that RAs typically will not need to address in their own ERDDAP datasets.

Requirements for Vertical Coordinate Variable:

In order for NDBC to be able to uniformly identify and interpret the vertical coordinate variable in IOOS datasets to ensure consistent processing, a specific implementation of the CF guidelines for vertical coordinate variables is necessary. This profile requires the vertical coordinate variable to include the following attributes:

Note: The vertical position of an individual observation must always be encoded using the vertical coordinate variable (as specified in the CF DSG guidelines) and cannot be included as an attribute on a data variable.

Requirements for the QARTOD Aggregate/Rollup Flag:

Currently, IOOS RAs push a custom XML format to NDBC for GTS ingestion, so they can exclude “QC fail” values while generating the XML. ERDDAP datasets will have all observed data, including observations marked “QC fail”, so NDBC needs a means to filter QC passing and failing data from the full timeseries. This will be accomplished using an “ancillary variable” according to the CF Ancillary Data guidelines that represents the QC aggregate/rollup flag for each observation variable marked for GTS ingestion, as described below.

Rules governing the ‘Aggregate/Rollup’ flag variable:

1. The value of the variable is the UNESCO/QARTOD v2 convention:
   • 9 = Missing
   • 2 = Not Eval
   • 1 = Pass
   • 3 = Suspect
   • 4 = Fail
2. The value of the rollup flag should be the worst result out of all of the individual tests. It’s called a “Summary Flag” in the QARTOD Data Flags manual (pg 3).
   • Here’s how it’s done in the ioos_qc library with the corresponding test.
3. The variable should have a standard name attribute of aggregate_quality_flag. See the QARTOD section above for more information.
4. NDBC should exclude any values that are QC fail (and missing) but include everything else (not eval, pass, suspect)
   
   

Notes on NDBC/GTS ingest and QC flagging:

• Although having gts_ingest on both the dataset itself and each of its variables is redundant, it allows for more efficient querying across a large ERDDAP server
• If you set gts_ingest on a variable that NDBC doesn’t currently accept, NDBC will just ignore it.
• RAs may publish ancillary variables with results of individual QC tests, however NDBC will only examine contents of the aggregate_quality_flag variable for filtering purposes for GTS harvest. The CF Standard Names table includes several names appropriate for individual QC tests - consult the QARTOD section for the current list.

Examples

The WQB_04 and WQB_05 PacIOOS buoys are actively used by NDBC for collecting data.

Rules for IOOS Asset Identifier Generation

For reference, the IOOS Asset Identifier 1.0 is composed of the following components:

Asset Identifier = urn:ioos:asset_type:authority:label[:component][:discriminant][#functional_parameters]`

The IOOS Asset Identifier can be constructed by combining the Metadata Profile attributes according to their definitions to match the above pattern.

Rules for generating the Asset Identifier from IOOS Metadata Profile attribution:

If platform_id present:

Asset Identifier = 'urn:ioos:' + platform + ':' + naming_authority + ':' + platform_id

If platform_id not present:

Asset Identifier = 'urn:ioos:' + platform + ':' + naming_authority + ':' + id

For datasets with a wmo_platform_code global attribute, a second Asset Identifier is also assumed, according to the following rules:

WMO Asset Identifier = 'urn:ioos:' + platform + ':wmo:' + wmo_platform_code

This means it is possible for a single dataset to have two valid IOOS Asset Identifiers. It is also possible to define as many custom global ‘id’ attributes as necessary. For example, RAs may define their own identification scheme outside those described here, or, if there is another organization (such as NCEI) for which an id is appropriate, it may be included as well (e.g. ncei_id = 123456789 or ncei_accession = 123456789). While these won’t technically be considered to be IOOS Asset Identifiers, there is still the opportunity to identify datasets using community-agreed-upon global identifier attributes.

Additional considerations regarding IOOS Asset Identifiers:

• At the moment, the specification constrains the valid asset_types to be: glider, station, network, sensor, and survey. For the purposes of the IOOS Metadata Profile, the values of `platform’ can be more broad, however the resulting Asset Identifiers may not be valid according to that specification.

• If an instrument_variable is defined for the dataset, it becomes possible for each data variable associated with it (via the :instrument variable attribute) to have its own Asset Identifier that is more fully qualified than the global-variable-defined dataset Asset Identifier described above. See the documentation for instrument_variable:component and instrument_variable:discriminant for details.

IOOS Asset Identifier rules for a dataset with variables that include an instrument_variable reference:

Asset Identifier = 'urn:ioos:' + platform + ':' + naming_authority + ':' + platform_id + ':' + instrument_variable:component + ':' + instrument_variable:discriminant