U.S. IOOS Service Registration Process · Guidelines for IOOS DMAC Metadata Publication on NGDC Geoportal and IOOS Catalog Smiley face

IOOS Service Registration Process Guidelines

The registration of the IOOS services is a process that allows the wide range of various clients to efficiently discover U.S. IOOS data. Currently, all U.S. IOOS DMAC services must be registered with the Service Registry (“Registry”), which is hosted and operated by the NGDC.

The Registry is the official list of service URLs that provide access to U.S. IOOS data through DMAC-compliant services. The Registry performs the following operations to support the registration process:

  1. harvesting metadata from submitted URLs of the DMAC services or Web Accessible Folders (WAFs);
  2. transforming metadata to ISO, generate and maintain WAFs for the ISO metadata records;
  3. injecting the ISO metadata into NGDC Geoportal.

Upon completion of the Registry’s operations, the IOOS data can be discovered through the Web-based NGDC Geoportal interface.

The primary client for the service registry is the IOOS Catalog. The IOOS Catalog fetches metadata from the Geoportal through CSW interface. For more about the Catalog, go to the catalog repository on github for information and code supporting the generation of the catalog.

This repository is the primary source for documentation for the registration process and contains steps, process descriptions, and examples.

The IOOS Service Registration Webinar was presented by Anna Milan from NGDC, and recorded in February 2014.

Steps for registering IOOS Service Metadata

The following resources can be registered with the Registry (Figure 1):

NOTE: The individual catalogs must be registered; a catalog that just points to other catalogs cannot be registered — although the catalog tree is crawled for all child datasets, the other catalogs referenced by CatalogRef are ignored.

NOTE: WAFs offer the most control over the metadata that will appear in the registry, but require some efforts to create and maintain.


Registration process              Figure 1: Proposed IOOS Registry and Catalog Architecture

1. Submitting URL of a Service or WAF to the Collection Source Table

The registration process starts with submitting a request on the issue tracker in the IOOS Registry Github Repository.

For each request, the following information should be included:

The URL submitted is manually added to the Collection Source Table, and the service is listed as “submitted”.

2. Harvesting and Converting Metadata

3. Publishing Metadata

  • On day 3 (a day after the status has been changed to “approved “) by 07:15 MT/ 09:15 ET, the ISO metadata records are automatically posted to a production WAFs in EMMA.
  • The NGDC Geoportal automatically harvests ISO records from the production WAFs by 09:00 MT/ 11:00 ET.

The Service Registration process ends when a valid ISO record has been created and added to the production WAF. The metadata should be able to be found in the NGDC Geoportal.

4. Harvesting Metadata by the IOOS Catalog

The IOOS Catalog queries the NGDC Geoportal daily for a list of services registered to IOOS. These services are then individually queried by the Catalog and their metadata is harvested and indexed.

Briefly, the current (03 October 2014) harvest schedule for the catalog is:

  • harvests start nightly at 7:10am UTC (2:10am eastern)
  • cleanups start nightly at 8:10am UTC (3:10am eastern)
  • reindex daily daily at 6:30am UTC (1:30am eastern)
  • daily status emails at 6:20am UTC (1:20am eastern)

See the Catalog documentation for more information on what is indexed and how.

Registration Process FAQ

How do I check to see if my metadata has been registered?

How do I update or remove metadata in the service registry?

The registry augments today’s harvest with all previous harvests and sometimes this can result in old out-of-date records that are no longer applicable.

  1. If the content of the service has changed, but the service should still be registered then:

    • Create an issue in the github IOOS/ Registry repository or send an email to ioos.catalog@noaa.gov requesting a ‘clean out’ of a particular service or WAF.
    • The NGDC administrator will manually set a flag to remove ALL previous metadata records before harvest. This will result in an entirely new refresh of the content for that web accessible folder (WAF).

  2. If the service is outdated, and should no longer be registered with IOOS then:

    • Send an email to ioos.catalog@noaa.gov requesting that the service be removed.
    • The service status will be changed to ‘For Removal’.
    • The NGDC administrator will manually set a flag to remove ALL previous metadata records before harvest. This will result in an entirely new refresh of the content for that web accessible folder.
    • The NGDC admin will change the service status to ‘Removed’.

How do I register a THREDDS catalog that contain catalogRef elements?

In THREDDS catalogs it is common to use catalogRef elements to reference other catalogs, especially in the top level catalog. The NGDC crawler is currently not following catalogRef links, however. So if you had a top level catalog that looked like this:

<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
    xmlns:xlink="http://www.w3.org/1999/xlink"
    name="THREDDS Top Catalog, points to other THREDDS catalogs" version="1.0.1">

    <dataset name="NCSU MEAS THREDDS catalogs">
        <catalogRef xlink:href="gomtox_catalog.xml" xlink:title="GOMTOX (Gulf of Maine) Ocean Model" name=""/>
        <catalogRef xlink:href="sabgom_catalog.xml" xlink:title="SABGOM (South Atlantic Bight and Gulf of Mexico) Ocean Model" name=""/>
    </dataset>

</catalog>

and you wanted all the content to be harvested, you should not submit the top level catalog (which would harvest nothing).

So, if a top level URL looks something like

then Instead of that top level URL, you should submit the child catalog URLs for harvesting. An example of a child catalog URL would be like this

Where can I find the UUIDs of the registered metadata collections?

To search for metadata on Geoportal, you must submit the metadata collection sys.siteuuid parameter for each query. The Master UUID listing for Geoportal metadata collections (e.g. RAs) is located in IOOS/Registry repository on GitHub.

How can I search for the registered metadata on the NGDC Geoportal?

The metadata stored in the WAFs can be searched through via either the NGDC Geoportal’s REST API or OGC Catalog Service for Web interface. Below are selected examples of the search queries that can be issued to the NGDC Geoportal to search for IOOS Regional Association assets.
Additional information on the Geoportal in general can be found on the ESRI Sourceforge site

Search via Geoportal REST interface

Date Search Example

Search within the PacIOOS collection for records with start date 2009-02-01 to end date 2012-02-01 with JSON response:

PacIOOS Date Search Example Link

Decoded Parameters:

rid=local&ridName=NOAA's Geophysical Data Center&searchText=startDate:[1800-01-01 TO 2012-02-01] AND endDate:[2009-02-01 TO 2100-01-01] AND sys.siteuuid:"{68FF11D8-D66B-45EE-B33A-21919BB26421}"&start=1&max=10&orderBy=relevance&maxSearchTimeMilliSec=10000&f=pjson

Keyword Example

Search within the PacIOOS WAF for keywords sea_water_salinity with JSON response:

PacIOOS Keyword Example Link

Decoded Parameters:

rid=local&ridName=NOAA's Geophysical Data Center&searchText=keywords: sea_water_salinity AND sys.siteuuid:"{68FF11D8-D66B-45EE-B33A-21919BB26421}"&start=1&max=1000&orderBy=relevance&maxSearchTimeMilliSec=10000&f=pjson

Geographic Search Example

Search within the PacIOOS WAF for metadata records within -164.9246,16.6012,-149.4899,25.3959 with JSON response:

PacIOOS Geo Example Link

Decoded Parameters:

rid=local&ridName=NOAA's Geophysical Data Center&searchText=sys.siteuuid:"{68FF11D8-D66B-45EE-B33A-21919BB26421}"&start=1&max=1000&orderBy=relevance&spatialRel=esriSpatialRelWithin&bbox=-164.9246,16.6012,-149.4899,25.3959&maxSearchTimeMilliSec=10000&f=pjson

Multi-Criteria Example

Search within the PacIOOS WAF for metadata records with all of the above with JSON response:

PacIOOS Multi Example Link

Decoded Parameters:

rid=local&ridName=NOAA's Geophysical Data Center&rids=local&searchText=keywords: sea_water_salinity AND endDate:[2009-02-01 TO 2100-01-01] AND startDate:[1800-01-01 TO 2012-02-01] AND sys.siteuuid:"{68FF11D8-D66B-45EE-B33A-21919BB26421}"&start=1&max=1000&orderBy=relevance&spatialRel=esriSpatialRelWithin&bbox=-164.9246,16.6012,-149.4899,25.3959&maxSearchTimeMilliSec=10000&f=pjson

WMS Example

Search within the PacIOOS WAF for metadata records with a WMS service endpoint with GeoRSS response:

PacIOOS WMS Example Link

Decoded Parameters:

rid=local&ridName=NOAA's Geophysical Data Center&rids=local&searchText=wms.resource.url:* AND sys.siteuuid:"{68FF11D8-D66B-45EE-B33A-21919BB26421}"&start=1&max=1000&orderBy=relevance&maxSearchTimeMilliSec=10000&f=georss

WCS Example

Search within the PacIOOS WAF for metadata records with a WCS service endpoint with html response: PacIOOS WCS Example Link

Decoded Parameters:

rid=local&ridName=NOAA's Geophysical Data Center&rids=local&searchText=wcs.resource.url:* AND sys.siteuuid:"{68FF11D8-D66B-45EE-B33A-21919BB26421}"&start=1&max=1000&orderBy=relevance&maxSearchTimeMilliSec=10000&f=html

SOS Example

Search within the PacIOOS WAF for metadata records with an SOS service endpoint with html response: PacIOOS SOS Example Link

Decoded Parameters:

rid=local&ridName=NOAA's Geophysical Data Center&rids=local&searchText=sos.resource.url:* AND sys.siteuuid:"{68FF11D8-D66B-45EE-B33A-21919BB26421}"&start=1&max=1000&orderBy=relevance&maxSearchTimeMilliSec=10000&f=html

OPeNDAP Example

Search within the PacIOOS WAF for metadata records with a OpenDAP service endpoint with html response:

PacIOOS OPeNDAP Example Link

Decoded Parameters:

rid=local&ridName=NOAA's Geophysical Data Center&rids=local&searchText=odp.resource.url:* AND sys.siteuuid:"{68FF11D8-D66B-45EE-B33A-21919BB26421}"&start=1&max=1000&orderBy=relevance&maxSearchTimeMilliSec=10000&f=html

Search via CSW interface

The CSW queries must be issued to the NGDC CSW Service Endpoint as XML POST requests.

ISO Date Modified Search

<csw:GetRecords xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" version="2.0.2" service="CSW" resultType="results" startPosition="1" maxRecords="11" outputSchema="http://www.isotc211.org/2005/gmd"> <csw:Query typeNames="csw:Record" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gml="http://www.opengis.net/gml" > 
<csw:ElementSetName>full</csw:ElementSetName> 
<csw:Constraint version="1.1.0"> 
  <ogc:Filter>
    <ogc:And>
      <ogc:PropertyIsGreaterThan> 
      <ogc:PropertyName>apiso:modified</ogc:PropertyName> <ogc:Literal>2011-09-30</ogc:Literal>
      </ogc:PropertyIsGreaterThan> 
      <ogc:PropertyIsLessThan> 
      <ogc:PropertyName>apiso:modified</ogc:PropertyName> <ogc:Literal>2011-10-02</ogc:Literal>
      </ogc:PropertyIsLessThan> 
    </ogc:And>
  </ogc:Filter>
</csw:Constraint> 
</csw:Query> 
</csw:GetRecords>

WMS Search Example

<?xml version="1.0"?>   
<csw:GetRecords xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" version="2.0.2" service="CSW" resultType="results" 
outputSchema="http://www.isotc211.org/2005/gmd" startPosition="1" maxRecords="1000">
  <csw:Query typeNames="csw:Record" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gml="http://www.opengis.net/gml">
  <csw:ElementSetName>full</csw:ElementSetName> 
  <csw:Constraint version="1.1.0">
  <ogc:Filter>
    <ogc:And>
      <ogc:PropertyIsEqualTo>
        <ogc:PropertyName>sys.siteuuid</ogc:PropertyName>
        <ogc:Literal>{68FF11D8-D66B-45EE-B33A-21919BB26421}</ogc:Literal>
      </ogc:PropertyIsEqualTo>
      <ogc:PropertyIsLike wildCard="*" escape="\" singleChar="?"> 
        <ogc:PropertyName>apiso:ServiceType</ogc:PropertyName>
        <ogc:Literal>*wms*</ogc:Literal>
      </ogc:PropertyIsLike>
    </ogc:And> 
  </ogc:Filter> 
</csw:Constraint> 
</csw:Query>
</csw:GetRecords>

WCS Search Example

<?xml version="1.0"?>   
<csw:GetRecords xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" version="2.0.2" service="CSW" resultType="results" 
outputSchema="http://www.isotc211.org/2005/gmd" startPosition="1" maxRecords="1000">
  <csw:Query typeNames="csw:Record" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gml="http://www.opengis.net/gml">
  <csw:ElementSetName>full</csw:ElementSetName> 
  <csw:Constraint version="1.1.0">
  <ogc:Filter>
    <ogc:And>
      <ogc:PropertyIsEqualTo>
        <ogc:PropertyName>sys.siteuuid</ogc:PropertyName>
        <ogc:Literal>{68FF11D8-D66B-45EE-B33A-21919BB26421}</ogc:Literal>
      </ogc:PropertyIsEqualTo>
      <ogc:PropertyIsLike wildCard="*" escape="\" singleChar="?"> 
        <ogc:PropertyName>apiso:ServiceType</ogc:PropertyName>
        <ogc:Literal>*wcs*</ogc:Literal>
      </ogc:PropertyIsLike>
    </ogc:And> 
  </ogc:Filter> 
</csw:Constraint> 
</csw:Query>
</csw:GetRecords>

OPeNDAP Search Example

<?xml version="1.0"?>   
<csw:GetRecords xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" version="2.0.2" service="CSW" resultType="results"
outputSchema="http://www.isotc211.org/2005/gmd" startPosition="1" maxRecords="1000">
  <csw:Query typeNames="csw:Record" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gml="http://www.opengis.net/gml">
  <csw:ElementSetName>full</csw:ElementSetName> 
  <csw:Constraint version="1.1.0">
  <ogc:Filter>
    <ogc:And>
      <ogc:PropertyIsEqualTo>
        <ogc:PropertyName>sys.siteuuid</ogc:PropertyName>
        <ogc:Literal>{68FF11D8-D66B-45EE-B33A-21919BB26421}</ogc:Literal>
      </ogc:PropertyIsEqualTo>
      <ogc:PropertyIsLike wildCard="*" escape="\" singleChar="?"> 
        <ogc:PropertyName>apiso:ServiceType</ogc:PropertyName>
        <ogc:Literal>*opendap*</ogc:Literal>
      </ogc:PropertyIsLike>
    </ogc:And> 
  </ogc:Filter> 
</csw:Constraint> 
</csw:Query>
</csw:GetRecords>

SOS Search Example

<?xml version="1.0"?>   
<csw:GetRecords xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" version="2.0.2" service="CSW" resultType="results"
outputSchema="http://www.isotc211.org/2005/gmd" startPosition="1" maxRecords="1000">
  <csw:Query typeNames="csw:Record" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gml="http://www.opengis.net/gml">
  <csw:ElementSetName>full</csw:ElementSetName> 
  <csw:Constraint version="1.1.0">
  <ogc:Filter>
    <ogc:And>
      <ogc:PropertyIsEqualTo>
        <ogc:PropertyName>sys.siteuuid</ogc:PropertyName>
        <ogc:Literal>{68FF11D8-D66B-45EE-B33A-21919BB26421}</ogc:Literal>
      </ogc:PropertyIsEqualTo>
      <ogc:PropertyIsLike wildCard="*" escape="\" singleChar="?"> 
        <ogc:PropertyName>apiso:ServiceType</ogc:PropertyName>
        <ogc:Literal>*sos*</ogc:Literal>
      </ogc:PropertyIsLike>
    </ogc:And> 
  </ogc:Filter> 
</csw:Constraint> 
</csw:Query>
</csw:GetRecords>