Read realtime data from IOOS Sensor Map via ERDDAP tabledap

Created: 2017-03-21

Updated: 2022-07-07

Web Map Services are a great way to find data you may be looking for in a particular geographic area.

Suppose you are exploring the IOOS Sensor Map, you select Oxygen and click on the only returned value, the Moss Landing Marine Laboratories (MLML) station.

One can download the data in multiple forms from the site, including generating an ERDDAP URL for the request.

These features makes Sensor map an extremely useful tool for quick data explorations but now imagine if you want automate that instead of exploring the Sensor Map interactively? Or if you want to make multiple small modification to your query? It would be very tedious and error prone to try that with the Sensor Map interface. The good news is that we cab automate that by querying the ERDDAP server directly.

First we need to instantiate a server object.

from erddapy import ERDDAP

server = "http://erddap.sensors.ioos.us/erddap"
e = ERDDAP(server=server, protocol="tabledap")

Now we can search for “dissolved oxygen moss landing” and inspect the dataset_ids.

import pandas as pd

url = e.get_search_url(search_for="dissolved oxygen moss landing", response="csv")

print(url)
df = pd.read_csv(url)
df

http://erddap.sensors.ioos.us/erddap/search/advanced.csv?page=1&itemsPerPage=1000&protocol=tabledap&cdm_data_type=(ANY)&institution=(ANY)&ioos_category=(ANY)&keywords=(ANY)&long_name=(ANY)&standard_name=(ANY)&variableName=(ANY)&minLon=(ANY)&maxLon=(ANY)&minLat=(ANY)&maxLat=(ANY)&minTime=&maxTime=&searchFor=dissolved+oxygen+moss+landing

	griddap	Subset	tabledap	Make A Graph	wms	files	Title	Summary	FGDC	ISO 19115	Info	Background Info	RSS	Email	Institution	Dataset ID
0	NaN	NaN	http://erddap.sensors.ioos.us/erddap/tabledap/...	http://erddap.sensors.ioos.us/erddap/tabledap/...	NaN	NaN	CeNCOOS in situ water quality monitoring using...	The Seawater intake station is maintained by M...	http://erddap.sensors.ioos.us/erddap/metadata/...	http://erddap.sensors.ioos.us/erddap/metadata/...	http://erddap.sensors.ioos.us/erddap/info/mlml...	https://sensors.ioos.us/#metadata/48035/station	http://erddap.sensors.ioos.us/erddap/rss/mlml_...	http://erddap.sensors.ioos.us/erddap/subscript...	Moss Landing Marine Laboratory	mlml_mlml_sea
1	NaN	NaN	http://erddap.sensors.ioos.us/erddap/tabledap/...	http://erddap.sensors.ioos.us/erddap/tabledap/...	NaN	NaN	CeNCOOS in situ water quality monitoring at Mo...	The Monterey shore station is maintained by Mo...	http://erddap.sensors.ioos.us/erddap/metadata/...	http://erddap.sensors.ioos.us/erddap/metadata/...	http://erddap.sensors.ioos.us/erddap/info/mlml...	https://sensors.ioos.us/#metadata/20362/station	http://erddap.sensors.ioos.us/erddap/rss/mlml_...	http://erddap.sensors.ioos.us/erddap/subscript...	Moss Landing Marine Laboratory	mlml_monterey

Interesting, we found two dataset_ids instead of the one showed in the sensor map! Let’s investigate the datasets we found and try to figure out why there is a discrepancy.

for k, v in df[["Dataset ID", "Title", "Summary"]].T.items():
    dataset_id = v["Dataset ID"]
    title = v["Title"]
    summary = v["Summary"].replace("\\n", "\n")
    print(f"{dataset_id}: {title}\n\n{summary}\n")

mlml_mlml_sea: CeNCOOS in situ water quality monitoring using the seawater input for Moss Landing Marine Laboratory.

The Seawater intake station is maintained by Moss Landing Marine Laboratories who share the data with CeNCOOS. Data collected includes temperature, conductivity, salinity, fluorescence, beam attenuation, transmission, dissolved oxygen, dissolved organic saturation, pH, and tide height. Seawater data observations are collected from raw seawater drawn through an intake pipe. The pipe intake opening is located at 36.8025N and 121.7915W and is ~16.6m (54.4ft) below MLLW. The seawater sensors are cleaned of biofouling agents on a weekly/twice-weekly interval, though some data drift can be observed in transmission, beam attenution, and fluorescence. These nearshore sensors are part of the Central and Norther California Ocean Observing System (CeNCOOS). They measure various water quality parameters at fixed points along the California coast.

cdm_data_type = TimeSeries
VARIABLES:
time (seconds since 1970-01-01T00:00:00Z)
latitude (degrees_north)
longitude (degrees_east)
z (Altitude, m)
pco2 (microatm)
pco2_qc_agg (pCO2 QARTOD Aggregate Quality Flag)
pco2_qc_tests (pCO2 QARTOD Individual Tests)
sea_water_electrical_conductivity (Conductivity, mS.cm-1)
sea_water_electrical_conductivity_qc_agg (Conductivity QARTOD Aggregate Quality Flag)
sea_water_electrical_conductivity_qc_tests (Conductivity QARTOD Individual Tests)
fluorescence (microg.L-1)
fluorescence_qc_agg (Fluorescence QARTOD Aggregate Quality Flag)
fluorescence_qc_tests (Fluorescence QARTOD Individual Tests)
mole_concentration_of_nitrate_in_sea_water (Nitrate Concentration, micromol.L-1)
mole_concentration_of_nitrate_in_sea_water_qc_agg (Nitrate Concentration QARTOD Aggregate Quality Flag)
mole_concentration_of_nitrate_in_sea_water_qc_tests (Nitrate Concentration QARTOD Individual Tests)
mole_concentration_of_dissolved_molecular_oxygen_in_sea_water (Dissolved Oxygen Molecular Concentration, micromol.L-1)
mole_concentration_of_dissolved_molecular_oxygen_in_sea_water_qc_agg (Dissolved Oxygen Molecular Concentration QARTOD Aggregate Quality Flag)
mole_concentration_of_dissolved_molecular_oxygen_in_sea_water_qc_tests (Dissolved Oxygen Molecular Concentration QARTOD Individual Tests)
... (22 more variables)


mlml_monterey: CeNCOOS in situ water quality monitoring at Monterey Bay Commercial Wharf.

The Monterey shore station is maintained by Moss Landing Marine Labs. The station has been operational since 2012 and consists of in water sensors that are fixed to a pier sampling every 15 minutes. These sensors provide near-real time observations of ocean water salinity, temperature, dissolved oxygen, chlorophyll fluorescence, turbidity and pH. These sensors provide near-real time observations of ocean water conductivity, salinity, temperature, depth, dissolved oxygen, chlorophyll fluorescence, turbidity and nitrate. These nearshore sensors are part of the Central and Norther California Ocean Observing System (CeNCOOS). They measure various water quality parameters at fixed points along the California coast.

cdm_data_type = TimeSeries
VARIABLES:
time (seconds since 1970-01-01T00:00:00Z)
latitude (degrees_north)
longitude (degrees_east)
z (Altitude, m)
mass_concentration_of_chlorophyll_in_sea_water (Chlorophyll, microg.L-1)
mass_concentration_of_chlorophyll_in_sea_water_qc_agg (Chlorophyll QARTOD Aggregate Quality Flag)
mass_concentration_of_chlorophyll_in_sea_water_qc_tests (Chlorophyll QARTOD Individual Tests)
sea_water_electrical_conductivity (Conductivity, mS.cm-1)
sea_water_electrical_conductivity_qc_agg (Conductivity QARTOD Aggregate Quality Flag)
sea_water_electrical_conductivity_qc_tests (Conductivity QARTOD Individual Tests)
mass_concentration_of_oxygen_in_sea_water (Dissolved Oxygen Concentration, mg.L-1)
mass_concentration_of_oxygen_in_sea_water_qc_agg (Dissolved Oxygen Concentration QARTOD Aggregate Quality Flag)
mass_concentration_of_oxygen_in_sea_water_qc_tests (Dissolved Oxygen Concentration QARTOD Individual Tests)
sea_water_practical_salinity (Salinity, 1e-3)
sea_water_practical_salinity_qc_agg (Salinity QARTOD Aggregate Quality Flag)
sea_water_practical_salinity_qc_tests (Salinity QARTOD Individual Tests)
sea_water_pressure (decibars)
sea_water_pressure_qc_agg (Sea Water Pressure QARTOD Aggregate Quality Flag)
sea_water_pressure_qc_tests (Sea Water Pressure QARTOD Individual Tests)
sea_water_temperature (Water Temperature, degree_Celsius)
... (9 more variables)

The Sensor map returned only the dataset_id mlml_mlml_sea which corresponds to the CeNCOOS in situ water quality monitoring using the seawater input for Moss Landing Marine Laboratory. However, we also got the dataset_id for mlml_monterey, or the commercial wharf water quality monitoring.

Let’s request the ERDDAP info on the latter to try to understand why it is available on the server but did not make into the Sensor Map. We usually suspect some sort of cut off due to the time coverage.

df = pd.read_csv(e.get_info_url(dataset_id="mlml_monterey", response="csv"))

df.loc[df["Attribute Name"] == "time_coverage_end"]

	Row Type	Variable Name	Attribute Name	Data Type	Value
69	attribute	NC_GLOBAL	time_coverage_end	String	2019-11-29T15:45:00Z

Bingo! Looks like station is no longer collecting data. The sensor map only shows real-time observations (past 4-hours).

For the sake of simplicity let’s use only the mlml_mlml_sea and download the last month of data.

e.dataset_id = "mlml_mlml_sea"
e.variables = ["time", "mole_concentration_of_dissolved_molecular_oxygen_in_sea_water"]
e.constraints = {"time>=": "now-30days"}

df = e.to_pandas(index_col="time (UTC)")
ax = df.plot(figsize=(17, 5));

We hope this example demonstrate the flexibility of direct requests to the ERDDAP server used in the Sensor Map. In this notebook we:

• Search the server with keywords.

• Found dataset_ids and checked their metadata.

• Identified the dataset_id of interested and request data at a specific time period.