IOOS GTS Statistics

The Global Telecommunication System (GTS) is a coordinated effort for rapid distribution of observations. The GTS monthly reports show the number of messages released to GTS for each station. The reports contain the following fields:

  • location ID: Identifier that station messages are released under to the GTS;

  • region: Designated IOOS Regional Association (only for IOOS regional report);

  • sponsor: Organization that owns and maintains the station;

  • Met: Total number of met messages released to the GTS

  • Wave: Total number of wave messages released to the GTS

In this notebook we will explore the statistics of the messages IOOS is releasing to GTS.

Using this notebook

  1. Pick the appropriate date range of interest.

  2. Edit the variables start_date and end_date in the cell below to reflect your time period of interest (use YYYY-MM-DD format).

  3. Run all the cells in the notebook.

The first step is to pick the appropriate date range of interest.

start_date = '2021-07-01'
end_date = '2021-10-30'

Now we download the data. We will use the NDBC ioosstats server that hosts the CSV files with the ingest data.

import datetime as dt
import pandas as pd
# example https://www.ndbc.noaa.gov/ioosstats/rpts/2021_03_ioos_regional.csv

start = dt.datetime.strptime(start_date,'%Y-%m-%d')
end = dt.datetime.strptime(end_date,'%Y-%m-%d')

# build an array for days between dates
date_array = (start + dt.timedelta(days=x) for x in range(0, (end - start).days))

# get a unique list of year-months for url build
months=[]
for date_object in date_array:
    months.append(date_object.strftime("%Y-%m"))
months = sorted(set(months))

df = pd.DataFrame(columns=['locationID', 'region', 'sponsor', 'met', 'wave'])
for month in months:
  url = 'https://www.ndbc.noaa.gov/ioosstats/rpts/%s_ioos_regional.csv' % month.replace("-","_")
  print('Loading %s' % url)
  df1 = pd.read_csv(url, dtype={'met':float, 'wave':float})
  df1['time (UTC)'] = pd.to_datetime(month)
  df = pd.concat([df,df1])

df.describe()
Loading https://www.ndbc.noaa.gov/ioosstats/rpts/2021_07_ioos_regional.csv
Loading https://www.ndbc.noaa.gov/ioosstats/rpts/2021_08_ioos_regional.csv
Loading https://www.ndbc.noaa.gov/ioosstats/rpts/2021_09_ioos_regional.csv
Loading https://www.ndbc.noaa.gov/ioosstats/rpts/2021_10_ioos_regional.csv
met wave
count 695.000000 695.000000
mean 5492.500719 1207.418705
std 5841.635417 2841.529514
min 0.000000 0.000000
25% 0.000000 0.000000
50% 2576.000000 0.000000
75% 8789.000000 1409.000000
max 17814.000000 17814.000000
#df = e.to_pandas(parse_dates=True)

df["locationID"] = df["locationID"].str.lower()

df['time (UTC)'].unique()
array(['2021-07-01T00:00:00.000000000', '2021-08-01T00:00:00.000000000',
       '2021-09-01T00:00:00.000000000', '2021-10-01T00:00:00.000000000'],
      dtype='datetime64[ns]')

The table has all the ingest data. We can now explore it grouping the data by IOOS Regional Association (RA).

groups = df.groupby("region")

ax = groups.sum().plot(kind="bar", figsize=(11, 3.75))
ax.yaxis.get_major_formatter().set_scientific(False)
ax.set_ylabel("# observations");
../../../_images/2020-10-10-GTS_6_0.png

Let us check the montly sum of data released both for individual met and wave and the totals.

import pandas as pd

df["time (UTC)"] = pd.to_datetime(df["time (UTC)"])
# Remove time-zone info for easier plotting, it is all UTC.
df["time (UTC)"] = df["time (UTC)"].dt.tz_localize(None)

groups = df.groupby(pd.Grouper(key="time (UTC)", freq="M"))

We can create a table of observations per month,

s = groups[['time (UTC)','met','wave']].sum() # reducing the columns so the summary is digestable
totals = s.assign(total=s["met"] + s["wave"])
totals.index = totals.index.to_period("M")

print('Monthly totals:\n',totals,'\n')

print('Sum for time period %s to %s: %i'%(totals.index.min(),totals.index.max(),totals['total'].sum()))
Monthly totals:
                   met      wave      total
time (UTC)                                
2021-07     1000690.0  227112.0  1227802.0
2021-08      967746.0  226282.0  1194028.0
2021-09      923172.0  205020.0  1128192.0
2021-10      925680.0  180742.0  1106422.0 

Sum for time period 2021-07 to 2021-10: 4656444

and visualize it in a bar plot.

%matplotlib inline
import matplotlib.dates as mdates
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(11, 3.75))

s.plot(ax=ax, kind="bar")
ax.set_xticklabels(
    labels=s.index.to_series().dt.strftime("%Y-%b"),
    rotation=70,
    rotation_mode="anchor",
    ha="right",
)
ax.yaxis.get_major_formatter().set_scientific(False)
ax.set_ylabel("# observations")
Text(0, 0.5, '# observations')
../../../_images/2020-10-10-GTS_12_1.png

Those plots are intersting to understand the RAs role in the GTS ingest and how much data is being released over time. It would be nice to see those per buoy on a map.

For that we need to get the position of the NDBC buoys. Let’s get a table of all the buoys and match with what we have in teh GTS data.

import xml.etree.ElementTree as et

import pandas as pd
import requests


def make_ndbc_table():
    url = "https://www.ndbc.noaa.gov/activestations.xml"
    with requests.get(url) as r:
        elems = et.fromstring(r.content)
    df = pd.DataFrame([elem.attrib for elem in list(elems)])
    df["id"] = df["id"].str.lower()
    return df.set_index("id")


buoys = make_ndbc_table()
buoys["lon"] = buoys["lon"].astype(float)
buoys["lat"] = buoys["lat"].astype(float)

buoys.head()
lat lon name owner pgm type met currents waterquality dart elev seq
id
00922 30.0 -90.0 OTN201 - 4800922 Dalhousie University IOOS Partners other n n n n NaN NaN
00923 30.0 -90.0 OTN200 - 4800923 Dalhousie University IOOS Partners other n n n n NaN NaN
01500 30.0 -90.0 SP031 - 3801500 SCRIPPS IOOS Partners other n n n n NaN NaN
01502 30.0 -90.0 Penobscot - 4801502 University of Maine IOOS Partners other n n n n NaN NaN
01503 30.0 -90.0 Saul - 4801503 Woods Hole Oceanographic Institution IOOS Partners other n n n n NaN NaN

For simplificty we will plot the total of observations per buoys.

groups = df.groupby("locationID")
location_sum = groups.sum()
buoys = buoys.T

extra_cols = pd.DataFrame({k: buoys.get(k) for k, row in location_sum.iterrows()}).T
extra_cols = extra_cols[["lat", "lon", "type", "pgm", "name"]]

map_df = pd.concat([location_sum, extra_cols], axis=1)
map_df = map_df.loc[map_df["met"] + map_df["wave"] > 0]

And now we can overlay an HTML table with the buoy information and ingest data totals.

from ipyleaflet import AwesomeIcon, Marker, Map, LegendControl, FullScreenControl, Popup
from ipywidgets import HTML


m = Map(center=(35, -95), zoom=4)
m.add_control(FullScreenControl())

legend = LegendControl(
    {
        "wave": "#FF0000",
        "met": "#FFA500",
        "both": "#008000"
    },
    name="GTS",
    position="bottomright",
)
m.add_control(legend)


def make_popup(row):
    classes = "table table-striped table-hover table-condensed table-responsive"
    return pd.DataFrame(row[["met", "wave", "type", "name", "pgm"]]).to_html(
        classes=classes
    )

for k, row in map_df.iterrows():
    if (row["met"] + row["wave"]) > 0:
        location = row["lat"], row["lon"]
        if row["met"] == 0:
            color = "red"
        elif row["wave"] == 0:
            color = "orange"
        else:
            color = "green"
        marker = Marker(
            draggable=False,
            icon=AwesomeIcon(name="life-ring", marker_color=color),
            location=location,
        )
        msg = HTML()
        msg.value = make_popup(row)
        marker.popup = msg
        m.add_layer(marker)
m