There has been a lot of interest in connecting Large Language Models (LLMs) to IOOS data, with an emphasis on being able to provide data-backed responses back to the user. Model Context Protocol (MCP) is currently the standard way the community is connecting LLMs to data and external systems. (https://modelcontextprotocol.io/docs/2026-07-28/getting-started/intro)

There have already been a variety of MCP servers for ERDDAP started:

These tools get us started connecting to data sources, but beyond that, how do we start establishing trust in the responses returned? Did the LLM actually grab the correct data and use it in a reasonable manner in its output?

The more we can automate metric gathering against these systems, the more evidence we can gather to support or invalidate these approaches for real use-cases.

Expected Outcomes

  • Identify capability gaps in existing MCP solutions.
  • Identify data sources currently unreachable to LLMs (e.g. no MCP component, API too slow, data is hard to find, etc.)
  • Implement more skills/capabilities to enable MCP data access.
  • Discuss evaluation criteria; how to judge a good versus bad response.
  • Recommend future direction for MCP development.

Skills required

  • Basic understanding of data access patterns and APIs.
  • Familiarity with how LLMs work and interact with tooling.
  • Domain expertise - help establish cases, judge good vs bad responses.

Expertise

Novice

Topic Lead(s)

@jonmjoyce

Relevant links

Functioning Prototype

TBD

Workflow

Tasks:

Updates