Cloud-Native Ocean Model Workflows via Agentic Coding
TL;DR
This project merges IOOS Code Sprint topics #80 and #83. It focuses on utilizing agentic coding tools (e.g., Claude) to construct cloud-native workflows for ocean model output. Key deliverables include generating virtual Icechunk datasets, creating STAC catalogs stored in GeoParquet, and developing downstream applications and Jupyter notebooks that query these cloud-optimized datasets (icechunks).
Dive in via our project website or play with some cloud-optimized datasets in Colab
Background
This effort combines proposals for agentic coding application and cloud-optimized virtual ocean-model dataset development. Managing and distributing large-scale coastal and ocean model output presents structural challenges for the IOOS community. Cloud-native data architectures, particularly Zarr and Icechunk, paired with SpatioTemporal Asset Catalogs (STAC), provide an efficient mechanism for data access and discovery. This project evaluates the application of agentic coding assistants to accelerate the development of these data engineering workflows.
Objectives
- Agentic Coding Integration: Utilize AI coding agents (e.g., Claude) to script, debug, and implement the data pipeline.
- Virtual Icechunk Construction: Employ tools such as VirtualiZarr to generate virtual Icechunk datasets from existing ocean model output (e.g., NetCDF), avoiding direct data duplication.
- STAC Catalog Generation & GeoParquet Storage: Generate STAC items and collections for the virtual Icechunks. Transition STAC item storage to GeoParquet using rustac-py and pyarrow to enable performant querying of large asset collections.
- Workflow Development: Develop reproducible analytical workflows (e.g., Python/Xarray/HoloViz notebooks) and prototype applications (e.g., TypeScript/React with Zarr-layers) that initiate data retrieval via STAC queries.
Methodology
The workflow begins with existing model output on cloud storage. Assisted by agentic tools, participants will execute Python scripts to generate virtual Icechunk representations. STAC metadata will then be generated and embedded into GeoParquet schemas. The final phase involves constructing downstream applications to demonstrate STAC-driven data discovery, opening the Icechunk representations via Xarray for visualization and analysis.
Why cloud-optimized, why VirtualiZarr and Icechunk?
Ocean model output especially is so so many netcdf files. Sadly much is unsuitable for virtualization (it is unchunked and files are one single massive chunk), but lots is suitable and we have tricks for dealing with some unchunked types. But we need more icechunks built with different types of source files so that we can learn.
Expected Outcomes
- New cloud-optimized icechunks. We have a Source Coop organization for hosting.
- New notebooks that show people how to do cool things with Icechunks. See our examples
- Example STAC catalogs hosted in GeoParquet format.
- Reference implementations of analytical workflows and prototype web applications driven by STAC queries.
Skills required
Ability to type some prompts!
Expertise
Understanding what users might like.
Topic Lead(s)
Eli Holmes, Rich Signell
Relevant links
- Our tutorials
- Source Coop Ocean Icechunks
- Tom Nichols (EarthMover) How to make ai ready archives
- JupyterHub for the hackathon
What are we making?
Tasks: