From EO archives to operational data services

The problem

Many organisations hold valuable satellite-derived and model datasets as scattered files on a workstation or HPC share. Nobody outside the analysis team can reach them, updates are manual, and every new dashboard repeats the same data wrangling. Making the data available through standard services (OPeNDAP, WMS/WCS, REST) is what lets partners, apps and decision-makers use it reliably.

What you get

• Automated ETL that converts and harmonises incoming files (NetCDF, HDF, GRIB, GeoTIFF) with CF-style metadata
• THREDDS Data Server and/or ncWMS catalogue for multidimensional data, and GeoServer for map layers
• PostGIS database for vector and statistical outputs
• A web viewer or dashboard (Tethys, React/Next.js or Vue.js) on top of the services
• Linux deployment, cron scheduling and an operations runbook

How it works

1. 1

   Audit

   Inventory the existing datasets, formats, update cycles and the people and systems that need access.

2. 2

   Design

   Define the data model, catalogue structure, service endpoints and server layout.

3. 3

   Build

   Write the ETL, configure THREDDS/ncWMS/GeoServer and PostGIS, and build the viewer.

4. 4

   Operate

   Deploy on Linux servers, set up scheduled updates and monitoring, document the setup and train the team or continue on retainer.

Tools

• Python (xarray, NumPy)
• CDO / NCO
• GDAL
• THREDDS Data Server
• ncWMS
• GeoServer
• PostgreSQL / PostGIS
• Tethys Platform
• Apache Tomcat
• Linux