Data infrastructure

Context

An ocean observatory data management system must address several challenges:

  • data products are generated from heterogeneous and regionally distributed observatory nodes in Europe and have different characteristics in format, metadata, resolution and validation of data.
  • data archive centres exist for some data types but are not yet fully adapted for other parameters. Moreover, the responsibility for supporting each node may be divided among several different agencies.
  • in order to access data products from multiple sources, data need to be quality controlled in a uniform manner. Coordination is required between data providers and data users.
  • the network may be extended and has to be integrated with existing and future GEOSS related earth observation systems; especially, the thematic portal policy of GMES is under construction. ESONET will coordinate its schedule with GMES. Integration with other systems is hampered because present concepts and implementations of spatial data infrastructures (SDI) follow different architectures using different standards or de facto standards (INSPIRE).
  • monitoring of environmental processes generates large amounts of data. Location independent usage of those data and related services require significant storage and computer resources and appropriate network bandwidths. Current computing and storage capacities in European data centres as well as bandwidth in European networks are at least sufficient for the demonstration missions in WP 4. Depending on the data type and the capture frequency additional capacities might be needed for filtering, processing, evaluation and visualization of data in productive implementations.
  • long-term preservation and publication of data have to be ensured.

Activities

Concretely, below items shall be deeply processed:

  • Data management concepts have to be develop for a networked system including organizational schemes for possible data flows. A fundamental underlying principle is the full and open exchange of data and information for scientific and educational purposes (GEOSS data sharing principles).
  • A hardware independent spatial data infrastructure (SDI), including sensor registry, shall be implemented and checked. Interoperability will be largely based on the implementation of globally accepted information standards (e.g. Sensor ML, ISO19xxx family of standards, SOAP/WSDL, GRID standards) and existing Spatial Data Infrastructures (SDI). The activity will be carried out in close collaboration with WP2 (standardization).
  • Data capture (also real time or near time and event based), long-term archiving, publication and dissemination of observatory data, metadata, and data products using European and international data centres shall be organized. For all these activities, ESONET shall lean on global and pan-European infrastructures for marine data management (SEADATANET / NODCs and ICSU-WDCs) and closely cooperate with EU INSPIRE initiative and other European projects like CarboOcean, MERSEA or HERMES. The integrative usage of European long-term capacities in GRID like structures is not only the basis for the WP4 demonstration missions but it will also guarantee a sustainable operation of a possible productive observatory network in the FP7.
  • Available ESONET services shall be conceived upon common data infrastructure (e.g. data discovery and download, handling of extreme events).
  • Above items shall be implemented and checked within demonstration missions (WP4), tests (WP1, WP2, WP3) and site surveys (WP5).

Following wiki-page deals with these issues http://dataportals.pangaea.de/esonet.

Data exchange format specifications (sensorML, O&M) releases.