quoi de neuf à coriolis en 2008 ? gmmc 13-15 octobre 2008 s pouliquen & coriolis team
DESCRIPTION
Quoi de Neuf à Coriolis en 2008 ? GMMC 13-15 Octobre 2008 S Pouliquen & Coriolis team. Developing a Data Management System for Glider data in Europe Sylvie Pouliquen GROOM Annual meeting 4-5 June 2013. The Objectives of an integrated data system. - PowerPoint PPT PresentationTRANSCRIPT
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3
Quoi de Neuf à Coriolis en 2008 ?GMMC 13-15 Octobre 2008
S Pouliquen & Coriolis team
Developing a Data Management System for Glider data in EuropeSylvie Pouliquen
GROOM Annual meeting
4-5 June 2013
Developing a Data Management System for Glider data in EuropeSylvie Pouliquen
GROOM Annual meeting
4-5 June 2013
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3 The Objectives of an integrated data system
• Data accessible easily from a unique point
• Data coherent in term of :Data formatData QualityProcessing chain ( clearly documented)
• Serve both Operational and Research UsersData are available in near real time ( within less than
24 hours)Data are available in delayed mode after calibration
and /or validation
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3 What About Gliders
• Glider activities are presently driven by individual research drivers
• Gliders can deliver real-time data for core parameters ( T, S Current , Chl, O2) that are useful for both research an operational users
• Benefit from what has been developed by Argo and OceanSites Developing integrated Data Management system Common Data format to users Real –time QC of core parameters
• Gliders are complementary to other platforms and synergy should be developed Developing a deployment strategy for other needs than pure
research ( ie. GMES Marine Core Service in Europe)
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3 What is the starting point for GROOM
• Data management is done by the different research communities using their own methods
• Some harmonization activities started in FP5-MFSTEP and FP6-MERSEA projects and FP7 MyOcean
• Data exchange in Real time is working on a best effort schema through EGO but without any commitment nor from providers or data managers
• Link with GMES MCS is done though Coriolis providing Glider data as profiles
• No agreement neither on RTQC or DMQC but best practices on RTQC through MyOcean INSTAC not widely known by the glider teams
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3
Groom Data Management activities
Task 3.2
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3 GROOM Data Management Objectives
• Data accessible easily from a unique point
• Data coherent in term of :Data formatData QualityProcessing chain ( clearly documented)
• Serve both Operational and Research UsersData are available in near real time ( within less than
24 hours)Data are available in delayed mode after calibration
and /or validation
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3 Objective of task 3.2
• Improve coherency of Glider dataset in Europe
• Facilitate access to Glider data by defining a Data system for Glider data
• The system is built to exchange glider data Must first be useful for Glider users and Glider
operators and therefore provide access to as much as possible information (metadata, scientific, technical data) provided by Gliders
Delivering of Glider Data either on GTS or FTP for Operational users ( i.e. MyOcean) will be carried on as specific delivery system that will convert the GDAC data into products usable by these operational users
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3
Who does what ?
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3
GROOM Data flow
PIS
Scientific UsersOperational
Centers
GDACFrance
Final correctionData to DAC - GDAC
RT DATA QUALITY CONTROLData assembly Centres
GTS FEED
RT within hours
DAC DAC DAC
Within hours
DAC
Glider Operators
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3 Roles of Glider Actors
• PI : Principal InvestigatorsTeam or scientists who define the glider mission,
deploy the glider and carry out post-recovery delayed mode QC
• Glider OperatorsTeam in charge of steering the glider, collecting all
the metadata and the deployment information required for processing, collect all the data transferred in realtime by the glider. Collect the post-recovery high resolution data
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3 Roles of Glider Actors
• the Dacs : the DAC is the facility set up by one or more nations/institutes to provide RealTime and Delayed mode glider data to the users. It collects the data from the Glider Operator,converts to standard exchange format,applies standardized real-time quality control,delivers data to the GTS and GDACs within few
hours of the surfacing and to PIs ,coordinates glider data handling for the gliders
under their control.
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3 List of DACsCountry DAC Glider Operator Delayed Mode PI
UK BODC UEA , SAMS, NERC (NOC, POL) CEFAS, BAS
UEA , SAMS, NERC (NOC, POL), CEFAS, BAS
France CORIOLIS LOCEAN, LOV, MIO, LEGOS, LPO, DT-INSU
LOCEAN, LOV,MIO, LEGOS, LPO
Italy CMRE CMRE To Be Defined
OGS/CORIOLIS
OGS OGS
Germany HZG HZG HZG
CORIOLIS GEOMAR,AWI GEOMAR, AWI
SPAIN CORIOLIS PLOCAN PLOCAN
SOCIB SOCIB, CSIC SOCIB, CSIC
Norway UiB/IMR UiB UiB
Cyprus OC-UCY OC-UCY OC-UCY
Greece CORIOLIS HCMR ? To Be Defined
Finland To Be Defined To Be Defined To Be Defined
Ireland To Be Defined MI MI
Poland CORIOLIS IOPAS IOPAS
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3 Roles of Glider Actors
• GDAC : The GDAC operates the data services where the master copies of the data resides. It doesn’t perform any additional individual glider QC activities. Central point for data distribution on Internet for all
GROOM gliders Can perform data format transformation, of set up
additional services ( OGC viewing service, OpenDap/Oceanotron download services ,…) to fulfil additional needs.
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3
Common data format
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3 Improve Coherency of Glider dataset in Europe
• Worked on the definition of a Glider , of a deployment for a glider.
A glider is moving platform that is steerable. It can have a propeller and this information must be recorded in the metadata.
• Define Level of processing: Level0 : Data provided by the glider without any unit
transformation or geophysical interpretation. Level1: Geophysical parameters with a quality indicator set up
by automatic QC procedures together with the data acquired by the glider. This is the level shared in Near Real Time.
Level2 : Geophysical parameters calibrated after glider recovery together with quality flag information, if possible error estimation together with the non-corrected data provided at Level1. This is the level shared in Delayed Mode
Level3 and after : Product derived from glider data ( gridded fields, additional parameters calculated …) This is not addressed in the present GROOM data management activities.
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3 How to Describe a Glider
Sensor S Sensor S
Sensor S Sensor S
Takes on board
Deployment NDeployment N
Deployment NDeployment N
Performs
Description of the glider for this deployment•What are the sensors on board•What is the configuration
Is described by
Transmits for each dive
Technical information measurementsTechnical information measurements
Technical information measurementsTechnical information measurements
Technical information measurements
Scientific measurementsScientific measurements
Scientific measurementsScientific measurements
Scientific measurements
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3 Improve Coherency of Glider dataset in Europe
• Worked on common data format to share the Glider data based on OceanSites NetCDF format already used in EuroGOOS/ MyOcean/ SeaDataNetDefined how to store Metadata to register the
mission description and its evolution in time when changed through down link
Enhanced OceanSites format for Scientific information by adding if necessary the new parameters sampled
Defined how to store the technical information by definition common vocabulary
First version of the user manual was delivered Common tools to produce these Netcdf files are
under way
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3
RT Processing chain modules
Source data Seaglider Slocum Other gliders
Processing modules
Data delivery
Common Readers in matlab that convert Manufacturers Data into .mat files
Conversion of .mat output to EGO format NetCDF for transmission
Real Time quality control routines
PI GDAC GTS
Generic Description-Platform-Sensors
My Glider description
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3 Data Stream in Real time
Glider Operator
DAC GDAC
Sci and Tec data
Sci and Tec
users
Customize to meet
other needs
Sci Data
GTS Users
MyOcean Users
DATA
DATA
Manufacturer Format
Common Format
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3 Define Real Time QC procedure
• Action taken to work on common Near Real Time QC procedures in agreement with what exist already within EuroGOOS/MyOcean/SeaDataNetAdopt EuroGOOS/MyOcean NRT QC procedures
for T&S, Chl and O2
Enhance if necessary these procedures to take into account Glider specific behavior
Develop new recommendations in Partnership with Myo/SDN for additional parameters available in real time
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3 Data Stream after recovery
Glider Operator
DAC GDAC
Common Format
Common Format
users
HR
HR TEC
High Resolution scientifi
c data
Real Tim
e Data in
Common fo
rmat -
-----
---
-Delayed m
ode High re
solution data
Manufacturer Format
Common Format
High Res Manufacturer Format
Customize to meet
other needs
Sci Data
GTS Users
MyOcean Users
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3 Post recovery procedures
• The Goal : correct the data transmitted in real timeCheck RTQC again. Additional run taking into
account past and future data is possible (future data not available in RT !)
If necessary Merge RT data with flash card data in order to fill the gaps left by the RT data transmission in terms of resolution and/or parameters measured
Cross-calibration along the whole deployment with reference to in-lab water samples measurements when possible.
Working group to develop recommendations for T&S, Chla, Oxygen, Current
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3
Possible extention outside Europe through the EGO Cost action
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3
GROOM Data flow
PIS
Scientific UsersOperational
Centers
GDACFrance
Final correctionData to DAC - GDAC
RT DATA QUALITY CONTROLData assembly Centres
GTS FEED
RT within hours
DAC DAC DAC
Within hours
DAC
Glider Operators
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3 Possible extention outside Europe
PIS
Scientific UsersOperational
Centers
Final correctionData to DAC - GDAC
RT DATA QUALITY CONTROLData assembly Centres
GTS FEED
GliderInfo
Center?
RT within hours
Monitoring
GDAC?
Archive?
WWW FTP
GDACFrance
DAC DAC DAC DAC
Within hoursGlider
Operators
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3 Data Format and QC procedures
• Data Format : based on OceanSites netcdf format with CF convention also used by IMOS /AustraliaCompare the format metadata to be sure that we are
coherent (link with ODIP EU project): • name the same thing the same way • list of mandatory fields that should be in all files• Share tools to generate netcdf files
• NRT QC procedures : Both IMOS and QUARTOD procedures are taken as input for defining the EU RTQC procedures for core parameters.
• Post-recovery procedure : still and R&D activity but collaboration with international partners should be encouraged
GR
OO
M A
nn
ual m
eeti
ng
D
ata
Man
ag
em
en
t iss
ues
T
riest
e 4
-5 Ju
ne 2
01
3
Quoi de Neuf à Coriolis en 2008 ?GMMC 13-15 Octobre 2008
S Pouliquen & Coriolis team
Thank You
Questions ?
Thank You
Questions ?