coriolis data center
Post on 21-Feb-2017
318 Views
Preview:
TRANSCRIPT
Distributed system development, including standards and interoperability
ByDick M.A. Schaap – technical coordinator
Bologna, September 08
SeaDataNet Infrastructure objectives
SeaDataNet aims to set up and operate an efficient Pan-European distributed infrastructure for managing marine and ocean data by connecting:
40 National Oceanographic Data Centres (NODC’s), national oceanographic focal points, and ocean satellite data centres, in Europethese Data Centres are mostly divisions of major national marine research institutes and based in 35 countries, surrounding the European seas
SeaDataNet aims to ensure a comparable quality of data sets and to make data sets easily accessible on-line through a unique portal, while the data sets are stored and managed at the Data Centres.
SeaDataNet Infrastructure overall concept
SeaDataNet Infrastructure JRA activities
JRA1: to define and develop common standards and protocols for SeaDataNet, that will the basis for interoperability inside the SeaDataNet network of Data Centres and outside to other systems.
JRA2: to design and develop the architecture and software components for the SeaDataNet infrastructure and to coordinate its implementation at the Data Centres and central SeaDataNet portal.
Close interaction between JRA1 and JRA2. The outcome of JRA1 also sets requirements towards JRA3: adapting the stand-alone ODV data analysis and viewing software package for integrated use with SeaDataNet output.
Because of this close relation the activities in JRA1, JRA2 and JRA3 are coordinated by a Technical Task Team.
SeaDataNet Technical Task Team
Oversees and coordinates the technical developments
Includes 11 core technical partners
8 Meetings sofar:Paris, France 18-19 May 2006Hamburg, Germany 4-5 September 2006The Hague, The Netherlands 7-8 December 2006Trieste, Italy 21-22 March 2007 The Hague, The Netherlands 2-3 July 2007Nice, France, 30-31 August 2007Liverpool, UK, 18-19 December 2007 Paris, France, 13-14 May 2008
SeaDataNet Technical Task Team
Methodology: Agenda with relevant topics, leading speakers, open discussion and brainstorming, possible subgroups for further explorations and analyses, resulting in technical specifications of system architecture and its modules, followed by implementation, evaluation and operation
All presentations and documents are included in the SeaDataNet Extranet in the TTT section. Note: All Project Partners are encouraged to visit this regularly.
All actions / decisions are minuted in a TTT Action List, which is updated regularly and also can be found in the Extranet TTT section
SeaDataNet Infrastructure versions and planning
Version 0 = Continuation and maintenance of existing Sea-Search systems with minor modifications
Version 1 = Harmonised and upgraded metadatabases; Transparent data access involving all Data Centres. Prototype ready by March 2008; all TTT partners by end 2008; thereafter gradually migrating all partners – operational by mid 2009.
Version 2 = Adding OGC viewing services and further virtualisation of data access. Operational by 2010.
SeaDataNet Version 0
Set-up of SeaDataNet portal (www.seadatanet.org) with CMS
Continuation of existing Sea-Search metadata systems:EDMED - European Directory of Marine Environmental Data SetsEDMERP – European Directory of Marine Research ProjectsCSR – Cruise Summary Reports EDIOS – European Directory of Ocean observing SystemsEDMO – European Directory of Marine OrganisationsCDI – Common Data Index
Operation of existing data access systems / procedures of all Data Centres, referred in the Common Data Index directory
Set-up of SeaDataNet extranet
Set-up of SeaDataNet mailing lists
SeaDataNet Portal website
www.seadatanet.org
SeaDataNet System Approach for V1 and V2
An approach has been adopted, which is in line with INSPIRE. The SeaDataNet infrastructure should consist of the following services:
Discovery services = Metadata directoriesSecurity services = Authentication, Authorization & Accounting (AAA)Delivery services = Data access & downloading of data setsViewing services = Visualisation of metadata, data and data products Product services = Generic and standard products Monitoring services = Statistics on usage and performance of the system Maintenance services = Updating of metadata by Data Centres
A network of interconnected Data Centres and a central Portal, that will give users access to the various SeaDataNet services, and information on data management standards, tools and protocols.
SeaDataNet User’s portal architecture V1
services for metadataData downloading servicesVisualization services (WMS) for V2
Ifremer Database
BODC Database
BSH Database
...
Download managers in Data centers
CSR EDIOS
EDMED
CDI
Shopping basket
Requestsstatus
manager
datametadata
CSR
CDI
Project info
Metadata & Data catalogues
Data request Status of request
General request Metadata request
Organisation +data source id
Data download
at BODCat BSH at MARIS+ EDMERP
Entry point for access hits
EDMO
EDIOS EDMEDEDMERP EDMO
SoftwareVocabularies
Standards Cross search
AAA
Registr.My transact.
User Register
Userregistration
SeaDataNet Interoperability
Interoperability is the key to distributed data management system success. This is achieved in SeaDataNet via:Using common quality control protocols and flag scale Using common and controlled vocabularies, including international content governanceAdopting the ISO 19115 metadata standard for all metadata directoriesProviding XML Validation Services to quality control the metadata maintenance Providing standard metadata entry toolsUsing harmonised Data Transport Formats (NetCDF, ODV ASCII and MedAtlas ASCII) for data sets delivery Adopting of OGC standards for mapping and viewing servicesUsing SOAP Web Services in the SeaDataNet architecture
SeaDataNet Quality Control GuidelineA guideline (V1) of recommended QC procedures, reviewing NODC schemes and other known schemes (e.g. WGMDM guidelines, World Ocean Database, GTSPP, Argo, WOCE, QARTOD, ESEAS,SIMORC, etc.).
QC methods for CTD (temperature and salinity), Current meter data (including ADCP), Wave data, Sea level dataA scheme of QC flags to be used in SeaDataNet.These flags are for assigning to individual data values. They are not for allocating to whole data series, or to accompanying information.
Compiled in discussion with IOC, ICES and JCOM, to ensure an international acceptance and tuning. Important feedback from the joint IODE/JCOMM Forum on Oceanographic Data Management and Exchange Standards (January 2008), joined by SeaDataNet and international experts to consider on-going work on standards and to seek harmonisation, where possible.
Now extending the guideline with QC methods for surface underway data, nutrients, geophysical data, and biological data. V2: April 2009.
SeaDataNet Common VocabulariesUse of common vocabularies in all metadatabases and data formats is
an important prerequisite towards consistency and interoperability.
Set-up and population of Common Vocabularies. The SeaDataNet Vocabulary service is based upon the NERC DataGrid (NDG) vocabulary Web service. For end-users there is a vocabulary Client Interface for searching and browsing and to export selected entries in csv format.
The Web service is compliant to WS Basis Profile 1.1, which is adopted as standard for all Web services in SeaDataNet.
Content governance of the vocabularies is very important and is done by a combined SeaDataNet and MarineXML Vocabulary Content Governance Group (SeaVoX), moderated by BODC, and including experts from SeaDataNet, MMI, MOTIIVE, JCOMMOPS and more international groups. SeaVox operates by mailing list server.
SeaDataNet Common Vocabularies
Vocabularies User Interface http://seadatanet.maris2.nl/v_bodc_vocab/welcome.aspx
SeaDataNet Common Data Transport Formats
V1: data sets are accessable by downloading services. Delivery of data sets to users requires common data transport formats, which interact with other SeaDataNet standards (Vocabularies, Quality Flag Scale) and analysis & presentation tools (ODV, DIVA)
The following formats have been defined:SeaDataNet ODV ASCII for profiles, time series and trajectoriesSeaDataNet MedAtlas as optional extra format.NetCDF with CF compliance for gridded data sets
ODV and MedAtlas have been outfitted with a SeaDataNet semantic header
International cooperation is underway from SeaDataNet with the CF community for a common NetCDF format for the oceanographic and meteorological domains, including a semantic header
Authentication, Authorization and AdministrationSingle Sign On system required for access to distributed system
User’s authentication information based on personal login / password
User must register in order to get one loginWeb form to provide necessary information online user agreement on “SeaDataNet General Licence”After processing, login/password sent by email (email check)
Choice in V1 for CAS system (= Centralized)
SeaDataNet Data Policy
Authorisation based on “Roles”
User registration and registration validation process
RegistrationWeb form
SDN licence agreement + User information
SeaDataNet web portal
1 2
ValidationWeb form
+ SeaDataNetrole
Validation of user registrationand SeaDataNet role assignment
SeaDataNet user directory
User
SDN User Desk
3
3
Transmission by email
User personal identifier (login) + password
Registration request
Validation
NODC of the user’s country or SDN User Desk (default)
User directoryupdate
SeaDataNet AAA service
Interface for central log-in
SeaDataNet infrastructure V1Discovery services
EDMED - Data SetsEDMERP – Research ProjectsCSR – Cruise Summary Reports EDIOS – Monitoring systemsEDMO – Marine OrganisationsCDI – Common Data Index
PLUS
Cross search on top of the directories
SeaDataNet Discovery servicesActivities undertaken for:
Reviewing and streamlining the logical formats of each of the Directories
Expanding the number of Common vocabularies, further population and upgrading of Vocabularies Web services
Defining XML schema’s and formats, using the ISO 19115 metadata standard as basis
Defining and developing maintenance modalities for each of the Directories
SeaDataNet Discovery services
Defining and developing new User Interfaces for each of the Directories
Defining and implementing XML Validation Web services, that will be used to validate XML output from data centres, before import into the public Directories
Developing Web services for the Directories
Vocabs
EDMO
…
…
……
CDI EDMERP
CSR
EDMED
EDIOS
SeaDataNet : Formats review Review of the format and use of common vocabularies for
each of the Directories to achieve harmonisation and integration, and paving the path to data access via the CDI
Data Access
…
SeaDataNet XML Schema’s The ISO 19115 content model is the basis for the XML formats and exchange schema’s (XSD).
Guidance document has been prepared on how to use XML for SeaDataNet, including how to declare references to Common Vocabularies, EDMO and EDMERP
For each Directory has been prepared:Description of the format and XML tagsXML SchemaXML example file
Extended XML Schema’s have been prepared, using Schematron and OCL to support the checking of mandatory fields, use of codes from the Common Vocabularies and use of organization codes from EDMO. These schema’s are used in the XML Validation Web services
SeaDataNet Maintenance modalities• Maintenance:
Depending on the Directory, the following maintenance modalities are provided:
Online maintenance via online Content Management System XML export from local system
Local XML export can be produced by partners via:own softwareusing the MIKADO Java tool, that has been developed for entering and editing partner’s shares of the V1 Directories. MIKADO interacts with the Web services of the vocabularies, EDMO and EDMERP and produces valid XML files, that can be imported into the central V1 Directories.MIKADO also includes functionality for coupling to partner’s local database(s) for generating CDI XML files in bulk
SeaDataNet MIKADO Java tool Available under multiple environments :
Microsoft : Windows 2000, XP, VISTAAPPLEUnix - SolarisLinux
MIKADOJava code
Native DriversMYSQL
ORACLEPOSTGRESSQLServer
Bridge Driversusing Microsoft ODBC
(ACCESS, EXCEL, SQL SERVER)
DATABASE
JDBCJava DataBase Connectivity
EXCEL File
XML filesfor SeaDataNet
catalogues
CSREDMEDEDMERPCDI[EDIOS]
Manual
Automatic
Other DriversDownloaded from ad hoc Websites (Copied in the dist/lib MIKADO directory)
SeaDataNet Maintenance modalities
Directory Online CMS by partners
XML exchange via use of MIKADO
atpartners
XML exchange
via local system
at partners
Online CMS bymoderator
XML Validationservice
EDMED(BODC =
authority)
SDN DC’s initial contentconversion freetext => vocabsvia ‘sandbox’
Stand-alone incl. local storage +QC – loop for contentSynchronisation
Generated from in-house system
Validation ofXML syntaxand use ofVocabs,
EDMOand EDMERP
EDMERP(MARIS =
authority)
SDN DC’s manage theirnational records +‘sandbox’ forinstitutes
Stand-alone incl. local storage
Generated from in-house system
Validation ofXML syntaxand use ofVocabs,
EDMOand EDMERP
CSR(BSH =
authority)
Online entries by chief scientists
Stand-alone incl.local storage
Generatedfrom in-housesystem
Validation ofXML syntaxand use ofVocabs,
EDMOand EDMERP
SeaDataNet Maintenance modalities
Directory Online CMS by partners
XML exchange via use of MIKADO at partners
XML exchange via local system at partners
Online CMS by moderator
XML Validation service
CDI(MARIS = authority)
Tool embedded inlocal system
Generated from in-house system
Validation ofXML syntaxand use ofVocabs, EDMOand EDMERP
EDIOS(BODC = authority)
XML SCHEMA NOT YET READY
SDN DC’s initial revision + new entries via ‘sandbox’
Stand-alone incl. local storage
NOT YET READY
Generated from in-house system
NOT YET READY
Validation of XML syntax and use of Vocabs, EDMO and EDMERP
EDMO(MARIS = authority)
SDN DC’s manage their national records
Vocabularies(BODC = authority)
BODC with SeaVox governance mailing list
User Register + AAA service(IFREMER = authority)
SDN DC’s manage their national records
SeaDataNet Content Management Systems
EDMERP
CSREDMED
EDMO
SeaDataNet new User Interfaces
EDMERPCSR
EDIOS EDMO
SeaDataNet Discovery services - statusReviewing and streamlining the logical formats of each of the Directories – READY
Expanding the number of Common vocabularies, further population and upgrading of Vocabularies Web services – READY
Defining XML schema’s and formats, using the ISO 19115 metadata standard as basis – READY, EXCEPT FOR EDIOS
Defining and developing maintenance modalities for each of the Directories:
EDMO, EDMERP, EDMED and CSR – READY VIA ONLINE CMSEDMED, EDMERP, CSR and CDI – READY VIA MIKADO UPGRADECommon Vocabularies and AAA - READY VIA MASTER CMSIMPORT OF XML FOR CDI READYIMPORT OF XML FOR EDMERP, CSR, and EDMED ALMOST READYEDIOS – VIA MIKADO AND XML IMPORT NOT YET READY, WAITS FOR XML SCHEMA
Defining and implementing XML Validation Web services, that will be used to validate XML output from data centres, before import into the public Directories – READY FOR CDI, EDMED, EDMERP and CSR
SeaDataNet Discovery services - status
Defining and implementing new User Interfaces READY FOR EDMO, CDI, EDMERP, CSR, EDIOS, Common Vocabularies and AAA servicesUNDER DEVELOPMENT FOR EDMED CROSS SEARCH DEVELOPMENT STARTS WHEN ALL ARE READY
Content upgrading from V0 to V1 – Activity by all NODC’s EDMO READY EDMERP and CSR WELL UNDERWAY – DEADLINE NOV 08EDMED START VERY SOON – DEADLINE DEC 08 EDIOS – so far done for United Kingdom and underway for Black Sea countries; others waiting for MIKADO extension
Implementing Web services EDMO, EDMERP, Common Vocabularies, Validator and AAA services READY
Note: CDI upgrade is undertaken as part of the Data Access activities;
SeaDataNet : URLsURL for CSR V1 Retrieval:
http://seadata.bsh.de/csr/retrieve/V1_index.html
URL for CSR V1 CMS
http://seadata.bsh.de/csr/online/V1_index.html---------------------------URL for EDIOS V1 Retrieval
http://seadatanet.maris2.nl/v_edios/search.asp
---------------------------URL for EDMED V1 CMS
http://seadatanet.maris2.nl/vu_edmed/welcome.asp
SeaDataNet : URLsEDMERP Retrieval
http://seadatanet.maris2.nl/v_edmerp/search.asp
EDMERP CMS
http://seadatanet.maris2.nl/vu_edmerp/welcome.asp
------------------------
EDMO V1 retrieval
http://seadatanet.maris2.nl/edmo/
EDMO V1 CMS
http://seadatanet.maris2.nl/vu_organisations/welcome.asp
SeaDataNet Delivery Services
See Separate presentation
top related