IntroductionΒΆ

This tutorial is structured in a series of incremental steps with the overall goal of creating and deploying a data pipeline.

The tutorial is divided in two parts.

The first deals with the User Stories:

User Story WFP-01-01-01-US-01 Service integration

As a data processor developer, I want to implement, integrate and package an algorithm processing Sentinel-1 GRD products using the SNAP toolbox and a notebook with the following processing steps:

  • Application of orbit file (if available, but should not hold the processing of the product)
  • Border noise removal (if necessary)
  • Calibration
  • (Multi-temporal) Speckle filtering
  • Terrain correction
  • Conversion to dB

The package is available as an archive and as an execution container in my workspace with a workflow identifier and a version number.

User Story 2 WFP-01-01-01-US-02 Service deployment

As a data processor developer, I want to deploy the service package integrated in WFP-01-01-01-US-01 in the deployer using the corresponding identifier and version number. This deployment returns the Web Processing Service end-point to my application identified by the deployer, the service identifier and the version number.

Steps 1 to 9 of the tutorial address those two User Stories:

  • Step 1 - Select and instantiate an archetype
  • Step 2 - Set the software repository
  • Step 3 - Stage-in the EO data
  • Step 4 - Implement the data transformation steps
  • Step 5 - Deploy and run the Web Processing Service locally
  • Step 6 - Release the data transformation application
  • Step 7 - Prepare the application for the Production Centre
  • Step 8 - Deploy the application on the Production Centre
  • Step 9 - Run the application on the Production Centre

The second part of the tutorial address the User Stories:

User Story 3 WFP-01-01-01-US-03 Data pipeline definition

As a data pipeline developer, I want to define a data pipeline trigger for the application deployed in WFP-01-01-01-US-02, with the following configurable parameters:

Data stream unique identifier Area of interest (AOI) Time range (start date / end date) Data polling period The data pipeline trigger is saved as a service package and execution container that are deployed on the data pipeline deployer with a unique trigger identifier.

User Story 4 WFP-01-01-01-US-04 Data pipeline submission

As a data pipeline operator, I want to define a data stream for the trigger deployed in WFP-01-01-01-US-03, using the corresponding deployer and trigger identifier, with the following parameters:

Data stream unique identifier Area of interest (AOI) Time range (start date / end date) Data polling period The data pipeline is submitted as a coordinated job triggering new instances of the pipeline running as streams periodically at the frequency and for the time range specified.

Steps 10 to 14 of the tutorial address the User Stories 3 and 4:

  • Step 10 - Create the Trigger for systematic processing
  • Step 11 - Prepare the Trigger for the production centre
  • Step 12 - Deploy and run the Trigger for systematic processing
  • Step 13 - Deploy the Trigger coordinators
  • Step 14 - Configure a Thematic Application