Step 1 - Select and instantiate an Ellip Workflows archetype¶

Ellip Workflows archetypes¶

Ellip Workflows archetypes are like templates that avoid the “white page syndrome”.

The Ellip Workflows archetypes can be used form implementing the data pipelines data transformation application.

At this stage there are three groups of archetypes:

Sentinel Application Platform (SNAP) archetypes the use a SNAP graph to process one or more EO input data
Orfeo ToolBox (OTB) archetypes that invoke OTB applications using the Python bindings
Jupyter Notebooks that use notebooks as streaming executables

SNAP archetypes¶

The SNAP archetypes are Ellip Workflows templates that provide you with a ready-to run application to process EO data (Sentinel-1 and 2, Landsat-8, COSMO-SkyMed, RADARSAT-2 and TerraSAR-X) with a default SNAP graph.

The SNAP archetypes can thus be used to develop new SNAP application where only a few customisation steps are required:

Provide a compatible graph tested locally for example
Expose as many graph parameters as desired for the fine grain processing scenarios

All these archetypes are written in bash.

The SNAP archetypes include:

Sentinel-1 one-to-one

Sentinel-1 products processed in parallel with the SNAP graph taking one Sentinel-1 product and producing one result per Sentinel-1 input.

The graph provided in this archetype does the terrain correction of a Sentinel-1 product

Sentinel-1 pair

One pair of Sentinel-1 products processed to generate a single result.

The graph provided in this archetype generates an interferogram.

Sentinel-2 one-to-one

Sentinel-2 products processed in parallel with the SNAP graph taking one Sentinel-2 product and producing one result per Sentinel-2 input

The graph provided in this archetype does a K-Means Cluster Analysis of the Sentinel-2 products provided as inputs.

COSMO-SkyMed pair

One pair of COSMO-SkyMed products processed to generate a single result.

The graph provided in this archetype generates an interferogram.

RADARSAT-2 pair

One pair of RADARSAT-2 products processed to generate a single result.

The graph provided in this archetype generates an interferogram.

TerraSAR-X pair

One pair of TerraSAR-X products processed to generate a single result.

The graph provided in this archetype generates an interferogram.

OrfeoToolbox (OTB) archetypes¶

The OTB archetypes are Ellip templates that provide you with a ready-to run application to process EO data (Sentinel-1 and 2, Landsat-8, COSMO-SkyMed, RADARSAT-2 and TerraSAR-X) with a single or a sequence of OTB applications.

All these archetypes are written in Python.

The OTB archetypes can thus be used to develop new OTB application where only a few customisation steps are required:

Define what OTB steps to apply to EO data
Expose as many parameters as desired for the fine grain processing scenarios

The OTB archetypes include:

Landsat 8 one-to-one

Landsat-8 products processed in parallel with a sequence of OTB applications taking one Landsat-8 product and producing one result per Landsat-8 input.

The application provided here calculates the NDVI vegetation index with Landsat-8 OLI data. This archetypes uses the OTB application XML to define the operations.

Sentinel-1 one-to-one

Sentinel-1 products processed in parallel with a sequence of OTB applications taking one Sentinel-1 product and producing one result per Sentinel-1 input.

The application provided here does the SAR Deburst of a Sentinel-1 product.

Sentinel-2 one-to-one

Sentinel-2 products processed in parallel with a sequence of OTB applications taking one Sentinel-2 product and producing one result per Sentinel-2 input.

The application provided here does a K-Means Classification of the Sentinel-2 products provided as inputs.

Pleiades one-to-one

Pléiades products processed in parallel with a sequence of OTB applications taking one Pléiades product and producing one result per Pléiades input.

The application provided here does the optical calibration and adds an alpha band for visualisation purposes.

Kompsat-2 one-to-one

Kompsat-2 products processed in parallel with a sequence of OTB applications taking one Kompsat-2 product and producing one result per Kompsat-2 input.

The application provided here does the optical calibration and performs the pansharpening for visualisation purposes.

Kompsat-3 one-to-one

Kompsat-3 products processed in parallel with a sequence of OTB applications taking one Kompsat-3 product and producing one result per Kompsat-3 input.

The application provided here does the optical calibration and performs the pansharpening for visualisation purposes.

Jupyter Notebook archetypes¶

The Jupyter Notebook archetypes are Ellip Workflow templates that allow using a Jupyter Notebook as a streaming executable, and thus such archetypes provide the functionalities to:

Use a Jupyter Notebook to process and plot EO data using toolboxes such as GDAL, OTB and SNAP
Using notebooks as streaming executables in processing services to systematically generate higher-level EO based products

The notebook can be written using any of the supported kernels. Today these are:

Python 2
R

The archetypes based on Jupyter Notebook are split into two groups:

Stage-in: EO data references are staged-in locally, before instantiating and executing the streaming notebook. Two templates: n-to-n and m-to-1
No Stage-in: It implies doing the stage-in within the streaming notebook (it only provides the references). Two templates: n-to-n and m-to-1

The Jupyter Notebook archetypes include:

No data stage-in, one-to-one

This template requires doing the stage-in in the streaming notebook and is meant for processing one EO product during the notebook execution (n to n).

no data stage-in, many-to-one

This template this template requires doing the stage-in in the streaming notebook and is meant for processing all EO products during the notebook execution (m to 1).

one-to-one

This template is meant for processing one EO product during the notebook execution (n to n).

many-to-one

This template is meant for processing all EO products during the notebook execution (m to 1).

Complete list of Ellip Workflows archetypes¶

The full list of Ellip Workflows archetypes is:

ewf-bash-archetype Archetype for Ellip Workflows Bash Applications - v2
ewf-python-archetype Archetype for Ellip Workflows Python Applications - v2
ewf-R-archetype Archetype for Ellip Workflows R Applications - BETA
ewf-snap-S1-one-to-one Ellip Workflows using SNAP and Sentinel-1 (one-to-one)
ewf-snap-S1-pair Ellip Workflows using SNAP and Sentinel-1 (pair of products)
ewf-snap-S1-stack Ellip Workflows using SNAP and Sentinel-1 (stack of products)
ewf-snap-S2-one-to-one Ellip Workflows using SNAP and Sentinel-2 (one-to-one)
ewf-snap-CSK-ifg Ellip Workflows using SNAP for COSMO-SkyMed interferogram generation
ewf-snap-RS2-ifg Ellip Workflows using SNAP for RADARSAT-2 interferogram generation
ewf-snap-TSX-ifg Ellip Workflows using SNAP for TerraSAR-X interferogram generation
ewf-otb-S1-one-to-one Ellip Workflows using OTB and Sentinel-1 (one-to-one)
ewf-otb-S2-one-to-one Ellip Workflows using OTB and Sentinel-2 (one-to-one)
ewf-otb-L8-one-to-one Ellip Workflows using OTB and Landsat-8 (one-to-one)
ewf-otb-K2-one-to-one Ellip Workflows using OTB and KOMPSAT-2 (one-to-one)
ewf-otb-K3-one-to-one Ellip Workflows using OTB and KOMPSAT-3 (one-to-one)
ewf-notebook-1 Ellip Workflows using Jupyter Notebook (no stage-in, one-to-one)
ewf-notebook-2 Ellip Workflows using Jupyter Notebook (no stage-in, many-to-one)
ewf-notebook-stagein-1 Ellip Workflows using Jupyter Notebook with data stage-in (one-to-one)
ewf-notebook-stagein-2 Ellip Workflows using Jupyter Notebook with data stage-in (many-to-one)

Ellip Workflows using Jupyter Notebook for BETTER data transformation application development¶

Instantiating an archetype¶

Use the Jupyter Launcher to open a Terminal and type:

cd /workspace
mvn -q archetype:generate

The output will show the list of the available archetypes and ask you to choose one using its number:

Ellip Workflows archetypes

Select the ewf-notebook-stagein-1 Ellip Workflows using Jupyter Notebook with data stage-in (one-to-one) archetype
Follow the wizard to fill the information:

groupId: this is the affiliation domain, set the value to eu.ec-better

artifactId: this is the name of the data pipeline transformation application which must be same as the software repository (upcoming step 2). Set the value to wfp-01-01-01-n where ‘n’ is the letter assigned for your training

Note: The artifactId values must be lower-case

version: set the application snapshot version, set the value to 0.1-SNAPSHOT

package: same as groupId, set the value to eu.ec-better

community: set the value to ec-better

description: a short description, Sentinel-1 backscatter timeseries

summary: a short summary for your application, Sentinel-1 backscatter timeseries

Confirm the values with Y

Note: This information can be updated in a later stage by editing the pom.xml file

Under /workspace/wfp-01-01-01-n (where ‘n’ is the letter assigned for your training) there’s now a data transformation application structure:

src/main/app-resources with notebook/run, the wrapping streaming executable, a python script that instantiates, parametrizes and invokes the Jupyter notebook and finally publishes the generated results (depending on the template it stages-in the inputs) - no edits here!
src/main/app-resources/notebook/libexec/input.ipynb, the Jupyter Notebook streaming executable
src/test with the unit tests for the application

Ellip Workflows archetype instantiated for the wfp-01-01-01 data pipeline

Step 2 - Set the software repository of the tutorial puts these files in a GitLab software repository, ** Step 3 - Stage-in the EO data** will download a Sentinel-1 product and Step 4 - Implement the data transformation steps shows how to implement the Sentinel backscatter extraction and process the downloaded Sentinel-1 product