Hands-On Exercise 9: using an OpenSearch catalogue¶

In this exercise we will use an OpenSearch [1] catalogue URL as the input source of the workflow, and query the catalogue with OpenSearch parameters to get the input products.

Prerequisites¶

You have cloned the Hands-On git repository (see Clone the Hands-On repository),
(Only for python) You have installed the required software (see Prerequisites when using python).

Install the Hands-On¶

Install the Hands-On Exercise 9, just type:

cd
cd dcs-hands-on
mvn clean install -D hands.on=9 -P bash

Inspect the application.xml¶

Inspect the application.xml, it is slightly different than the one of the previous exercise a multi-node workflow:

<?xml version="1.0" encoding="UTF-8"?>
<application id="beam_arithm">
  <jobTemplates>
    <!-- BEAM BandMaths operator job template  -->
    <jobTemplate id="expression">
      <streamingExecutable>/application/expression/run</streamingExecutable>
      <defaultParameters>           
        <parameter id="expression">l1_flags.INVALID?0:radiance_13>15?0:100+radiance_9-(radiance_8+(radiance_10-radiance_8)*27.524/72.570)</parameter>
        <parameter id="startdate" type="opensearch" target="time:start">2012-04-06T10:24:29.000Z</parameter>
        <parameter id="enddate" type="opensearch" target="time:end">2012-04-07</parameter>
        <parameter id="qbbox" type="opensearch" target="geo:box">2.99,58.45,0.53,58.26</parameter>
      </defaultParameters>
    </jobTemplate>
    <!-- BEAM Level 3 processor job template  -->
    <jobTemplate id="binning">
      <streamingExecutable>/application/binning/run</streamingExecutable>
      <defaultParameters>           
        <parameter id="cellsize">9.28</parameter>
        <parameter id="bandname">out</parameter>
        <parameter id="bitmask">l1_flags.INVALID?0:radiance_13>15?0:100+radiance_9-(radiance_8+(radiance_10-radiance_8)*27.524/72.570)</parameter>
        <parameter id="bbox">-180,-90,180,90</parameter>
        <parameter id="algorithm">MIN_MAX</parameter>
  <parameter id="outputname">binned</parameter>
        <parameter id="resampling">binning</parameter>
        <parameter id="palette">#MCI_Palette
color0=0,0,0
color1=0,0,154
color2=54,99,250
color3=110,201,136
color4=166,245,8
color5=222,224,0
color6=234,136,0
color7=245,47,0
color8=255,255,255
numPoints=9
sample0=98.19878118960284
sample1=98.64947122314665
sample2=99.10016125669047
sample3=99.5508512902343
sample4=100.0015413237781
sample5=100.4522313573219
sample6=100.90292139086574
sample7=101.35361142440956
sample8=101.80430145795337</parameter>
        <parameter id="band">1</parameter>
        <parameter id="tailor">true</parameter>
      </defaultParameters>
      <defaultJobconf>
              <property id="ciop.job.max.tasks">1</property>
            </defaultJobconf>
    </jobTemplate>
  </jobTemplates>
    <workflow id="hands-on-9" title="Exercise, using an OpenSearch catalogue" abstract="Exercise 9, using an OpenSearch catalogue">           
    <workflowVersion>1.0</workflowVersion>
    <node id="node_expression">       
      <job id="expression"></job>     
      <sources>
        <source refid="cas:series">https://catalog.terradue.com/eo-samples/series/mer_rr__1p/description</source>
      </sources>
      <parameters>          
      </parameters>
    </node>
    <node id="node_binning">        
      <job id="binning"></job>      
      <sources>
        <source refid="wf:node">node_expression</source>        
      </sources>
      <parameters>
        <parameter id="bitmask"/>   
      </parameters>
    </node>
  </workflow>
</application>

Note the different source in the node_expression:

    <workflowVersion>1.0</workflowVersion>
    <node id="node_expression">       
      <job id="expression"></job>     
      <sources>
        <source refid="cas:series">https://catalog.terradue.com/eo-samples/series/mer_rr__1p/description</source>
      </sources>
      <parameters>          
      </parameters>
    </node>

Note also the OpenSearch parameters defined in the jobTemplate expression:

    <jobTemplate id="expression">
      <streamingExecutable>/application/expression/run</streamingExecutable>
      <defaultParameters>           
        <parameter id="expression">l1_flags.INVALID?0:radiance_13>15?0:100+radiance_9-(radiance_8+(radiance_10-radiance_8)*27.524/72.570)</parameter>
        <parameter id="startdate" type="opensearch" target="time:start">2012-04-06T10:24:29.000Z</parameter>
        <parameter id="enddate" type="opensearch" target="time:end">2012-04-07</parameter>
        <parameter id="qbbox" type="opensearch" target="geo:box">2.99,58.45,0.53,58.26</parameter>
      </defaultParameters>
    </jobTemplate>

Run and debug the workflow¶

Run the node_expression:

ciop-run node_expression

Copy the Tracking URL and paste it in a browser,
Check the log of one of the two Tasks, as described in make a robust workflow and debug it. It will be similar to:

Note that the input product is now downloaded from an external repository, resulting from the initial OpenSearch query to the catalogue, and from the way that catalogue references datasets.

Recap¶

We used an OpenSearch catalogue as the source of the first node of the workflow;
We defined a number of OpenSearch parameters to query the catalogue;
We processed the query results in the node_expression.

Footnotes

[1]	OpenSearch