.. _publication-metadata-generation :

Metadata generation
===================

Metadata is a set of information that describes actual data (in our case the outputs of a processing job) in order to make that data retrievable for further use and reprocessing. Metadata can be generated in two ways:

* By using an automated generic process that extracts metadata automatically, the recast process, or
* Manually with more specific knowledge of the product contents and purpose.

A mixed approach by which the data is recast and then manually refined is also possible.



.. _publication-recast :

Using the recast service
-------------------------

Recasting is the process of automatically extracting metadata from output files of processing jobs or uploaded data in order to make them searchable and retrievable for users or other applications and further processing. It also performs an analysis and publication of the data in order to ensure correct visualization on the platform.

This is done by the recast web service (compliant with the OGC WPS standard) at the following URL:
https://recast.terradue.com/t2api/ows?service=WPS&request=DescribeProcess&version=1.0.0&identifier=analyzeResults&status=true

It requires the following input parameters:

* ``repoKey``: The name of the repository.
  
  - e.g. *eo-samples*

* ``folderPath``: The path to the folder (relative to the repository root) containing the files to be processed.
  
  - e.g. *data-publication-sample/data-publication-sample.tif*

* ``mode``: The recasting mode, one of the values *none*, *basic* or *extended*.
  
  - Set it to *extended* in order to create also a map overlay (if possible) to be visualised in the GeoBrowser map tool.

* ``_T2Username``: The name of the Terradue platform user performing the recast operation (requires reading access to the repository containing the products).

* ``_T2ApiKey``: The corresponding Terradue platform API key.
  You can get it from your account profile on terradue.com



Using the web service with Python is very simple; below is the relevant code snippet to show the basic usage:

.. code-block:: python

    # Set WPS server information
    wps_url = "https://recast.terradue.com/t2api/ows"
    wps_process_id = "analyzeResults"

    # Define parameters

    repo_name = '...'         # the name of the repository on the Terradue storage containing the data to be analysed
    folder_path = '...'       # the folder within that repository to be analysed
    username = '...'          # Ellip username (see above)
    api_key = '...'           # corresponding API key

    wps_inputs = [
        ('repoKey', repo_name),
        ('folderPath', folder_path),
        ('mode', 'extended'),
        ('_T2Username', username),
        ('_T2ApiKey', api_key)
    ]

    # Get WPS instance
    wps = WebProcessingService(wps_url)

    # Invoke recast process
    wps_execution = wps.execute(wps_process_id, 
                                wps_inputs, 
                                output = [('result_osd', True)]
    )

    # Poll job status until job finishes (takes a few seconds)
    monitorExecution(wps_execution, sleepSecs=10)



The following Jupyter notebook demonstrates how to use Python code to generate a metadata file for a product via the recast service:

`Publish metadata on the Terradue catalog <notebooks/publish-metadata.ipynb>`_ (up to step 2).


.. _publication-metadata-generation-manual :

Alternative: Fully manual metadata generation
---------------------------------------------

The metadata we need to generate is an ATOM feed containing entries for individual products. A product often consist of several files (if not packaged as a zipped archive), and those files are enclosure links within the same entry.


.. code-block:: xml

    <feed xmlns="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:eop="http://www.opengis.net/eop/2.1">
    <entry>
        <title type="text">data-publication-sample</title>
        <summary type="html">data-publication-sample</summary>
        <published>2019-04-23T13:16:10.643Z</published>
        <updated>2019-04-24T08:45:54.447055Z</updated>
        <link rel="enclosure" type="image/tiff" title="Product GeoTiff" length="31624950" href="https://store.terradue.com/myindex/data-publication-sample/data-publication-sample.tif"/>
        <license xmlns="http://www.w3.org/2005/Atom">https://creativecommons.org/licenses/by/4.0</license>
        <eop:productType>SAMPLE_TYPE</eop:productType>
        <georss:polygon xmlns:georss="http://www.georss.org/georss">40.604756 13.587901 41.234180 13.737746 41.077912 14.856833 40.448583 14.695978 40.604756 13.587901</georss:polygon>
        <dc:date>2017-02-11T05:10:43.490699Z/2018-01-01T05:10:51.413166Z</dc:date>
        <dc:identifier>data-publication-sample</dc:identifier>

        <eop:EarthObservation xmlns:gml="http://www.opengis.net/gml/3.2" xmlns:om="http://www.opengis.net/om">
        ...
        </eop:EarthObservation>
    </entry>
    </feed>


The ``<eop:EarthObservation>`` node contains the most important information, and there can also be specialised subtypes for different types of data (e.g. RADAR or optical products). Below is an example for SAR metadata:

.. code-block:: xml

    <sar:EarthObservation xmlns:eop="http://www.opengis.net/eop/2.1" xmlns:gml="http://www.opengis.net/gml/3.2" xmlns:om="http://www.opengis.net/om" xmlns:sar="http://www.opengis.net/sar/2.1">
      <phenomenonTime xmlns="http://www.opengis.net/om/2.0">
        <gml:TimePeriod>
          <gml:beginPosition>2017-02-11T05:10:43.490699Z</gml:beginPosition>
          <gml:endPosition>2018-01-01T05:10:51.413166Z</gml:endPosition>
        </gml:TimePeriod>
      </phenomenonTime>
      <procedure xmlns="http://www.opengis.net/om/2.0">
        <eop:EarthObservationEquipment>
          <eop:platform>
            <eop:Platform>
              <eop:shortName>PLATFORM</eop:shortName>
            </eop:Platform>
          </eop:platform>
          <eop:instrument>
            <eop:Instrument>
              <eop:shortName>INSTRUMENT</eop:shortName>
              <eop:description>Sample Instrument</eop:description>
            </eop:Instrument>
          </eop:instrument>
          <eop:sensor>
            <eop:Sensor>
              <eop:sensorType>RADAR</eop:sensorType>
              <eop:operationalMode>IW</eop:operationalMode>
            </eop:Sensor>
          </eop:sensor>
          <eop:acquisitionParameters>
            <sar:Acquisition>
              <eop:orbitNumber/>
              <eop:lastOrbitNumber/>
              <eop:wrsLongitudeGrid>22</eop:wrsLongitudeGrid>
              <sar:antennaLookDirection>Right</sar:antennaLookDirection>
            </sar:Acquisition>
          </eop:acquisitionParameters>
        </eop:EarthObservationEquipment>
      </procedure>
      <featureOfInterest xmlns="http://www.opengis.net/om/2.0">
        <eop:Footprint>
          <eop:multiExtentOf>
            <gml:MultiSurface>
              <gml:surfaceMembers>
                <gml:Polygon>
                  <gml:exterior>
                    <gml:LinearRing>
                      <gml:posList count="5">40.604756 13.587901 41.234180 13.737746 41.077912 14.856833 40.448583 14.695978 40.604756 13.587901</gml:posList>
                    </gml:LinearRing>
                  </gml:exterior>
                </gml:Polygon>
              </gml:surfaceMembers>
            </gml:MultiSurface>
          </eop:multiExtentOf>
        </eop:Footprint>
      </featureOfInterest>
      <result xmlns="http://www.opengis.net/om/2.0">
        <eop:EarthObservationResult>
          <eop:browse>
            <eop:BrowseInformation>
              <eop:referenceSystemIdentifier>EPSG4326</eop:referenceSystemIdentifier>
              <eop:fileName>
                <ServiceReference xmlns:p11="http://www.w3.org/1999/xlink" xmlns:p10="http://www.opengis.net/ows/2.0" xmlns="http://www.opengis.net/ows/2.0" p10:type="simple" p11:href="https://store.terradue.com/myindex/data-publication-sample/data-publication-sample.png">
                  <p10:RequestMessage/>
                </ServiceReference>
              </eop:fileName>
            </eop:BrowseInformation>
          </eop:browse>
          <eop:product>
            <eop:ProductInformation>
              <eop:referenceSystemIdentifier>EPSG4326</eop:referenceSystemIdentifier>
              <eop:fileName>
                <ServiceReference xmlns:p11="http://www.w3.org/1999/xlink" xmlns:p10="http://www.opengis.net/ows/2.0" xmlns="http://www.opengis.net/ows/2.0" p10:type="simple" p11:href="https://store.terradue.com/myindex/data-publication-sample/data-publication-sample.tif" p11:title="simple">
                  <p10:RequestMessage/>
                </ServiceReference>
              </eop:fileName>
              <eop:size>31624950</eop:size>
            </eop:ProductInformation>
          </eop:product>
          <eop:parameter>
            <eop:ParameterInformation>
              <eop:unitOfMeasure uom="N/A, deg, deg, m, cm/yr, N/A, N/A, N/A, N/A, cm"/>
            </eop:ParameterInformation>
          </eop:parameter>
        </eop:EarthObservationResult>
      </result>
      <eop:metaDataProperty>
        <eop:EarthObservationMetaData>
          <eop:identifier>data-publication-sample</eop:identifier>
          <eop:acquisitionType>OTHER</eop:acquisitionType>
          <eop:productType>SAMPLE_TYPE</eop:productType>
          <eop:status>ARCHIVED</eop:status>
          <eop:statusSubType>ON-LINE</eop:statusSubType>
          <eop:imageQualityStatus>NOMINAL</eop:imageQualityStatus>
          <eop:productQualityStatus>DEGRADED</eop:productQualityStatus>
          <eop:processing>
            <eop:ProcessingInformation>
              <eop:processingCenter>Sample processing centre</eop:processingCenter>
              <eop:processingDate>2018-05-22T23:32:23Z</eop:processingDate>
              <eop:method>Sample method</eop:method>
              <eop:processorName>Sample processor</eop:processorName>
              <eop:processorVersion>2.5</eop:processorVersion>
              <eop:processingLevel>L2</eop:processingLevel>
              <eop:nativeProductFormat>ASCII</eop:nativeProductFormat>
            </eop:ProcessingInformation>
          </eop:processing>
          <eop:vendorSpecific>
            <eop:SpecificInformation>
              <eop:localAttribute>verticalResolution</eop:localAttribute>
              <eop:localValue>37</eop:localValue>
            </eop:SpecificInformation>
          </eop:vendorSpecific>
        </eop:EarthObservationMetaData>
      </eop:metaDataProperty>
    </sar:EarthObservation>

.. note:: Vendor-specific information for which there are no standard elements in the metadata schema has to go into the ``<vendorSpecific>`` node within the ``<EarthObservationMetadata>`` element (see at the bottom of content above).