Personal index

In this part, we will discover how to publish our own entries in the catalog.

Personal index management

Every user of the platform has its own index called with the username (e.g user Mario Rossi [username: mrossi] has an index called mrossi).

This index should be created automatically when the user is registered on the T2 Cloud Platform. You can check the existence of your own index by querying the description deocument of it.

curl -u mrossi:ABcdEF "https://catalog.terradue.com/mrossi/description"

Note

replace mrossi with your own username

If, for some reason, your index does not exist yet, you can simply create it by yourself with the following command.

curl -X PUT -u mrossi:ABcdEF "https://catalog.terradue.com/mrossi"

A successful command should return

  {
    "name": "mrossi_v1",
    "shards": {
        "total": 10,
        "successful": 10,
        "failed": 0
    },
    "types": [
        {
            "name": "series",
            "version": 2,
            "description": "Type representing a series"
        },
        {
            "name": "gtfeature",
            "version": 2,
            "description": "Type representing a feature geo and timed"
        },
        {
            "name": "eopfeature",
            "version": 2,
            "description": "Type representing an Earth Observation feature"
        },
        {
            "name": "eopseries",
            "version": 2,
            "description": "Type representing a EarthObservation series"
        },
        {
            "name": "eoporbit",
            "version": 1,
            "description": "Type representing an orbit information"
        },
        {
            "name": "eoppoint",
            "version": 2,
            "description": "Type representing an orbit information"
        }
    ]
}

Index Metadata update

Note

coming soon

Now your index ready, you can start publishing entries in the catalog. The following section describes how to create, update or delete entries in your index.

Create entries

As described in the Metadata model section, one or more entries are represented in a feed and this is this feed you will post to the catalog to be indexed. The supported formats of this feed are listed in the Content Type section but here we will always use Atom.

Let’s insert our first entry. We then write a file on your local file system (e.g. myfirstfeed.atom) with our first feed:

<feed xmlns="http://www.w3.org/2005/Atom">
  <title type="text">My first feed</title>
  <id>test</id>
  <entry>
    <id>entry1</id>
    <identifier xmlns="http://purl.org/dc/elements/1.1/">S2A_OPER_MSI_L1C_TL_SGS__20160625T170310_A005267_T32TQM_N02.04</identifier>
    <title type="text">Imager - Sentinel2 - 2016-06-25T10:06:17</title>
    <summary type="html">
      &lt;table&gt;
        &lt;tr&gt;&lt;td&gt;&lt;strong&gt;Mission&lt;/strong&gt;&lt;/td&gt;&lt;td&gt;Sentinel-2&lt;/td&gt;&lt;/tr&gt;
        &lt;tr&gt;&lt;td&gt;&lt;strong&gt;Orbit&lt;/strong&gt;&lt;/td&gt;&lt;td&gt;005267 DESCENDING&lt;/td&gt;&lt;/tr&gt;
        &lt;tr&gt;&lt;td&gt;&lt;strong&gt;Track&lt;/strong&gt;&lt;/td&gt;&lt;td&gt;122&lt;/td&gt;&lt;/tr&gt;
        &lt;tr&gt;&lt;td&gt;&lt;strong&gt;Cloud Cover&lt;/strong&gt;&lt;/td&gt;&lt;td&gt;0.1674&lt;/td&gt;&lt;/tr&gt;
        &lt;tr&gt;&lt;td&gt;&lt;strong&gt;Date&lt;/strong&gt;&lt;/td&gt;&lt;td&gt;2016-06-25T10:06:17.4130000Z&lt;/td&gt;&lt;/tr&gt;
      &lt;/table&gt;
    </summary>
    <date xmlns="http://purl.org/dc/elements/1.1/">2016-06-25T10:06:17.4130000Z</date>
    <spatial xmlns="http://purl.org/dc/terms/">POLYGON((11.430710453849 42.4269119937756,12.7055623149861 42.4269119937756,12.7055623149861 41.4040347436697,11.430710453849 41.4040347436697,11.430710453849 42.4269119937756))</spatial>
    <published>2014-12-05T20:58:38.024Z</published>
    <link rel="enclosure" type="application/octet-stream" length="4195720" href="https://store.terradue.com/" />
  </entry>
</feed>

This is a single entry feed for training purpose but it could contains many more. Let’s explain it element by element.

  • feed mandatory : this is the main container

  • feed/title optional : this is the title of the feed and is there only for educational purpose but is useless for the catalog

  • feed/id optional : this is the identifier of the feed and is there only for educational purpose but is useless for the catalog

  • feed/entry mandatory : this is the container for an entry in the catalog

  • feed/entry/id optional : this is the identifier of the entry and is there only for educational purpose but is useless for the catalog

  • feed/identifier mandatory : this is the unique identifier of the entry. This is a very important element since it will define the unique identifier of the entry in the index. Any other element with the same identifier in the same index will be overriden.

  • feed/entry/title mandatory : this is the title of the entry and is usually used by other component (e.g. portal) to display the item caption (e.g. in the list of results).

  • feed/entry/summary optional : this is the short description (abstract) of the entry. It can be set as HTML (XML encoded) as in the example but can also be a simple text. In this latter case, the attribute type will be set to “text”. Even if optional, this element is important because it is often used by the other component such as the portal to display a summary of the item. For instance, in the thematic applications of the portal, the summary is used to fill in the info bubble of the item displayed on the map.

  • feed/entry/date optional : this is one of the element used in the Geo & Time model. It defines the temporal charateristic of the entry. It may define a time instant (cf. example) or a time range. The format of the date(s) must follow ISO8601.

    <!-- Time instant -->
<date xmlns="http://purl.org/dc/elements/1.1/">2016-06-25T10:06:17.4130000Z</date>

    <!-- Time range -->
<date xmlns="http://purl.org/dc/elements/1.1/">2016-06-25T10:06:17.4130000Z/2016-06-25T10:36:17.4130000Z</date>

  • feed/entry/spatial optional : this is the other element used in the Geo & Time model. It defines the spatial charateristic of the entry. There are several way of defining the geometry of the entry:

    <!-- Well-Known-Text -->
<spatial xmlns="http://purl.org/dc/terms/">POLYGON((11.430710453849 42.4269119937756,12.7055623149861 42.4269119937756,12.7055623149861 41.4040347436697,11.430710453849 41.4040347436697,11.430710453849 42.4269119937756))</spatial>

    <!-- GeoRSS polygon -->
<georss:polygon xmlns:georss="http://www.georss.org/georss">42.4269119937756 11.430710453849 42.4269119937756 12.7055623149861 41.4040347436697 12.7055623149861 41.4040347436697 11.430710453849 42.4269119937756 11.430710453849</georss:polygon>

    <!-- GeoRSS where with GML -->
<georss:where xmlns:georss="http://www.georss.org/georss">
  <gml:MultiSurface>
    <gml:surfaceMembers>
      <gml:Polygon>
        <gml:exterior>
          <gml:LinearRing>
            <gml:posList count="5">
            42.4269119937756 11.430710453849 42.4269119937756 12.7055623149861 41.4040347436697 12.7055623149861 41.4040347436697 11.430710453849 42.4269119937756 11.430710453849
            </gml:posList>
          </gml:LinearRing>
        </gml:exterior>
      </gml:Polygon>
    </gml:surfaceMembers>
  </gml:MultiSurface>
</georss:polygon>

  • feed/entry/published optional : this is the element defining the publication date of the entry in the index. If not specified, it will be set at the time the entry is indexed in the catalog. The format of the date must follow ISO8601.

  • feed/entry/link optional : several links may be associated to an entry in the catalog. They are important references for other components of the platform. The link is set in href sttribute. The resulting content of the link is defined by the type attribute and the purpose of the link is defined by the rel attribute:

Value of rel attribute Purpose
enclosure Identifies a related resource that is potentially large and might require special handling. Usually used for the dataset download that the entry is describing. Many enclosures may be defined to specify many download point.
alternate Refers to a substitute for this entry.

Now we have our file we can send it to the catalog for indexing.

curl -u mrossi:ABcdEF -X POST -H "Content-Type: application/atom+xml" -d@myfirstfeed.atom "https://catalog.terradue.com/mrossi"

Note

Please note the Content-Type header set to application/atom+xml. This is important to indicate the catalog the Content Type of the feed posted.

The resulting response from the catalog is a JSON document reporting the actions done in the index

{
  "added": 1,
  "updated": 0,
  "deleted": 0,
  "errors": 0,
  "items": [
      {
          "id": "S2A_OPER_MSI_L1C_TL_SGS__20160625T170310_A005267_T32TQM_N02.04",
          "type": "gtfeature",
          "operation": "Add"
      }
  ]
}

You have now a new entry in your index. You can check it has been indexed correctly by making some queries

This command returns the same feed you just sent (with some more information of the catalog):

curl -u mrossi:ABcdEF "https://catalog.terradue.com/mrossi/search?uid=S2A_OPER_MSI_L1C_TL_SGS__20160625T170310_A005267_T32TQM_N02.04"

This command using opensearch-client makes a temporal and spatial search and return the download link correctly

opensearch-client -u mrossi:ABcdEF -p bbox=10,40,12,42 -p start=2016-06-24 -p stop=2016-06-26 "https://catalog.terradue.com/mrossi/search" enclosure

Note

The above example showed the insertion of one entry at a time but remember that you can send as many entries in the same feed as you want. There is only a limit of 32Mbytes maximum by feed sent.

Update entries

The update of updating entries in the index is the same as per creation. If you specify the same identifier element, the corresponding entry will be simply updated. The catalog shall return a response similar to this one:

{
  "added": 0,
  "updated": 1,
  "deleted": 0,
  "errors": 0,
  "items": [
      {
          "id": "S2A_OPER_MSI_L1C_TL_SGS__20160625T170310_A005267_T32TQM_N02.04",
          "type": "gtfeature",
          "operation": "Update"
      }
  ]
}

Delete entries

Deleting one or more entries is done as per searching for them.

To delete a single entry by reference:

curl -u mrossi:ABcdEF -X DELETE "https://catalog.terradue.com/mrossi/query?uid=S2A_OPER_MSI_L1C_TL_SGS__20160625T170310_A005267_T32TQM_N02.04

and it returns :

{
  "added": 0,
  "updated": 0,
  "deleted": 1,
  "errors": 0,
  "items": [
      {
          "id": "S2A_OPER_MSI_L1C_TL_SGS__20160625T170310_A005267_T32TQM_N02.04",
          "type": "gtfeature",
          "operation": "Delete"
      }
  ]
}

To delete many entries by query (e.g. all data of 2012):

curl -u mrossi:ABcdEF -X DELETE "https://catalog.terradue.com/mrossi/query?start=2012-01-01&stop=2012-12-31

Note

Please note that the action part of the URL is query and not search. This is on purpose mainly to avoid confusions of operations, such as an accidental deletion of products. The query operation can only be used for deleting, while search cannot be used for deleting.

Create Series

It is also possible to create set of index entries based on common fixed parameters. In order to create one this set called Series, we will also index to the catalog a feed with one or more entries that will define one or more series. Let’s start with the file seriesitaly.atom

<feed xmlns="http://www.w3.org/2005/Atom">
  <title type="text">My second feed</title>
  <id>test</id>
  <entry>
    <id>series1</id>
    <title type="text">Data over Italy</title>
    <published>2016-06-03T10:30:45.879747Z</published>
    <link rel="describedBy" type="application/atom+xml" title="search filters for Italy AOI" href="https://catalog.terradue.com/mrossi/search?geom=POLYGON((6.372 47.01,19.028 47.01,18.896 36.527,6.46 36.598,6.372 47.01))" />
    <identifier xmlns="http://purl.org/dc/elements/1.1/">italy</identifier>
  </entry>
</feed>

The description of the elements done in the Create entries section are still valid. Here are the specific elements of a series.

  • feed/identifier mandatory : this is the unique identifier of the series. This is a very important element since it will define the unique identifier of the series in the index and the route (URL) to access the series. Any other series with the same identifier in the same index will be overriden.
  • feed/entry/link[rel=’describedBy’] mandatory : Those links are specific to series, it defines one or more set that constitutes the series. In the example the link is
https://catalog.terradue.com/mrossi/search?geom=POLYGON((6.372 47.01,19.028 47.01,18.896 36.527,6.46 36.598,6.372 47.01))

this link is a spatial filter request. All index entries returned by this query will be part of the series.

Now, let’s post the series in the catalog with the following command

curl -u mrossi:ABcdEF -X POST -H "Content-Type: application/atom+xml" -d@seriesitaly.atom "https://catalog.terradue.com/mrossi/series"

and returns

{
    "added": 1,
    "updated": 0,
    "deleted": 0,
    "errors": 0,
    "items": [
        {
            "id": "italy",
            "type": "series",
            "operation": "Add"
        }
    ]
}

The series is successfully created and we can query it

opensearch-client https://catalog.terradue.com/mrossi/series/italy/search

Updating and deleting series is done using the same mechanism as per entries but inserting the /series after the index name.