better-satcen-00001 data pipeline results (Sentinel-2 Vegetation and Water themtic index) : metadata loading and exporting (pickle file)

This Notebook shows how to load in memory (no data is actually stored into the local disk) the better-satcen-00001 metadata pipeline results

Result selection is done filtering by an AOI and a time range

Retrieved metadata frame is saved into a *pickle file* in order to allow the user working with it outside the platform

In [1]:

import pandas as pd
from geopandas import GeoDataFrame
import cioppy
import gdal
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from PIL import Image
%matplotlib inline

from urlparse import urlparse
import requests

Set the data pipeline catalogue access point

In [2]:

series = 'https://catalog.terradue.com/better-satcen-00001/series/results/description'

Area of Interest

In [3]:

wkt = 'POLYGON((21.29611111111111 39.58638888888889, 21.29611111111111 41.032, 19.89788888888889 41.032, 19.89788888888889 39.58638888888889, 21.29611111111111 39.58638888888889))'

Time of interest

  • Data selection can be done filtering on start and stop and/or update dates of the datasets:

    leave the update param empty to filter only by start/stop dates

    or

    leave start/stop params empty to filter by update dates

In [4]:

start_time = '2018-05-01T00:00:00Z'
stop_time = '2018-05-31T23:59:59.99Z'

update=''

In [5]:

search_params = dict([('start', start_time),
                      ('stop', stop_time),
                      ('update',update),
                      ('geom', wkt),
                      ('count', 900)])

Create the dataframe

We create the data frame with the following metadata: * enclosure - the URL reference to access the actual data for the download * identifier - the unique item identifier in the catalogue * self - Link to resource that identifies the resource in the catalogue * startdate - Start time of the dataset (UTC ISO 8601)
* enddate - End time of the dataset (UTC ISO 8601) * title - Title of the item, containing the string of the data filename
In [6]:

ciop = cioppy.Cioppy()

search = GeoDataFrame(ciop.search(end_point=series,
                                  params=search_params,
                                  output_fields='enclosure,identifier,self,startdate,enddate,title,wkt',
                                  timeout='300000',
                                  model='GeoTime'))
  • Showing the very 5 first results of our search
In [7]:

search.head()

Out[7]:
enclosure enddate identifier self startdate title wkt
0 https://store.terradue.com/better-satcen-00001... 2018-05-22T09:30:29.0270000Z 0712a0fdc17bad31138dfef3f6d57a6cb66962ea https://catalog.terradue.com//better-satcen-00... 2018-05-22T09:30:29.0270000Z S2B_MSIL2A_20180522T093029_N0207_R136_T34TEK_2... POLYGON((20.9997634361085 39.661229210097,21.2...
1 https://store.terradue.com/better-satcen-00001... 2018-05-22T09:30:29.0270000Z 07fb7271ad6b5c5d2b23ff305c617c0a532926bc https://catalog.terradue.com//better-satcen-00... 2018-05-22T09:30:29.0270000Z S2B_MSIL2A_20180522T093029_N0207_R136_T34TEK_2... POLYGON((20.9997634361085 39.661229210097,21.2...
2 https://store.terradue.com/better-satcen-00001... 2018-05-22T09:30:29.0270000Z 15412a409b5f269b90a6c4edd71a70355830ed6f https://catalog.terradue.com//better-satcen-00... 2018-05-22T09:30:29.0270000Z S2B_MSIL2A_20180522T093029_N0207_R136_T34TDL_2... POLYGON((19.8899808364281 40.5626923867771,21....
3 https://store.terradue.com/better-satcen-00001... 2018-05-22T09:30:29.0270000Z 238107f531ed150ebb3ee0763d3d690ec791b305 https://catalog.terradue.com//better-satcen-00... 2018-05-22T09:30:29.0270000Z S2B_MSIL2A_20180522T093029_N0207_R136_T34TEK_2... POLYGON((20.9997634361085 39.661229210097,21.2...
4 https://store.terradue.com/better-satcen-00001... 2018-05-22T09:30:29.0270000Z 2eebf44a805a2de437e839a6245260e6c2449948 https://catalog.terradue.com//better-satcen-00... 2018-05-22T09:30:29.0270000Z S2B_MSIL2A_20180522T093029_N0207_R136_T34TEK_2... POLYGON((20.9997634361085 39.661229210097,21.2... 
In [8]:

print '%s results have been retrieved' %len(search)

495 results have been retrieved
  • Formatting date time
In [9]:

search['startdate'] = pd.to_datetime(search['startdate'])
search['enddate'] = pd.to_datetime(search['enddate'])

In [10]:

search.head()


Out[10]:
enclosure enddate identifier self startdate title wkt
0 https://store.terradue.com/better-satcen-00001... 2018-05-22 09:30:29.027 0712a0fdc17bad31138dfef3f6d57a6cb66962ea https://catalog.terradue.com//better-satcen-00... 2018-05-22 09:30:29.027 S2B_MSIL2A_20180522T093029_N0207_R136_T34TEK_2... POLYGON((20.9997634361085 39.661229210097,21.2...
1 https://store.terradue.com/better-satcen-00001... 2018-05-22 09:30:29.027 07fb7271ad6b5c5d2b23ff305c617c0a532926bc https://catalog.terradue.com//better-satcen-00... 2018-05-22 09:30:29.027 S2B_MSIL2A_20180522T093029_N0207_R136_T34TEK_2... POLYGON((20.9997634361085 39.661229210097,21.2...
2 https://store.terradue.com/better-satcen-00001... 2018-05-22 09:30:29.027 15412a409b5f269b90a6c4edd71a70355830ed6f https://catalog.terradue.com//better-satcen-00... 2018-05-22 09:30:29.027 S2B_MSIL2A_20180522T093029_N0207_R136_T34TDL_2... POLYGON((19.8899808364281 40.5626923867771,21....
3 https://store.terradue.com/better-satcen-00001... 2018-05-22 09:30:29.027 238107f531ed150ebb3ee0763d3d690ec791b305 https://catalog.terradue.com//better-satcen-00... 2018-05-22 09:30:29.027 S2B_MSIL2A_20180522T093029_N0207_R136_T34TEK_2... POLYGON((20.9997634361085 39.661229210097,21.2...
4 https://store.terradue.com/better-satcen-00001... 2018-05-22 09:30:29.027 2eebf44a805a2de437e839a6245260e6c2449948 https://catalog.terradue.com//better-satcen-00... 2018-05-22 09:30:29.027 S2B_MSIL2A_20180522T093029_N0207_R136_T34TEK_2... POLYGON((20.9997634361085 39.661229210097,21.2...

Save the metadataframe

Saving the dataframe containing metadata as a pickle allows working with it outside the platform

In [11]:

search.to_pickle('better-satcen-00001_May_2018_Albania-GreeceBorder.pkl')