better-satcen-00001 data pipeline results (Sentinel-2 Vegetation and Water themtic index) : metadata loading and exporting (pickle file)¶
This Notebook shows how to load in memory (no data is actually stored into the local disk) the better-satcen-00001 metadata pipeline results
Result selection is done filtering by an AOI and a time range
Retrieved metadata frame is saved into a *pickle file* in order to allow the user working with it outside the platform
In [1]:
import pandas as pd
from geopandas import GeoDataFrame
import cioppy
import gdal
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from PIL import Image
%matplotlib inline
from urlparse import urlparse
import requests
Set the data pipeline catalogue access point¶
In [2]:
series = 'https://catalog.terradue.com/better-satcen-00001/series/results/description'
Area of Interest¶
In [3]:
wkt = 'POLYGON((21.29611111111111 39.58638888888889, 21.29611111111111 41.032, 19.89788888888889 41.032, 19.89788888888889 39.58638888888889, 21.29611111111111 39.58638888888889))'
Time of interest¶
Data selection can be done filtering on start and stop and/or update dates of the datasets:
leave the update param empty to filter only by start/stop dates
or
leave start/stop params empty to filter by update dates
In [4]:
start_time = '2018-05-01T00:00:00Z'
stop_time = '2018-05-31T23:59:59.99Z'
update=''
In [5]:
search_params = dict([('start', start_time),
('stop', stop_time),
('update',update),
('geom', wkt),
('count', 900)])
Create the dataframe¶
We create the data frame with the following metadata: * enclosure -
the URL reference to access the actual data for the download *
identifier - the unique item identifier in the catalogue * self -
Link to resource that identifies the resource in the catalogue *
startdate - Start time of the dataset (UTC ISO 8601)
* enddate - End time of the dataset (UTC ISO 8601) * title -
Title of the item, containing the string of the data filename
In [6]:
ciop = cioppy.Cioppy()
search = GeoDataFrame(ciop.search(end_point=series,
params=search_params,
output_fields='enclosure,identifier,self,startdate,enddate,title,wkt',
timeout='300000',
model='GeoTime'))
- Showing the very 5 first results of our search
In [7]:
search.head()
Out[7]:
enclosure | enddate | identifier | self | startdate | title | wkt | |
---|---|---|---|---|---|---|---|
0 | https://store.terradue.com/better-satcen-00001... | 2018-05-22T09:30:29.0270000Z | 0712a0fdc17bad31138dfef3f6d57a6cb66962ea | https://catalog.terradue.com//better-satcen-00... | 2018-05-22T09:30:29.0270000Z | S2B_MSIL2A_20180522T093029_N0207_R136_T34TEK_2... | POLYGON((20.9997634361085 39.661229210097,21.2... |
1 | https://store.terradue.com/better-satcen-00001... | 2018-05-22T09:30:29.0270000Z | 07fb7271ad6b5c5d2b23ff305c617c0a532926bc | https://catalog.terradue.com//better-satcen-00... | 2018-05-22T09:30:29.0270000Z | S2B_MSIL2A_20180522T093029_N0207_R136_T34TEK_2... | POLYGON((20.9997634361085 39.661229210097,21.2... |
2 | https://store.terradue.com/better-satcen-00001... | 2018-05-22T09:30:29.0270000Z | 15412a409b5f269b90a6c4edd71a70355830ed6f | https://catalog.terradue.com//better-satcen-00... | 2018-05-22T09:30:29.0270000Z | S2B_MSIL2A_20180522T093029_N0207_R136_T34TDL_2... | POLYGON((19.8899808364281 40.5626923867771,21.... |
3 | https://store.terradue.com/better-satcen-00001... | 2018-05-22T09:30:29.0270000Z | 238107f531ed150ebb3ee0763d3d690ec791b305 | https://catalog.terradue.com//better-satcen-00... | 2018-05-22T09:30:29.0270000Z | S2B_MSIL2A_20180522T093029_N0207_R136_T34TEK_2... | POLYGON((20.9997634361085 39.661229210097,21.2... |
4 | https://store.terradue.com/better-satcen-00001... | 2018-05-22T09:30:29.0270000Z | 2eebf44a805a2de437e839a6245260e6c2449948 | https://catalog.terradue.com//better-satcen-00... | 2018-05-22T09:30:29.0270000Z | S2B_MSIL2A_20180522T093029_N0207_R136_T34TEK_2... | POLYGON((20.9997634361085 39.661229210097,21.2... |
In [8]:
print '%s results have been retrieved' %len(search)
495 results have been retrieved
- Formatting date time
In [9]:
search['startdate'] = pd.to_datetime(search['startdate'])
search['enddate'] = pd.to_datetime(search['enddate'])
In [10]:
search.head()
Out[10]:
enclosure | enddate | identifier | self | startdate | title | wkt | |
---|---|---|---|---|---|---|---|
0 | https://store.terradue.com/better-satcen-00001... | 2018-05-22 09:30:29.027 | 0712a0fdc17bad31138dfef3f6d57a6cb66962ea | https://catalog.terradue.com//better-satcen-00... | 2018-05-22 09:30:29.027 | S2B_MSIL2A_20180522T093029_N0207_R136_T34TEK_2... | POLYGON((20.9997634361085 39.661229210097,21.2... |
1 | https://store.terradue.com/better-satcen-00001... | 2018-05-22 09:30:29.027 | 07fb7271ad6b5c5d2b23ff305c617c0a532926bc | https://catalog.terradue.com//better-satcen-00... | 2018-05-22 09:30:29.027 | S2B_MSIL2A_20180522T093029_N0207_R136_T34TEK_2... | POLYGON((20.9997634361085 39.661229210097,21.2... |
2 | https://store.terradue.com/better-satcen-00001... | 2018-05-22 09:30:29.027 | 15412a409b5f269b90a6c4edd71a70355830ed6f | https://catalog.terradue.com//better-satcen-00... | 2018-05-22 09:30:29.027 | S2B_MSIL2A_20180522T093029_N0207_R136_T34TDL_2... | POLYGON((19.8899808364281 40.5626923867771,21.... |
3 | https://store.terradue.com/better-satcen-00001... | 2018-05-22 09:30:29.027 | 238107f531ed150ebb3ee0763d3d690ec791b305 | https://catalog.terradue.com//better-satcen-00... | 2018-05-22 09:30:29.027 | S2B_MSIL2A_20180522T093029_N0207_R136_T34TEK_2... | POLYGON((20.9997634361085 39.661229210097,21.2... |
4 | https://store.terradue.com/better-satcen-00001... | 2018-05-22 09:30:29.027 | 2eebf44a805a2de437e839a6245260e6c2449948 | https://catalog.terradue.com//better-satcen-00... | 2018-05-22 09:30:29.027 | S2B_MSIL2A_20180522T093029_N0207_R136_T34TEK_2... | POLYGON((20.9997634361085 39.661229210097,21.2... |
Save the metadataframe¶
Saving the dataframe containing metadata as a pickle allows working with it outside the platform
In [11]:
search.to_pickle('better-satcen-00001_May_2018_Albania-GreeceBorder.pkl')