better-wfp-00003 data pipeline results (Sentinel-2 reflectances and vegetation indices) : metadata loading and exporting (pickle file)¶
This Notebook shows how to load in memory (no data is actually stored into the local disk) the better-wfp-00003 metadata pipeline results
Result selection is done filtering by an AOI and a time range
Retrieved metadata frame is saved into a *pickle file* in order to allow the user working with it outside the platform
In [1]:
import pandas as pd
from geopandas import GeoDataFrame
import cioppy
import gdal
from shapely.wkt import loads
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from PIL import Image
%matplotlib inline
from urlparse import urlparse
import requests
Set the data pipeline catalogue access point¶
In [3]:
series = 'https://catalog.terradue.com/better-wfp-00003/series/results/description'
Area of Interest¶
In [4]:
geom = 'MULTIPOLYGON (((6.4788 14.5973, 7.5577 14.5973, 7.5577 13.6328, 6.4788 13.6328, 6.4788 14.5973)), ((67.7116 37.9032, 68.791 37.9032, 68.791 36.9211, 67.7116 36.9211, 67.7116 37.9032)), ((-10.3668 15.3471, -9.351800000000001 15.3471, -9.351800000000001 14.3406, -10.3668 14.3406, -10.3668 15.3471)), ((67.62430000000001 36.7228, 68.116 36.7228, 68.116 35.6923, 67.62430000000001 35.6923, 67.62430000000001 36.7228)))'
wkt = loads(geom)[3].wkt
Time of interest¶
Data selection can be done filtering on start and stop dates of the datasets
In [5]:
start_time = '2017-06-01T00:00:00Z'
stop_time = '2017-09-30T23:59:59.99Z'
In [6]:
search_params = dict([('start', start_time),
('stop', stop_time),
('geom', wkt),
('count', 500)])
Or filtering on update time range: the update dates are the ones related to the catalogue metadata ingestion
In [7]:
update='2019-01-02T00:00:00Z/2019-01-02T23:59:59.99Z'
In [8]:
search_params = dict([('update', update),
('geom', wkt),
('count', 900)])
*Or* filtering on start, stop and update time range:
In [9]:
search_params = dict([('start', start_time),
('stop', stop_time),
('update', update),
('geom', wkt),
('count', 900)])
Create the dataframe¶
We create the data frame with the following metadata: * enclosure -
the URL reference to access the actual data for the download *
identifier - the unique item identifier in the catalogue * self -
Link to resource that identifies the resource in the catalogue *
startdate - Start time of the dataset (UTC ISO 8601)
* enddate - End time of the dataset (UTC ISO 8601) * title -
Title of the item, containing the string of the data filename
In [10]:
ciop = cioppy.Cioppy()
search = GeoDataFrame(ciop.search(end_point=series,
params=search_params,
output_fields='enclosure,identifier,self,startdate,enddate,title,wkt',
model='GeoTime'))
- Showing the very 5 first results of our search
In [11]:
search.head()
Out[11]:
| enclosure | enddate | identifier | self | startdate | title | wkt | |
|---|---|---|---|---|---|---|---|
| 0 | https://store.terradue.com/better-wfp-00003/_r... | 2017-09-22T06:16:31.0260000Z | 2412362a3137482a3b8b692ff621761f97b40577 | https://catalog.terradue.com//better-wfp-00003... | 2017-09-22T06:16:31.0260000Z | Reproducibility notebook used for generating S... | POLYGON((67.9974286897346 35.6884876797069,67.... |
| 1 | https://store.terradue.com/better-wfp-00003/_r... | 2017-09-22T06:16:31.0260000Z | 2e66b787b9be5fe0bfe39592a28d4662debc22d4 | https://catalog.terradue.com//better-wfp-00003... | 2017-09-22T06:16:31.0260000Z | Reproducibility notebook used for generating S... | POLYGON((67.989865566835 36.0488297143412,67.9... |
| 2 | https://store.terradue.com/better-wfp-00003/_r... | 2017-09-22T06:16:31.0260000Z | 4a9471ba1e80fc8a60aa29ba1557f4ced5cf38f3 | https://catalog.terradue.com//better-wfp-00003... | 2017-09-22T06:16:31.0260000Z | Reproducibility stage-in notebook for Sentinel... | POLYGON((67.989865566835 36.0488297143412,67.9... |
| 3 | https://store.terradue.com/better-wfp-00003/_r... | 2017-09-22T06:16:31.0260000Z | 52506c40158ff7486ccfc71423748d34d5bbcc17 | https://catalog.terradue.com//better-wfp-00003... | 2017-09-22T06:16:31.0260000Z | Reproducibility stage-in notebook for Sentinel... | POLYGON((68.1160119769251 35.690459964115,68.1... |
| 4 | https://store.terradue.com/better-wfp-00003/_r... | 2017-09-22T06:16:31.0260000Z | 5711d6b38f923d4de2524b179cc8fd09a2cfbadb | https://catalog.terradue.com//better-wfp-00003... | 2017-09-22T06:16:31.0260000Z | Reproducibility stage-in notebook for Sentinel... | POLYGON((68.1159808017597 36.0515787124833,68.... |
In [12]:
print '%s results have been retrieved' %len(search)
123 results have been retrieved
- Formatting date time
In [13]:
search['startdate'] = pd.to_datetime(search['startdate'])
search['enddate'] = pd.to_datetime(search['enddate'])
In [14]:
search.head()
Out[14]:
| enclosure | enddate | identifier | self | startdate | title | wkt | |
|---|---|---|---|---|---|---|---|
| 0 | https://store.terradue.com/better-wfp-00003/_r... | 2017-09-22 06:16:31.026 | 2412362a3137482a3b8b692ff621761f97b40577 | https://catalog.terradue.com//better-wfp-00003... | 2017-09-22 06:16:31.026 | Reproducibility notebook used for generating S... | POLYGON((67.9974286897346 35.6884876797069,67.... |
| 1 | https://store.terradue.com/better-wfp-00003/_r... | 2017-09-22 06:16:31.026 | 2e66b787b9be5fe0bfe39592a28d4662debc22d4 | https://catalog.terradue.com//better-wfp-00003... | 2017-09-22 06:16:31.026 | Reproducibility notebook used for generating S... | POLYGON((67.989865566835 36.0488297143412,67.9... |
| 2 | https://store.terradue.com/better-wfp-00003/_r... | 2017-09-22 06:16:31.026 | 4a9471ba1e80fc8a60aa29ba1557f4ced5cf38f3 | https://catalog.terradue.com//better-wfp-00003... | 2017-09-22 06:16:31.026 | Reproducibility stage-in notebook for Sentinel... | POLYGON((67.989865566835 36.0488297143412,67.9... |
| 3 | https://store.terradue.com/better-wfp-00003/_r... | 2017-09-22 06:16:31.026 | 52506c40158ff7486ccfc71423748d34d5bbcc17 | https://catalog.terradue.com//better-wfp-00003... | 2017-09-22 06:16:31.026 | Reproducibility stage-in notebook for Sentinel... | POLYGON((68.1160119769251 35.690459964115,68.1... |
| 4 | https://store.terradue.com/better-wfp-00003/_r... | 2017-09-22 06:16:31.026 | 5711d6b38f923d4de2524b179cc8fd09a2cfbadb | https://catalog.terradue.com//better-wfp-00003... | 2017-09-22 06:16:31.026 | Reproducibility stage-in notebook for Sentinel... | POLYGON((68.1159808017597 36.0515787124833,68.... |
Save the metadataframe¶
Saving the dataframe containing metadata as a pickle allows working with it outside the platform
In [15]:
search.to_pickle('better-wfp-00003_Jun-Sep_2017_Afghanistan.pkl')