better-wfp-00002 data pipeline results (Sentinel-1 Coherence timeseries): metadata loading and exporting (pickle file)¶
This Notebook shows how to load in memory (no data is actually stored into the local disk) the better-wfp-00002 metadata pipeline results
Result selection is done filtering by an AOI and a time range
Retrieved metadata frame is saved into a *pickle file* in order to allow the user working with it outside the platform
In [1]:
import pandas as pd
from geopandas import GeoDataFrame
import cioppy
import gdal
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from PIL import Image
%matplotlib inline
from urlparse import urlparse
import requests
Set the data pipeline catalogue access point¶
In [2]:
series = 'https://catalog.terradue.com/better-wfp-00002/series/results/description'
Area of Interest¶
In [3]:
wkt = 'POLYGON((30.360611 8.155859,28.120043 8.608053,28.444235 10.23355,30.695391 9.785728,30.360611 8.155859))'
Time of interest¶
Data selection can be done by filtering on start and stop dates of the datasets
In [4]:
start_time = '2017-01-11T00:00:00Z'
stop_time = '2017-01-17T23:59:59.99Z'
In [5]:
search_params = dict([('start', start_time),
('stop', stop_time),
('geom', wkt),
('count', 900)])
Or filtering on update time range: the update dates are the ones related to the catalogue metadata ingestion
In [6]:
update='2019-02-05T00:00:00.000000Z/2019-02-06T23:59:59.99000Z'
search_params = dict([('update', update),
('geom', wkt),
('count', 900)])
Create the dataframe¶
We create the data frame with the following metadata: * enclosure -
the URL reference to access the actual data for the download *
identifier - the unique item identifier in the catalogue * self -
Link to resource that identifies the resource in the catalogue *
startdate - Start time of the dataset (UTC ISO 8601)
* enddate - End time of the dataset (UTC ISO 8601) * title -
Title of the item, containing the string of the data filename
In [7]:
ciop = cioppy.Cioppy()
search = GeoDataFrame(ciop.search(end_point=series,
params=search_params,
output_fields='enclosure,identifier,self,startdate,enddate,title,wkt',
model='GeoTime'))
- Showing the very 5 first results of our search
In [8]:
search.head()
Out[8]:
| enclosure | enddate | identifier | self | startdate | title | wkt | |
|---|---|---|---|---|---|---|---|
| 0 | https://store.terradue.com/better-wfp-00002/_r... | 2017-01-17T03:50:08.0000000Z | 585efc39a7c4b1e6df4689c41123be2f17cecd8e | https://catalog.terradue.com//better-wfp-00002... | 2017-01-11T03:49:20.0000000Z | S1_COH_20170117035008_20170111034920_C2F1_116E... | POLYGON((29.0590832889222 7.90674554216732 |
| 1 | https://store.terradue.com/better-wfp-00002/_r... | 2017-01-17T03:50:08.0000000Z | 9221e259f96e36c100465124b204dd332d34a64a | https://catalog.terradue.com//better-wfp-00002... | 2017-01-11T03:49:20.0000000Z | Reproducibility notebook used for generating S... | POLYGON((29.0590832889222 7.90674554216732 |
| 2 | https://store.terradue.com/better-wfp-00002/_r... | 2017-01-17T03:50:08.0000000Z | 94d07730ae194ef7262a807bf2bc08f293dad340 | https://catalog.terradue.com//better-wfp-00002... | 2017-01-11T03:49:20.0000000Z | Reproducibility stage-in notebook for Sentinel... | POLYGON((29.0590832889222 7.90674554216732 |
| 3 | https://store.terradue.com/better-wfp-00002/_r... | 2017-01-11T03:49:20.0000000Z | 7c31ba69c43771b536906eb85226371195e7c1b1 | https://catalog.terradue.com//better-wfp-00002... | 2017-01-05T03:50:00.0000000Z | Reproducibility stage-in notebook for Sentinel... | POLYGON((29.1590086933083 8.57426607119175 |
| 4 | https://store.terradue.com/better-wfp-00002/_r... | 2017-01-11T03:49:20.0000000Z | e32e3854009a02076089eb4e2164cfc508cd1c07 | https://catalog.terradue.com//better-wfp-00002... | 2017-01-05T03:50:00.0000000Z | S1_COH_20170111034920_20170105035000_116E_0110... | POLYGON((29.1590086933083 8.57426607119175 |
In [9]:
print '%s results have been retrieved' %len(search)
6 results have been retrieved
- Formatting date time
In [10]:
search['startdate'] = pd.to_datetime(search['startdate'])
search['enddate'] = pd.to_datetime(search['enddate'])
In [11]:
search.head()
Out[11]:
| enclosure | enddate | identifier | self | startdate | title | wkt | |
|---|---|---|---|---|---|---|---|
| 0 | https://store.terradue.com/better-wfp-00002/_r... | 2017-01-17 03:50:08 | 585efc39a7c4b1e6df4689c41123be2f17cecd8e | https://catalog.terradue.com//better-wfp-00002... | 2017-01-11 03:49:20 | S1_COH_20170117035008_20170111034920_C2F1_116E... | POLYGON((29.0590832889222 7.90674554216732 |
| 1 | https://store.terradue.com/better-wfp-00002/_r... | 2017-01-17 03:50:08 | 9221e259f96e36c100465124b204dd332d34a64a | https://catalog.terradue.com//better-wfp-00002... | 2017-01-11 03:49:20 | Reproducibility notebook used for generating S... | POLYGON((29.0590832889222 7.90674554216732 |
| 2 | https://store.terradue.com/better-wfp-00002/_r... | 2017-01-17 03:50:08 | 94d07730ae194ef7262a807bf2bc08f293dad340 | https://catalog.terradue.com//better-wfp-00002... | 2017-01-11 03:49:20 | Reproducibility stage-in notebook for Sentinel... | POLYGON((29.0590832889222 7.90674554216732 |
| 3 | https://store.terradue.com/better-wfp-00002/_r... | 2017-01-11 03:49:20 | 7c31ba69c43771b536906eb85226371195e7c1b1 | https://catalog.terradue.com//better-wfp-00002... | 2017-01-05 03:50:00 | Reproducibility stage-in notebook for Sentinel... | POLYGON((29.1590086933083 8.57426607119175 |
| 4 | https://store.terradue.com/better-wfp-00002/_r... | 2017-01-11 03:49:20 | e32e3854009a02076089eb4e2164cfc508cd1c07 | https://catalog.terradue.com//better-wfp-00002... | 2017-01-05 03:50:00 | S1_COH_20170111034920_20170105035000_116E_0110... | POLYGON((29.1590086933083 8.57426607119175 |
Save the metadataframe¶
Saving the dataframe containing metadata as a pickle allows working with it outside the platform
In [12]:
search.to_pickle('better-wfp-00002_validations.pkl')