better-wfp-00005 data pipeline results (Sentinel-1 Sigma-0 backscatter all polarization): metadata loading and exporting (pickle file)¶
This Notebook shows how to load in memory (no data is actually stored into the local disk) the better-wfp-00005 metadata pipeline results
Result selection is done filtering by an AOI and a time range
Retrieved metadata frame is saved into a *pickle file* in order to allow the user working with it outside the platform
In [1]:
import pandas as pd
from geopandas import GeoDataFrame
import cioppy
import gdal
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from PIL import Image
%matplotlib inline
from urlparse import urlparse
import requests
Set the data pipeline catalogue access point¶
In [4]:
series = 'https://catalog.terradue.com/better-wfp-00005/series/results/description'
Area of Interest¶
In [5]:
wkt = 'POLYGON((67.62430000000001 36.7228, 68.116 36.7228, 68.116 35.6923, 67.62430000000001 35.6923, 67.62430000000001 36.7228))'
Time of interest¶
Data selection can be done filtering on start and stop dates of the datasets
In [6]:
start_time = '2017-06-01T00:00:00Z'
stop_time = '2017-09-30T23:59:59.99Z'
In [7]:
search_params = dict([('start', start_time),
('stop', stop_time),
('geom', wkt),
('count', 900)])
Or filtering on update time range: the update dates are the ones related to the catalogue metadata ingestion
In [8]:
update='2019-01-16T00:00:00Z/2019-01-17T23:59:59.99Z'
search_params = dict([('update', update),
('geom', wkt),
('count', 900)])
Or both
In [9]:
search_params = dict([('start', start_time),
('stop', stop_time),
('update', update),
('geom', wkt),
('count', 900)])
Create the dataframe¶
We create the data frame with the following metadata: * enclosure -
the URL reference to access the actual data for the download *
identifier - the unique item identifier in the catalogue * self -
Link to resource that identifies the resource in the catalogue *
startdate - Start time of the dataset (UTC ISO 8601)
* enddate - End time of the dataset (UTC ISO 8601) * title -
Title of the item, containing the string of the data filename
In [10]:
ciop = cioppy.Cioppy()
search = GeoDataFrame(ciop.search(end_point=series,
params=search_params,
output_fields='enclosure,identifier,self,startdate,enddate,title,wkt',
model='GeoTime'))
- Showing the very 5 first results of our search
In [11]:
search.head()
Out[11]:
| enclosure | enddate | identifier | self | startdate | title | wkt | |
|---|---|---|---|---|---|---|---|
| 0 | https://store.terradue.com/better-wfp-00005/_r... | 2017-09-29T13:30:24.2102540Z | 6a7b74fabadd30ed10879bdcf7cfce24dc336113 | https://catalog.terradue.com//better-wfp-00005... | 2017-09-29T13:29:59.2117860Z | Reproducibility stage-in notebook for Sentinel... | POLYGON((68.1159770636222 35.6876804486044,68.... |
| 1 | https://store.terradue.com/better-wfp-00005/_r... | 2017-09-29T13:30:24.2102540Z | 9343fecbd537bc255623cda53c875565a846e9b4 | https://catalog.terradue.com//better-wfp-00005... | 2017-09-29T13:29:59.2117860Z | Reproducibility notebook used for generating S... | POLYGON((68.1159770636222 35.6876804486044,68.... |
| 2 | https://store.terradue.com/better-wfp-00005/_r... | 2017-09-29T13:30:24.2102540Z | dc32e57be7b0335fb84ab32e55d1e13b13ebcffd | https://catalog.terradue.com//better-wfp-00005... | 2017-09-29T13:29:59.2117860Z | S1A_IW_GRDH_1SDV_20170929T132959_20170929T1330... | POLYGON((67.6060857657722 35.6923797948525,68.... |
| 3 | https://store.terradue.com/better-wfp-00005/_r... | 2017-09-24T13:22:15.0736690Z | 235f347476174c4bccbb3669a389d9188d61b9ce | https://catalog.terradue.com//better-wfp-00005... | 2017-09-24T13:21:50.0753790Z | Reproducibility stage-in notebook for Sentinel... | POLYGON((68.1159748230219 35.6882018821311,68.... |
| 4 | https://store.terradue.com/better-wfp-00005/_r... | 2017-09-24T13:22:15.0736690Z | b9c020654887ad9982d8c9aa8af93c8007020025 | https://catalog.terradue.com//better-wfp-00005... | 2017-09-24T13:21:50.0753790Z | Reproducibility notebook used for generating S... | POLYGON((68.1159748230219 35.6882018821311,68.... |
In [12]:
print '%s results have been retrieved' %len(search)
117 results have been retrieved
- Formatting date time
In [13]:
search['startdate'] = pd.to_datetime(search['startdate'])
search['enddate'] = pd.to_datetime(search['enddate'])
In [14]:
search.head()
Out[14]:
| enclosure | enddate | identifier | self | startdate | title | wkt | |
|---|---|---|---|---|---|---|---|
| 0 | https://store.terradue.com/better-wfp-00005/_r... | 2017-09-29 13:30:24.210254 | 6a7b74fabadd30ed10879bdcf7cfce24dc336113 | https://catalog.terradue.com//better-wfp-00005... | 2017-09-29 13:29:59.211786 | Reproducibility stage-in notebook for Sentinel... | POLYGON((68.1159770636222 35.6876804486044,68.... |
| 1 | https://store.terradue.com/better-wfp-00005/_r... | 2017-09-29 13:30:24.210254 | 9343fecbd537bc255623cda53c875565a846e9b4 | https://catalog.terradue.com//better-wfp-00005... | 2017-09-29 13:29:59.211786 | Reproducibility notebook used for generating S... | POLYGON((68.1159770636222 35.6876804486044,68.... |
| 2 | https://store.terradue.com/better-wfp-00005/_r... | 2017-09-29 13:30:24.210254 | dc32e57be7b0335fb84ab32e55d1e13b13ebcffd | https://catalog.terradue.com//better-wfp-00005... | 2017-09-29 13:29:59.211786 | S1A_IW_GRDH_1SDV_20170929T132959_20170929T1330... | POLYGON((67.6060857657722 35.6923797948525,68.... |
| 3 | https://store.terradue.com/better-wfp-00005/_r... | 2017-09-24 13:22:15.073669 | 235f347476174c4bccbb3669a389d9188d61b9ce | https://catalog.terradue.com//better-wfp-00005... | 2017-09-24 13:21:50.075379 | Reproducibility stage-in notebook for Sentinel... | POLYGON((68.1159748230219 35.6882018821311,68.... |
| 4 | https://store.terradue.com/better-wfp-00005/_r... | 2017-09-24 13:22:15.073669 | b9c020654887ad9982d8c9aa8af93c8007020025 | https://catalog.terradue.com//better-wfp-00005... | 2017-09-24 13:21:50.075379 | Reproducibility notebook used for generating S... | POLYGON((68.1159748230219 35.6882018821311,68.... |
Save the metadataframe¶
Saving the dataframe containing metadata as a pickle allows working with it outside the platform
In [15]:
search.to_pickle('better-wfp-00005_Jun-Sep_2017_Afghanistan.pkl')