better-wfp-00001 data pipeline results (Sentinel-1 backscatter time-series selected polarization): metadata loading and exporting (pickle file)¶
This Notebook shows how to load in memory (no data is actually stored into the local disk) the better-wfp-00001 metadata pipeline results
Result selection is done filtering by an AOI and a time range
Retrieved metadata frame is saved into a *pickle file* in order to allow the user working with it outside the platform
In [1]:
import pandas as pd
from geopandas import GeoDataFrame
import cioppy
import gdal
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from PIL import Image
%matplotlib inline
from urlparse import urlparse
import requests
- Provide here your ellip username and api key
In [2]:
import getpass
user = getpass.getpass("Enter your Ellip username:")
api_key = getpass.getpass("Enter the API key:")
Set the data pipeline catalogue access point¶
In [3]:
series = 'https://catalog.terradue.com/better-wfp-00001/series/results/description'
Area of Interest¶
In [4]:
wkt = 'POLYGON((34.977154 10.632262,32.722355 11.077297,33.070683 12.826534,35.340668 12.385893,34.977154 10.632262))'
Time of interest¶
Data selection can be done by filtering on start and stop dates of the datasets
In [5]:
start_time = '2017-06-01T00:00:00Z'
stop_time = '2017-09-30T23:59:59.99Z'
In [6]:
search_params = dict([('start', start_time),
('stop', stop_time),
('geom', wkt),
('count', 900)])
Or filtering on update time range: the update dates are the ones related to the catalogue metadata ingestion
In [5]:
update='2019-02-28T00:00:00Z/2019-03-06T23:59:59.99Z'
search_params = dict([('update', update),
('geom', wkt),
('count', 900)])
Create the dataframe¶
We create the data frame with the following metadata: * enclosure -
the URL reference to access the actual data for the download *
identifier - the unique item identifier in the catalogue * self -
Link to resource that identifies the resource in the catalogue *
startdate - Start time of the dataset (UTC ISO 8601)
* enddate - End time of the dataset (UTC ISO 8601) * title -
Title of the item, containing the string of the data filename
In [6]:
ciop = cioppy.Cioppy()
search = GeoDataFrame(ciop.search(end_point=series,
params=search_params,
output_fields='enclosure,identifier,self,startdate,enddate,title,wkt',
model='GeoTime'))
- Showing the very 5 first results of our search
In [7]:
search.head()
Out[7]:
| enclosure | enddate | identifier | self | startdate | title | wkt | |
|---|---|---|---|---|---|---|---|
| 0 | https://store.terradue.com/better-wfp-00001/_r... | 2017-10-10T03:33:29.5735860Z | 7D9FD84BD60A843F26FDEA800E9ECF9D396151A2 | https://catalog.terradue.com//better-wfp-00001... | 2017-10-10T03:33:00.5637550Z | Reproducibility notebook used for generating S... | POLYGON((35.3446002755045 10.6382850413833,35.... |
| 1 | https://store.terradue.com/better-wfp-00001/_r... | 2017-10-10T03:33:29.5735860Z | 7F3A3ACED0B5C7591BB1FDD7E6B27C6635369091 | https://catalog.terradue.com//better-wfp-00001... | 2017-10-10T03:33:00.5637550Z | Reproducibility stage-in notebook for Sentinel... | POLYGON((35.3446002755045 10.6382850413833,35.... |
| 2 | https://store.terradue.com/better-wfp-00001/_r... | 2017-10-10T03:33:29.5735860Z | 1F7C3F9E8C413A4873EC941EAD3490FB5E66E2F2 | https://catalog.terradue.com//better-wfp-00001... | 2017-10-10T03:33:00.5637550Z | S1A_IW_GRDH_1SDV_20171010T033300_20171010T0333... | POLYGON((32.7205219811478 10.6297848454969,35.... |
| 3 | https://store.terradue.com/better-wfp-00001/_r... | 2017-09-28T03:33:29.3197770Z | 01EA46331E2C43284B6994522631DB593C40102A | https://catalog.terradue.com//better-wfp-00001... | 2017-09-28T03:33:00.3203670Z | Reproducibility stage-in notebook for Sentinel... | POLYGON((35.3446346180086 10.6382893347105,35.... |
| 4 | https://store.terradue.com/better-wfp-00001/_r... | 2017-09-28T03:33:29.3197770Z | F5340A56D58BE30CE0913C22469A0722FE9FF2CC | https://catalog.terradue.com//better-wfp-00001... | 2017-09-28T03:33:00.3203670Z | Reproducibility notebook used for generating S... | POLYGON((35.3446346180086 10.6382893347105,35.... |
In [8]:
print '%s results have been retrieved' %len(search)
12 results have been retrieved
- Formatting date time
In [9]:
search['startdate'] = pd.to_datetime(search['startdate'])
search['enddate'] = pd.to_datetime(search['enddate'])
In [10]:
search.head()
Out[10]:
| enclosure | enddate | identifier | self | startdate | title | wkt | |
|---|---|---|---|---|---|---|---|
| 0 | https://store.terradue.com/better-wfp-00001/_r... | 2017-10-10 03:33:29.573586 | 7D9FD84BD60A843F26FDEA800E9ECF9D396151A2 | https://catalog.terradue.com//better-wfp-00001... | 2017-10-10 03:33:00.563755 | Reproducibility notebook used for generating S... | POLYGON((35.3446002755045 10.6382850413833,35.... |
| 1 | https://store.terradue.com/better-wfp-00001/_r... | 2017-10-10 03:33:29.573586 | 7F3A3ACED0B5C7591BB1FDD7E6B27C6635369091 | https://catalog.terradue.com//better-wfp-00001... | 2017-10-10 03:33:00.563755 | Reproducibility stage-in notebook for Sentinel... | POLYGON((35.3446002755045 10.6382850413833,35.... |
| 2 | https://store.terradue.com/better-wfp-00001/_r... | 2017-10-10 03:33:29.573586 | 1F7C3F9E8C413A4873EC941EAD3490FB5E66E2F2 | https://catalog.terradue.com//better-wfp-00001... | 2017-10-10 03:33:00.563755 | S1A_IW_GRDH_1SDV_20171010T033300_20171010T0333... | POLYGON((32.7205219811478 10.6297848454969,35.... |
| 3 | https://store.terradue.com/better-wfp-00001/_r... | 2017-09-28 03:33:29.319777 | 01EA46331E2C43284B6994522631DB593C40102A | https://catalog.terradue.com//better-wfp-00001... | 2017-09-28 03:33:00.320367 | Reproducibility stage-in notebook for Sentinel... | POLYGON((35.3446346180086 10.6382893347105,35.... |
| 4 | https://store.terradue.com/better-wfp-00001/_r... | 2017-09-28 03:33:29.319777 | F5340A56D58BE30CE0913C22469A0722FE9FF2CC | https://catalog.terradue.com//better-wfp-00001... | 2017-09-28 03:33:00.320367 | Reproducibility notebook used for generating S... | POLYGON((35.3446346180086 10.6382893347105,35.... |
Save the metadataframe¶
Saving the dataframe containing metadata as a pickle allows working with it outside the platform
In [11]:
search.to_pickle('better-wfp-00001_validation_Sigma0_all_Pol2.pkl')