better-wfp-00007 data pipeline results (CHIRPS Rainfall Estimates Aggregations): metadata loading and exporting (pickle file)¶
This Notebook shows how to load in memory (no data is actually stored into the local disk) the better-wfp-00007 metadata pipeline results
Result selection is done filtering by an AOI and a time range
Retrieved metadata frame is saved into a *pickle file* in order to allow the user working with it outside the platform
In [1]:
import pandas as pd
from geopandas import GeoDataFrame
import cioppy
import gdal
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from PIL import Image
%matplotlib inline
from urlparse import urlparse
import requests
- Provide here your ellip username and api key
In [2]:
import getpass
user = getpass.getpass("Enter your Ellip username:")
api_key = getpass.getpass("Enter the API key:")
Set the data pipeline catalogue access point¶
In [3]:
series = 'https://catalog.terradue.com/better-wfp-00007/series/results/description'
Area of Interest¶
In [4]:
wkt = 'POLYGON ((11.50307555189977 -11.11416337069092, 41.03432555189977 -11.11416337069092, 41.03432555189977 -34.97636566938584, 11.50307555189977 -34.97636566938584, 11.50307555189977 -11.11416337069092))'
Time of interest¶
Data selection can be done by filtering on start and stop dates of the datasets
In [5]:
start_time = '2017-05-01T00:00:00Z'
stop_time = '2017-05-31T23:59:59.99Z'
In [6]:
search_params = dict([('start', start_time),
('stop', stop_time),
('geom', wkt),
('count', 900)])
Or filtering on update time range: the update dates are the ones related to the catalogue metadata ingestion
In [7]:
update='2019-03-06T00:00:00Z/2019-03-06T23:59:59.99Z'
search_params = dict([('update', update),
('geom', wkt),
('count', 900)])
Create the dataframe¶
We create the data frame with the following metadata: * enclosure -
the URL reference to access the actual data for the download *
identifier - the unique item identifier in the catalogue * self -
Link to resource that identifies the resource in the catalogue *
startdate - Start time of the dataset (UTC ISO 8601)
* enddate - End time of the dataset (UTC ISO 8601) * title -
Title of the item, containing the string of the data filename * wkt
- The bounding box of the product
In [8]:
ciop = cioppy.Cioppy()
search = GeoDataFrame(ciop.search(end_point=series,
params=search_params,
output_fields='enclosure,identifier,self,startdate,enddate,title,wkt',
model='GeoTime'))
- Showing the very 5 first results of our search
In [9]:
search.head()
Out[9]:
| enclosure | enddate | identifier | self | startdate | title | wkt | |
|---|---|---|---|---|---|---|---|
| 0 | https://store.terradue.com/better-wfp-00007/_r... | 2017-01-31T00:00:00.0000000Z | 2F972053F721342E729EC1FA7062C4DAC0587A64 | https://catalog.terradue.com//better-wfp-00007... | 2017-01-01T00:00:00.0000000Z | Output CHIRPSv2_SouthernAfrica_N30_daystotal_2... | POLYGON((11.5030755518998 -11.1141633706909,41... |
| 1 | https://store.terradue.com/better-wfp-00007/_r... | 2017-01-31T00:00:00.0000000Z | E1E2526083E64B52661E4E4CCCB0E77597031521 | https://catalog.terradue.com//better-wfp-00007... | 2017-01-01T00:00:00.0000000Z | Output CHIRPSv2_SouthernAfrica_N30_countaboveo... | POLYGON((11.5030755518998 -11.1141633706909,41... |
| 2 | https://store.terradue.com/better-wfp-00007/_r... | 2017-01-31T00:00:00.0000000Z | F44BE47DF19F96AB7E9F1AAC01F009CB6229FFE7 | https://catalog.terradue.com//better-wfp-00007... | 2017-01-01T00:00:00.0000000Z | Output CHIRPSv2_SouthernAfrica_N30_dryspell_20... | POLYGON((11.5030755518998 -11.1141633706909,41... |
| 3 | https://store.terradue.com/better-wfp-00007/_r... | 2016-01-31T00:00:00.0000000Z | 670F163E595D0760E8FE51018117FCB0FD8ED9B0 | https://catalog.terradue.com//better-wfp-00007... | 2016-01-01T00:00:00.0000000Z | Output CHIRPSv2_SouthernAfrica_N30_daystotal_2... | POLYGON((11.5030755518998 -11.1141633706909,41... |
| 4 | https://store.terradue.com/better-wfp-00007/_r... | 2016-01-31T00:00:00.0000000Z | C44244E6BD37A41ECEE5FE949750FACF0ECEAA13 | https://catalog.terradue.com//better-wfp-00007... | 2016-01-01T00:00:00.0000000Z | Output CHIRPSv2_SouthernAfrica_N30_countaboveo... | POLYGON((11.5030755518998 -11.1141633706909,41... |
In [10]:
print '%s results have been retrieved' %len(search)
9 results have been retrieved
- Formatting date time
In [11]:
search['startdate'] = pd.to_datetime(search['startdate'])
search['enddate'] = pd.to_datetime(search['enddate'])
In [12]:
search.head()
Out[12]:
| enclosure | enddate | identifier | self | startdate | title | wkt | |
|---|---|---|---|---|---|---|---|
| 0 | https://store.terradue.com/better-wfp-00007/_r... | 2017-01-31 | 2F972053F721342E729EC1FA7062C4DAC0587A64 | https://catalog.terradue.com//better-wfp-00007... | 2017-01-01 | Output CHIRPSv2_SouthernAfrica_N30_daystotal_2... | POLYGON((11.5030755518998 -11.1141633706909,41... |
| 1 | https://store.terradue.com/better-wfp-00007/_r... | 2017-01-31 | E1E2526083E64B52661E4E4CCCB0E77597031521 | https://catalog.terradue.com//better-wfp-00007... | 2017-01-01 | Output CHIRPSv2_SouthernAfrica_N30_countaboveo... | POLYGON((11.5030755518998 -11.1141633706909,41... |
| 2 | https://store.terradue.com/better-wfp-00007/_r... | 2017-01-31 | F44BE47DF19F96AB7E9F1AAC01F009CB6229FFE7 | https://catalog.terradue.com//better-wfp-00007... | 2017-01-01 | Output CHIRPSv2_SouthernAfrica_N30_dryspell_20... | POLYGON((11.5030755518998 -11.1141633706909,41... |
| 3 | https://store.terradue.com/better-wfp-00007/_r... | 2016-01-31 | 670F163E595D0760E8FE51018117FCB0FD8ED9B0 | https://catalog.terradue.com//better-wfp-00007... | 2016-01-01 | Output CHIRPSv2_SouthernAfrica_N30_daystotal_2... | POLYGON((11.5030755518998 -11.1141633706909,41... |
| 4 | https://store.terradue.com/better-wfp-00007/_r... | 2016-01-31 | C44244E6BD37A41ECEE5FE949750FACF0ECEAA13 | https://catalog.terradue.com//better-wfp-00007... | 2016-01-01 | Output CHIRPSv2_SouthernAfrica_N30_countaboveo... | POLYGON((11.5030755518998 -11.1141633706909,41... |
Save the metadataframe¶
Saving the dataframe containing metadata as a pickle allows working with it outside the platform
In [15]:
search.to_pickle('better-wfp-00007_validation.pkl')