better-wfp-00006 data pipeline results (COPERNICUS Vegetation Indicators Aggregations): metadata loading and exporting (pickle file)¶
This Notebook shows how to load in memory (no data is actually stored into the local disk) the better-wfp-00006 metadata pipeline results
Result selection is done filtering by an AOI and a time range
Retrieved metadata frame is saved into a *pickle file* in order to allow the user working with it outside the platform
In [1]:
import pandas as pd
from geopandas import GeoDataFrame
import cioppy
import gdal
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from PIL import Image
%matplotlib inline
from urlparse import urlparse
import requests
- Provide here your ellip username and api key
In [2]:
import getpass
user = getpass.getpass("Enter your Ellip username:")
api_key = getpass.getpass("Enter the API key:")
Set the data pipeline catalogue access point¶
In [3]:
series = 'https://catalog.terradue.com/better-wfp-00006/series/results/description'
Area of Interest¶
In [5]:
wkt = 'POLYGON ((11.50307555189977 -11.11416337069092, 41.03432555189977 -11.11416337069092, 41.03432555189977 -34.97636566938584, 11.50307555189977 -34.97636566938584, 11.50307555189977 -11.11416337069092))'
Time of interest¶
Data selection can be done by filtering on start and stop dates of the datasets
In [ ]:
start_time = '2017-01-01T00:00:00Z'
stop_time = '2017-01-31T23:59:59.99Z'
In [ ]:
search_params = dict([('start', start_time),
('stop', stop_time),
('geom', wkt),
('count', 900)])
Or filtering on update time range: the update dates are the ones related to the catalogue metadata ingestion
In [8]:
update='2019-03-07T00:00:00Z/2019-03-07T23:59:59.99Z'
search_params = dict([('update', update),
('geom', wkt),
('count', 900)])
Create the dataframe¶
We create the data frame with the following metadata: * enclosure -
the URL reference to access the actual data for the download *
identifier - the unique item identifier in the catalogue * self -
Link to resource that identifies the resource in the catalogue *
startdate - Start time of the dataset (UTC ISO 8601)
* enddate - End time of the dataset (UTC ISO 8601) * title -
Title of the item, containing the string of the data filename * wkt
- The bounding box of the product
In [9]:
ciop = cioppy.Cioppy()
search = GeoDataFrame(ciop.search(end_point=series,
params=search_params,
output_fields='enclosure,identifier,self,startdate,enddate,title,wkt',
model='GeoTime'))
- Showing the very 5 first results of our search
In [10]:
search.head()
Out[10]:
| enclosure | enddate | identifier | self | startdate | title | wkt | |
|---|---|---|---|---|---|---|---|
| 0 | https://store.terradue.com/better-wfp-00006/_r... | 2017-01-31T00:00:00.0000000Z | 4C171ECE9E48E2D91F1FD6671774E34BF6F078D9 | https://catalog.terradue.com//better-wfp-00006... | 2017-01-10T00:00:00.0000000Z | Output LAI_SouthernAfrica_N3_maxvalues_2017-01... | POLYGON((11.5030755518998 -11.1141633706909,41... |
| 1 | https://store.terradue.com/better-wfp-00006/_r... | 2017-01-31T00:00:00.0000000Z | 69E4561CF3E34251D6F2C52E3DEE60873E2BE57E | https://catalog.terradue.com//better-wfp-00006... | 2017-01-10T00:00:00.0000000Z | Output FAPAR_SouthernAfrica_N3_averages_2017-0... | POLYGON((11.5030755518998 -11.1141633706909,41... |
| 2 | https://store.terradue.com/better-wfp-00006/_r... | 2017-01-31T00:00:00.0000000Z | 7963FF3C50C865EBB01F466263BC92A6645BC36A | https://catalog.terradue.com//better-wfp-00006... | 2017-01-10T00:00:00.0000000Z | Output LAI_SouthernAfrica_N3_averages_2017-01-... | POLYGON((11.5030755518998 -11.1141633706909,41... |
| 3 | https://store.terradue.com/better-wfp-00006/_r... | 2017-01-31T00:00:00.0000000Z | C3AE42343AC02E69EAB0F373716D654E8C22DC96 | https://catalog.terradue.com//better-wfp-00006... | 2017-01-10T00:00:00.0000000Z | Output FAPAR_SouthernAfrica_N3_maxvalues_2017-... | POLYGON((11.5030755518998 -11.1141633706909,41... |
| 4 | https://store.terradue.com/better-wfp-00006/_r... | 2016-01-31T00:00:00.0000000Z | 4644446E4102BA104DC8A694E23491BE786B7B86 | https://catalog.terradue.com//better-wfp-00006... | 2016-01-10T00:00:00.0000000Z | Output LAI_SouthernAfrica_N3_averages_2016-01-... | POLYGON((11.5030755518998 -11.1141633706909,41... |
In [11]:
print '%s results have been retrieved' %len(search)
12 results have been retrieved
- Formatting date time
In [12]:
search['startdate'] = pd.to_datetime(search['startdate'])
search['enddate'] = pd.to_datetime(search['enddate'])
In [13]:
search.head()
Out[13]:
| enclosure | enddate | identifier | self | startdate | title | wkt | |
|---|---|---|---|---|---|---|---|
| 0 | https://store.terradue.com/better-wfp-00006/_r... | 2017-01-31 | 4C171ECE9E48E2D91F1FD6671774E34BF6F078D9 | https://catalog.terradue.com//better-wfp-00006... | 2017-01-10 | Output LAI_SouthernAfrica_N3_maxvalues_2017-01... | POLYGON((11.5030755518998 -11.1141633706909,41... |
| 1 | https://store.terradue.com/better-wfp-00006/_r... | 2017-01-31 | 69E4561CF3E34251D6F2C52E3DEE60873E2BE57E | https://catalog.terradue.com//better-wfp-00006... | 2017-01-10 | Output FAPAR_SouthernAfrica_N3_averages_2017-0... | POLYGON((11.5030755518998 -11.1141633706909,41... |
| 2 | https://store.terradue.com/better-wfp-00006/_r... | 2017-01-31 | 7963FF3C50C865EBB01F466263BC92A6645BC36A | https://catalog.terradue.com//better-wfp-00006... | 2017-01-10 | Output LAI_SouthernAfrica_N3_averages_2017-01-... | POLYGON((11.5030755518998 -11.1141633706909,41... |
| 3 | https://store.terradue.com/better-wfp-00006/_r... | 2017-01-31 | C3AE42343AC02E69EAB0F373716D654E8C22DC96 | https://catalog.terradue.com//better-wfp-00006... | 2017-01-10 | Output FAPAR_SouthernAfrica_N3_maxvalues_2017-... | POLYGON((11.5030755518998 -11.1141633706909,41... |
| 4 | https://store.terradue.com/better-wfp-00006/_r... | 2016-01-31 | 4644446E4102BA104DC8A694E23491BE786B7B86 | https://catalog.terradue.com//better-wfp-00006... | 2016-01-10 | Output LAI_SouthernAfrica_N3_averages_2016-01-... | POLYGON((11.5030755518998 -11.1141633706909,41... |
Save the metadataframe¶
Saving the dataframe containing metadata as a pickle allows working with it outside the platform
In [14]:
search.to_pickle('better-wfp-00006_validation.pkl')