better-wfp-00004 data pipeline results (Landsat 8 reflectances and vegetation indices): metadata loading and saving (pickle file)¶
This Notebook shows how to load in memory (no data is actually stored into the local disk) the better-wfp-00004 metadata pipeline results
Result selection is done filtering by an AOI and a time range
Retrieved metadata frame is saved into a *pickle file* in order to allow the user working with it outside the platform
In [1]:
import pandas as pd
from geopandas import GeoDataFrame
import cioppy
import gdal
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from PIL import Image
%matplotlib inline
from urlparse import urlparse
import requests
Time of interest¶
In [76]:
start_time = '2017-08-01T00:00:00Z'
stop_time = '2017-08-31T23:59:59.99Z'
Query the data pipeline catalogue access point¶
In [77]:
series = 'https://catalog.terradue.com/better-wfp-00004/series/results/description'
Area of Interest¶
In [78]:
wkt = 'POLYGON ((67.62430000000001 36.7228, 68.116 36.7228, 68.116 35.6923, 67.62430000000001 35.6923, 67.62430000000001 36.7228))'
In [79]:
search_params = dict([('start', start_time),
('stop', stop_time),
('geom', wkt),
('count', 500)])
Create the dataframe¶
We create the data frame with the following metadata: * enclosure -
the URL reference to access the actual data for the download *
identifier - the unique item identifier in the catalogue * self -
Link to resource that identifies the resource in the catalogue *
startdate - Start time of the dataset (UTC ISO 8601)
* enddate - End time of the dataset (UTC ISO 8601) * title -
Title of the item, containing the string of the data filename
In [80]:
ciop = cioppy.Cioppy()
search = GeoDataFrame(ciop.search(end_point=series,
params=search_params,
output_fields='enclosure,identifier,self,startdate,enddate,title',
model='GeoTime'))
- Showing the very 5 first results of our search
In [81]:
search.head()
Out[81]:
| enclosure | enddate | identifier | self | startdate | title | |
|---|---|---|---|---|---|---|
| 0 | https://store.terradue.com/better-wfp-00004/_r... | 2017-08-29T06:00:29.1410000Z | 0f9c46ff86b5831beb987a60440893674372e7df | https://catalog.terradue.com//better-wfp-00004... | 2017-08-29T05:59:57.3710000Z | Reproducibility stage-in notebook for Landsat8... |
| 1 | https://store.terradue.com/better-wfp-00004/_r... | 2017-08-29T06:00:29.1410000Z | 342b427929ad9cb1dc251800f8f4888b4b809abd | https://catalog.terradue.com//better-wfp-00004... | 2017-08-29T05:59:57.3710000Z | Reproducibility notebook used for generating L... |
| 2 | https://store.terradue.com/better-wfp-00004/_r... | 2017-08-29T06:00:29.1410000Z | 2bfdd6e5b1c9c31810016de8d03c2a193d1abe74 | https://catalog.terradue.com//better-wfp-00004... | 2017-08-29T05:59:57.3710000Z | LC08_L1TP_153035_20170829_20170914_01_T1_R2.TIF |
| 3 | https://store.terradue.com/better-wfp-00004/_r... | 2017-08-29T06:00:29.1410000Z | 3d789cb37f3dfc2b09494c73cafdee402dde8f70 | https://catalog.terradue.com//better-wfp-00004... | 2017-08-29T05:59:57.3710000Z | LC08_L1TP_153035_20170829_20170914_01_T1_R5.TIF |
| 4 | https://store.terradue.com/better-wfp-00004/_r... | 2017-08-29T06:00:29.1410000Z | 412b568b84a9bdc74b0f90a3dcf28db4768a5e9e | https://catalog.terradue.com//better-wfp-00004... | 2017-08-29T05:59:57.3710000Z | LC08_L1TP_153035_20170829_20170914_01_T1_R3.TIF |
In [82]:
print '%s results have been retrieved' %len(search)
96 results have been retrieved
- Formatting date time
In [83]:
search['startdate'] = pd.to_datetime(search['startdate'])
search['enddate'] = pd.to_datetime(search['enddate'])
In [84]:
search.head()
Out[84]:
| enclosure | enddate | identifier | self | startdate | title | |
|---|---|---|---|---|---|---|
| 0 | https://store.terradue.com/better-wfp-00004/_r... | 2017-08-29 06:00:29.141 | 0f9c46ff86b5831beb987a60440893674372e7df | https://catalog.terradue.com//better-wfp-00004... | 2017-08-29 05:59:57.371 | Reproducibility stage-in notebook for Landsat8... |
| 1 | https://store.terradue.com/better-wfp-00004/_r... | 2017-08-29 06:00:29.141 | 342b427929ad9cb1dc251800f8f4888b4b809abd | https://catalog.terradue.com//better-wfp-00004... | 2017-08-29 05:59:57.371 | Reproducibility notebook used for generating L... |
| 2 | https://store.terradue.com/better-wfp-00004/_r... | 2017-08-29 06:00:29.141 | 2bfdd6e5b1c9c31810016de8d03c2a193d1abe74 | https://catalog.terradue.com//better-wfp-00004... | 2017-08-29 05:59:57.371 | LC08_L1TP_153035_20170829_20170914_01_T1_R2.TIF |
| 3 | https://store.terradue.com/better-wfp-00004/_r... | 2017-08-29 06:00:29.141 | 3d789cb37f3dfc2b09494c73cafdee402dde8f70 | https://catalog.terradue.com//better-wfp-00004... | 2017-08-29 05:59:57.371 | LC08_L1TP_153035_20170829_20170914_01_T1_R5.TIF |
| 4 | https://store.terradue.com/better-wfp-00004/_r... | 2017-08-29 06:00:29.141 | 412b568b84a9bdc74b0f90a3dcf28db4768a5e9e | https://catalog.terradue.com//better-wfp-00004... | 2017-08-29 05:59:57.371 | LC08_L1TP_153035_20170829_20170914_01_T1_R3.TIF |
Save the metadataframe¶
Saving the dataframe containing metadata as a pickle allows working with it outside the platform
In [85]:
search.to_pickle('better-wfp-00004_Aug_2017_Afghanistan.pkl')