better-ethz-00002 data pipeline results (Filtered DInSAR interferograms): metadata loading and exporting (pickle file)¶
This Notebook shows how to load in memory (no data is actually stored into the local disk) the better-ethz-00002 metadata pipeline results
Result selection is done filtering by an AOI and a time range
Retrieved metadata frame is saved into a *pickle file* in order to allow the user working with it outside the platform
In [1]:
import pandas as pd
from geopandas import GeoDataFrame
import cioppy
import gdal
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from PIL import Image
%matplotlib inline
from urlparse import urlparse
import requests
Set the data pipeline catalogue access point¶
In [2]:
series = 'https://catalog.terradue.com/better-ethz-00002/series/results/description'
Area of Interest¶
In [3]:
# South and eastern Europe
geom = 'POLYGON((12.261 42.294,28.652 36.457,29.971 39.707,26.938 46.134,12.085 43.676,12.261 42.294))'
Time of interest¶
Data selection can be done by filtering on start and stop dates and/or update dates range of the datasets
In [4]:
start_time = '2016-01-01T00:00:00Z'
stop_time = '2018-12-31T23:59:59.99Z'
update='2018-12-01T00:00:00Z/2018-12-31T23:59:59.99Z'
In [5]:
search_params = dict([('start', start_time),
('stop', stop_time),
('update', update),
('geom', geom),
('count', 100)])
Create the dataframe¶
We create the data frame with the following metadata: * enclosure -
the URL reference to access the actual data for the download *
identifier - the unique item identifier in the catalogue * self -
Link to resource that identifies the resource in the catalogue *
startdate - Start time of the dataset (UTC ISO 8601)
* enddate - End time of the dataset (UTC ISO 8601) * title -
Title of the item, containing the string of the data filename
In [6]:
ciop = cioppy.Cioppy()
search = GeoDataFrame(ciop.search(end_point=series,
params=search_params,
output_fields='enclosure,identifier,self,startdate,enddate,title,wkt',
model='GeoTime'))
- Showing the very 5 first results of our search
In [7]:
search.head(25)
Out[7]:
| enclosure | enddate | identifier | self | startdate | title | wkt | |
|---|---|---|---|---|---|---|---|
| 0 | https://store.terradue.com/better-ethz-00002/_... | 2018-10-28T00:38:11.6100000Z | 17e6950ca6f7bba41e355260a5b4eba51b66decb | https://catalog.terradue.com//better-ethz-0000... | 2018-10-28T00:38:11.6100000Z | us1000hi43 - GEO_AMPL_CHANGE | POINT(26.3972 45.6575) |
| 1 | https://store.terradue.com/better-ethz-00002/_... | 2018-10-28T00:38:11.6100000Z | 3a7914011d350db0841682f8ef4166904a932dfd | https://catalog.terradue.com//better-ethz-0000... | 2018-10-28T00:38:11.6100000Z | us1000hi43 - GEO_COH_AMPL | POINT(26.3972 45.6575) |
| 2 | https://store.terradue.com/better-ethz-00002/_... | 2018-10-28T00:38:11.6100000Z | 8b41d4a479b4ba25001822add9d6b0ba936b5f41 | https://catalog.terradue.com//better-ethz-0000... | 2018-10-28T00:38:11.6100000Z | us1000hi43 - GEO_DCOHER | POINT(26.3972 45.6575) |
| 3 | https://store.terradue.com/better-ethz-00002/_... | 2018-10-28T00:38:11.6100000Z | 929eb1dddd91c5cdd31d00dc2738f76e0a81cf6b | https://catalog.terradue.com//better-ethz-0000... | 2018-10-28T00:38:11.6100000Z | M 5.5 - 15km SE of Comandau | Romania (us1000hi43),POINT(26.3972 45.6575) |
| 4 | https://store.terradue.com/better-ethz-00002/_... | 2018-10-28T00:38:11.6100000Z | c30ab94080fff87625c8946df4bdcd2fd4112596 | https://catalog.terradue.com//better-ethz-0000... | 2018-10-28T00:38:11.6100000Z | Pickle file for search (us1000hi43) | POINT(26.3972 45.6575) |
| 5 | https://store.terradue.com/better-ethz-00002/_... | 2018-10-28T00:38:11.6100000Z | c5edcff5a63f8c3f7250ef1d65a4456f1d1c871c | https://catalog.terradue.com//better-ethz-0000... | 2018-10-28T00:38:11.6100000Z | us1000hi43 - GEO_DPHASE | POINT(26.3972 45.6575) |
| 6 | https://store.terradue.com/better-ethz-00002/_... | 2018-10-28T00:38:11.6100000Z | cb91ca2c7f9c9282d4d36d9e11068760305d91bf | https://catalog.terradue.com//better-ethz-0000... | 2018-10-28T00:38:11.6100000Z | Reproducibility notebook for event us1000hi43 | POINT(26.3972 45.6575) |
| 7 | https://store.terradue.com/better-ethz-00002/_... | 2018-10-28T00:38:11.6100000Z | d4322cb492cda913be0eb46b0c9bca6feb91f7af | https://catalog.terradue.com//better-ethz-0000... | 2018-10-28T00:38:11.6100000Z | Pickle file for selected (us1000hi43) | POINT(26.3972 45.6575) |
| 8 | https://store.terradue.com/better-ethz-00002/_... | 2018-08-11T15:38:33.5500000Z | 1498b0602eb35a3acea447daaffb799eb1039e90 | https://catalog.terradue.com//better-ethz-0000... | 2018-08-11T15:38:33.5500000Z | Reproducibility notebook for event us1000g6wx | POINT(20.1018 41.5869) |
| 9 | https://store.terradue.com/better-ethz-00002/_... | 2018-08-11T15:38:33.5500000Z | 59a78f5ddbac19d1d158358683502c2b5a4cac1b | https://catalog.terradue.com//better-ethz-0000... | 2018-08-11T15:38:33.5500000Z | us1000g6wx - GEO_AMPL_CHANGE | POINT(20.1018 41.5869) |
| 10 | https://store.terradue.com/better-ethz-00002/_... | 2018-08-11T15:38:33.5500000Z | 8a41561287ef3ab36d004f32040a58c3a112c207 | https://catalog.terradue.com//better-ethz-0000... | 2018-08-11T15:38:33.5500000Z | us1000g6wx - GEO_DPHASE | POINT(20.1018 41.5869) |
| 11 | https://store.terradue.com/better-ethz-00002/_... | 2018-08-11T15:38:33.5500000Z | 8c617069f322998c3b5422684eaca4a3698ceb8e | https://catalog.terradue.com//better-ethz-0000... | 2018-08-11T15:38:33.5500000Z | us1000g6wx - GEO_COH_AMPL | POINT(20.1018 41.5869) |
| 12 | https://store.terradue.com/better-ethz-00002/_... | 2018-08-11T15:38:33.5500000Z | b2b646f25e271b9e3a4b171f13197524cf119022 | https://catalog.terradue.com//better-ethz-0000... | 2018-08-11T15:38:33.5500000Z | us1000g6wx - GEO_DCOHER | POINT(20.1018 41.5869) |
| 13 | https://store.terradue.com/better-ethz-00002/_... | 2018-08-11T15:38:33.5500000Z | b55f5de1925a536eb00785a4ee08ddd413e1ac2a | https://catalog.terradue.com//better-ethz-0000... | 2018-08-11T15:38:33.5500000Z | M 5.1 - 8km ESE of Burrel | Albania (us1000g6wx),POINT(20.1018 41.5869) |
| 14 | https://store.terradue.com/better-ethz-00002/_... | 2018-08-11T15:38:33.5500000Z | c44b754bf299a477e0452efa5a0a95c38c001471 | https://catalog.terradue.com//better-ethz-0000... | 2018-08-11T15:38:33.5500000Z | Pickle file for search (us1000g6wx) | POINT(20.1018 41.5869) |
| 15 | https://store.terradue.com/better-ethz-00002/_... | 2018-08-11T15:38:33.5500000Z | f5326479d8cb4d3c95d8204cda88d81eab0ea05f | https://catalog.terradue.com//better-ethz-0000... | 2018-08-11T15:38:33.5500000Z | Pickle file for selected (us1000g6wx) | POINT(20.1018 41.5869) |
| 16 | https://store.terradue.com/better-ethz-00002/_... | 2018-07-04T09:01:09.5700000Z | 25a178045f0bda42b1c12e3a27a4d8b1c7c32cc6 | https://catalog.terradue.com//better-ethz-0000... | 2018-07-04T09:01:09.5700000Z | Pickle file for selected (us2000fwnu) | POINT(19.5828 41.4466) |
| 17 | https://store.terradue.com/better-ethz-00002/_... | 2018-07-04T09:01:09.5700000Z | 39453ed443e42b1b7482648e998aff9241aeb41d | https://catalog.terradue.com//better-ethz-0000... | 2018-07-04T09:01:09.5700000Z | us2000fwnu - GEO_DCOHER | POINT(19.5828 41.4466) |
| 18 | https://store.terradue.com/better-ethz-00002/_... | 2018-07-04T09:01:09.5700000Z | 522e7683a761e9323605bbb1f6801ee66cbccf07 | https://catalog.terradue.com//better-ethz-0000... | 2018-07-04T09:01:09.5700000Z | Pickle file for search (us2000fwnu) | POINT(19.5828 41.4466) |
| 19 | https://store.terradue.com/better-ethz-00002/_... | 2018-07-04T09:01:09.5700000Z | 667cafa12c7c473b05b3b4524d4eb9e479f7f206 | https://catalog.terradue.com//better-ethz-0000... | 2018-07-04T09:01:09.5700000Z | us2000fwnu - GEO_DPHASE | POINT(19.5828 41.4466) |
| 20 | https://store.terradue.com/better-ethz-00002/_... | 2018-07-04T09:01:09.5700000Z | 2b479bfa84941f75ac60ceb16c4637e4487b9689 | https://catalog.terradue.com//better-ethz-0000... | 2018-07-04T09:01:09.5700000Z | us2000fwnu - GEO_AMPL_CHANGE | POINT(19.5828 41.4466) |
| 21 | https://store.terradue.com/better-ethz-00002/_... | 2018-07-04T09:01:09.5700000Z | 486a367edfece142cb28115ce0cb67e5da5530ff | https://catalog.terradue.com//better-ethz-0000... | 2018-07-04T09:01:09.5700000Z | M 5.1 - 8km NW of Vore | Albania (us2000fwnu),POINT(19.5828 41.4466) |
| 22 | https://store.terradue.com/better-ethz-00002/_... | 2018-07-04T09:01:09.5700000Z | 4fcd784dc782662e028369bebc1aae8b780431c3 | https://catalog.terradue.com//better-ethz-0000... | 2018-07-04T09:01:09.5700000Z | Reproducibility notebook for event us2000fwnu | POINT(19.5828 41.4466) |
| 23 | https://store.terradue.com/better-ethz-00002/_... | 2018-07-04T09:01:09.5700000Z | 514ec417ff3dd9e72675d499299465b510c0d3fa | https://catalog.terradue.com//better-ethz-0000... | 2018-07-04T09:01:09.5700000Z | us2000fwnu - GEO_COH_AMPL | POINT(19.5828 41.4466) |
| 24 | https://store.terradue.com/better-ethz-00002/_... | 2017-11-24T21:49:15.1700000Z | 19b6ca32876571a28e754fccaf4bf0f00f872661 | https://catalog.terradue.com//better-ethz-0000... | 2017-11-24T21:49:15.1700000Z | us2000bu6g - GEO_AMPL_CHANGE | POINT(28.6223 37.0845) |
In [8]:
print '%s results have been retrieved' %len(search)
100 results have been retrieved
- Formatting date time
In [9]:
search['startdate'] = pd.to_datetime(search['startdate'])
search['enddate'] = pd.to_datetime(search['enddate'])
In [10]:
search.head(25)
Out[10]:
| enclosure | enddate | identifier | self | startdate | title | wkt | |
|---|---|---|---|---|---|---|---|
| 0 | https://store.terradue.com/better-ethz-00002/_... | 2018-10-28 00:38:11.610 | 17e6950ca6f7bba41e355260a5b4eba51b66decb | https://catalog.terradue.com//better-ethz-0000... | 2018-10-28 00:38:11.610 | us1000hi43 - GEO_AMPL_CHANGE | POINT(26.3972 45.6575) |
| 1 | https://store.terradue.com/better-ethz-00002/_... | 2018-10-28 00:38:11.610 | 3a7914011d350db0841682f8ef4166904a932dfd | https://catalog.terradue.com//better-ethz-0000... | 2018-10-28 00:38:11.610 | us1000hi43 - GEO_COH_AMPL | POINT(26.3972 45.6575) |
| 2 | https://store.terradue.com/better-ethz-00002/_... | 2018-10-28 00:38:11.610 | 8b41d4a479b4ba25001822add9d6b0ba936b5f41 | https://catalog.terradue.com//better-ethz-0000... | 2018-10-28 00:38:11.610 | us1000hi43 - GEO_DCOHER | POINT(26.3972 45.6575) |
| 3 | https://store.terradue.com/better-ethz-00002/_... | 2018-10-28 00:38:11.610 | 929eb1dddd91c5cdd31d00dc2738f76e0a81cf6b | https://catalog.terradue.com//better-ethz-0000... | 2018-10-28 00:38:11.610 | M 5.5 - 15km SE of Comandau | Romania (us1000hi43),POINT(26.3972 45.6575) |
| 4 | https://store.terradue.com/better-ethz-00002/_... | 2018-10-28 00:38:11.610 | c30ab94080fff87625c8946df4bdcd2fd4112596 | https://catalog.terradue.com//better-ethz-0000... | 2018-10-28 00:38:11.610 | Pickle file for search (us1000hi43) | POINT(26.3972 45.6575) |
| 5 | https://store.terradue.com/better-ethz-00002/_... | 2018-10-28 00:38:11.610 | c5edcff5a63f8c3f7250ef1d65a4456f1d1c871c | https://catalog.terradue.com//better-ethz-0000... | 2018-10-28 00:38:11.610 | us1000hi43 - GEO_DPHASE | POINT(26.3972 45.6575) |
| 6 | https://store.terradue.com/better-ethz-00002/_... | 2018-10-28 00:38:11.610 | cb91ca2c7f9c9282d4d36d9e11068760305d91bf | https://catalog.terradue.com//better-ethz-0000... | 2018-10-28 00:38:11.610 | Reproducibility notebook for event us1000hi43 | POINT(26.3972 45.6575) |
| 7 | https://store.terradue.com/better-ethz-00002/_... | 2018-10-28 00:38:11.610 | d4322cb492cda913be0eb46b0c9bca6feb91f7af | https://catalog.terradue.com//better-ethz-0000... | 2018-10-28 00:38:11.610 | Pickle file for selected (us1000hi43) | POINT(26.3972 45.6575) |
| 8 | https://store.terradue.com/better-ethz-00002/_... | 2018-08-11 15:38:33.550 | 1498b0602eb35a3acea447daaffb799eb1039e90 | https://catalog.terradue.com//better-ethz-0000... | 2018-08-11 15:38:33.550 | Reproducibility notebook for event us1000g6wx | POINT(20.1018 41.5869) |
| 9 | https://store.terradue.com/better-ethz-00002/_... | 2018-08-11 15:38:33.550 | 59a78f5ddbac19d1d158358683502c2b5a4cac1b | https://catalog.terradue.com//better-ethz-0000... | 2018-08-11 15:38:33.550 | us1000g6wx - GEO_AMPL_CHANGE | POINT(20.1018 41.5869) |
| 10 | https://store.terradue.com/better-ethz-00002/_... | 2018-08-11 15:38:33.550 | 8a41561287ef3ab36d004f32040a58c3a112c207 | https://catalog.terradue.com//better-ethz-0000... | 2018-08-11 15:38:33.550 | us1000g6wx - GEO_DPHASE | POINT(20.1018 41.5869) |
| 11 | https://store.terradue.com/better-ethz-00002/_... | 2018-08-11 15:38:33.550 | 8c617069f322998c3b5422684eaca4a3698ceb8e | https://catalog.terradue.com//better-ethz-0000... | 2018-08-11 15:38:33.550 | us1000g6wx - GEO_COH_AMPL | POINT(20.1018 41.5869) |
| 12 | https://store.terradue.com/better-ethz-00002/_... | 2018-08-11 15:38:33.550 | b2b646f25e271b9e3a4b171f13197524cf119022 | https://catalog.terradue.com//better-ethz-0000... | 2018-08-11 15:38:33.550 | us1000g6wx - GEO_DCOHER | POINT(20.1018 41.5869) |
| 13 | https://store.terradue.com/better-ethz-00002/_... | 2018-08-11 15:38:33.550 | b55f5de1925a536eb00785a4ee08ddd413e1ac2a | https://catalog.terradue.com//better-ethz-0000... | 2018-08-11 15:38:33.550 | M 5.1 - 8km ESE of Burrel | Albania (us1000g6wx),POINT(20.1018 41.5869) |
| 14 | https://store.terradue.com/better-ethz-00002/_... | 2018-08-11 15:38:33.550 | c44b754bf299a477e0452efa5a0a95c38c001471 | https://catalog.terradue.com//better-ethz-0000... | 2018-08-11 15:38:33.550 | Pickle file for search (us1000g6wx) | POINT(20.1018 41.5869) |
| 15 | https://store.terradue.com/better-ethz-00002/_... | 2018-08-11 15:38:33.550 | f5326479d8cb4d3c95d8204cda88d81eab0ea05f | https://catalog.terradue.com//better-ethz-0000... | 2018-08-11 15:38:33.550 | Pickle file for selected (us1000g6wx) | POINT(20.1018 41.5869) |
| 16 | https://store.terradue.com/better-ethz-00002/_... | 2018-07-04 09:01:09.570 | 25a178045f0bda42b1c12e3a27a4d8b1c7c32cc6 | https://catalog.terradue.com//better-ethz-0000... | 2018-07-04 09:01:09.570 | Pickle file for selected (us2000fwnu) | POINT(19.5828 41.4466) |
| 17 | https://store.terradue.com/better-ethz-00002/_... | 2018-07-04 09:01:09.570 | 39453ed443e42b1b7482648e998aff9241aeb41d | https://catalog.terradue.com//better-ethz-0000... | 2018-07-04 09:01:09.570 | us2000fwnu - GEO_DCOHER | POINT(19.5828 41.4466) |
| 18 | https://store.terradue.com/better-ethz-00002/_... | 2018-07-04 09:01:09.570 | 522e7683a761e9323605bbb1f6801ee66cbccf07 | https://catalog.terradue.com//better-ethz-0000... | 2018-07-04 09:01:09.570 | Pickle file for search (us2000fwnu) | POINT(19.5828 41.4466) |
| 19 | https://store.terradue.com/better-ethz-00002/_... | 2018-07-04 09:01:09.570 | 667cafa12c7c473b05b3b4524d4eb9e479f7f206 | https://catalog.terradue.com//better-ethz-0000... | 2018-07-04 09:01:09.570 | us2000fwnu - GEO_DPHASE | POINT(19.5828 41.4466) |
| 20 | https://store.terradue.com/better-ethz-00002/_... | 2018-07-04 09:01:09.570 | 2b479bfa84941f75ac60ceb16c4637e4487b9689 | https://catalog.terradue.com//better-ethz-0000... | 2018-07-04 09:01:09.570 | us2000fwnu - GEO_AMPL_CHANGE | POINT(19.5828 41.4466) |
| 21 | https://store.terradue.com/better-ethz-00002/_... | 2018-07-04 09:01:09.570 | 486a367edfece142cb28115ce0cb67e5da5530ff | https://catalog.terradue.com//better-ethz-0000... | 2018-07-04 09:01:09.570 | M 5.1 - 8km NW of Vore | Albania (us2000fwnu),POINT(19.5828 41.4466) |
| 22 | https://store.terradue.com/better-ethz-00002/_... | 2018-07-04 09:01:09.570 | 4fcd784dc782662e028369bebc1aae8b780431c3 | https://catalog.terradue.com//better-ethz-0000... | 2018-07-04 09:01:09.570 | Reproducibility notebook for event us2000fwnu | POINT(19.5828 41.4466) |
| 23 | https://store.terradue.com/better-ethz-00002/_... | 2018-07-04 09:01:09.570 | 514ec417ff3dd9e72675d499299465b510c0d3fa | https://catalog.terradue.com//better-ethz-0000... | 2018-07-04 09:01:09.570 | us2000fwnu - GEO_COH_AMPL | POINT(19.5828 41.4466) |
| 24 | https://store.terradue.com/better-ethz-00002/_... | 2017-11-24 21:49:15.170 | 19b6ca32876571a28e754fccaf4bf0f00f872661 | https://catalog.terradue.com//better-ethz-0000... | 2017-11-24 21:49:15.170 | us2000bu6g - GEO_AMPL_CHANGE | POINT(28.6223 37.0845) |
Save the metadataframe¶
Saving the dataframe containing metadata as a pickle allows working with it outside the platform
In [11]:
search.to_pickle('better-ethz-00002_South_Eastern_Europe.pkl')