{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 4 - Implement the data transformation application steps" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The step takes one Sentinel-1 product, staged-in during [Step 3 - Stage-in the EO data](step-3.ipynb), and generate the backscatter using the Sentinel Application Platform (SNAP) Python bindings." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The data transformation application steps below use a Jupyter Notebook to:\n", "\n", "* Read the Sentinel-1 product using the SNAP reader classes\n", "* Application of orbit file using the SNAP Apply-Orbit-File operator\n", "* Border noise removal using the SNAP ThermalNoiseRemoval operator\n", "* Calibration using the SNAP Calibration operator\n", "* Speckle filtering using the SNAP Speckle-Filter operator\n", "* Terrain correction using the SNAP Terrain-Correction operator\n", "* Conversion to dB using the SNAP linearToFromdB operator" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Procedure" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This step provides an example of data transformation application.\n", "\n", "#### Obtain the notebook file\n", "\n", "* On the JupyterLab Launcher, start a new Terminal\n", "* Type:\n", "\n", "```bash\n", "cd /workspace\n", "git clone https://gitlab.com/ellip/quick-start/jupyterlab/data-transformation.git\n", "```\n", "\n", "* Copy the notebook file into your application. Type:\n", "\n", "```bash\n", "APP_NAME=\n", "\n", "cp -f data-transformation/input.ipynb ${APP_NAME}/src/main/app-resources/notebook/libexec/\n", "```\n", "\n", "* Using the JupyterLab Left Sidebar, navigate to `${APP_NAME}/src/main/app-resources/notebook/libexec`, and open the file `input.ipynb`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Change the service definition" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The data transformation application will be exposed as a Web Processing Service. The definition of the Web Processing Service information such as the _title_ and the _abstract_ is done with a Python dictionary. \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* Change the cell containing the `service` definition, using the proper value for the id, using the value of `APP_NAME`. For instance:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "service = dict([('title', 'Sentinel-1 backscatter timeseries'),\n", " ('abstract', 'Data transformation application - Sentinel-1 backscatter timeseries'),\n", " ('id', 'myapp')])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Run the notebook\n", "\n", "* Type:\n", "\n", "```bash\n", "input_identifier=\"S1A_IW_GRDH_1SDV_20171210T182024_20171210T182049_019644_021603_0A33\"\n", "```\n", "\n", "* Click _Kernel_/_Restart Kernel and Run All Cells_ and wait for the conclusion,\n", "\n", " * This step can take several minutes to conclude. You can be sure that it has finished when all the cells have number within the squared parentheses. On the other hand, the symbol [*] close to a cell means that the execution of that cell is ongoing.\n", "It can happen that the symbol [*] hangs on the last two cells of the notebook. If you observe that it is stuck **for more than 30 minutes**, follow the procedure below to check if the result is complete and valid.\n", "\n", " * Compute the MD5 hash value of the output file:\n", "```bash\n", "md5sum /workspace/${APP_NAME}/src/main/app-resources/notebook/libexec/${input_identifier}_Beta0_VV.tif | awk '{print $1}'\n", "```\n", " * See the following output:\n", "```bash\n", "26fe39e2d58eb4e283bd58b345917bd6\n", "```\n", "If your output corresponds to the above value, you can continue with the procedure. Otherwise, you would need to wait for the completion and possibly retry this check procedure." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* Using the JupyterLab Launcher, start a Terminal and type:\n", "\n", "```bash\n", "mv /workspace/${APP_NAME}/src/main/app-resources/notebook/libexec/${input_identifier}_Beta0_VV.tif /workspace\n", "```\n", "\n", "* Using the JupyterLab Left Sidebar, navigate to the Home and find the product `${input_identifier}_Beta0_VV.tif`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Going further" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Parameter Definition" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The data transformation application may have to expose parameters that can be changed via the Web Processing Service interface at submission time. These parameters are defined using a Python dictionary that defines the parameter identifier, its title and abstract and finally its default value:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "filterSizeX = dict([('id', 'filterSizeX'),\n", " ('value', '5'),\n", " ('title', 'Speckle-Filter filterSizeX'),\n", " ('abstract', 'Set the Speckle-Filter filterSizeX (defaults to 5)')])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To use the parameter value, symply do:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "int(filterSizeX['value'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Runtime parameter definition" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Runtime parameters are mandatory and define those parameters whose values will be changed at runtime.\n", "\n", "These are:\n", " \n", "* `input_identifier` - this is the Sentinel-1 product identifier. At runtime its value is replaced with the Sentinel-1 product identifier being processed\n", "* `input_reference` - this is the Sentinel-1 product catalogue entry URL. At runtime its value is also replaced with the Sentinel-1 product catalogue entry URL being processed\n", "* `data_path` - this is the local path where the Sentinel-1 was staged-in in [Step 3 - Stage-in the EO data](step-3.ipynb). At runtime its value is replaced by a folder with an unique value" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Discover parameters for a given SNAP operator" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You might be wondering how to discover what parameters should be used with a given SNAP operator. The cell below shows how to do it programmatically:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "operator = 'ThermalNoiseRemoval'\n", "\n", "op_spi = GPF.getDefaultInstance().getOperatorSpiRegistry().getOperatorSpi(operator)\n", "\n", "op_params = op_spi.getOperatorDescriptor().getParameterDescriptors()\n", "\n", "for param in op_params:\n", " print(param.getName(), param.getDefaultValue())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Documenting the Jupyter Notebook streaming executable" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One of the nice features of Jupyter Notebooks is that they can incorporate text that documents what is done." ] }, { "cell_type": "markdown", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "The `input.ipynb` file contains already for a proposal for documenting the Sentinel-1 backscatter timeseries data transformation streaming notebook. You can use it as a starting point." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Next step \n", "The next step is to deploy the data transformation as a local Web Processing Servic and test it against a Sentinel-1 product using the Sandbox resources (see [Step 5 - Deploy and run the Web Processing Service locally](step-5.ipynb))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.12" } }, "nbformat": 4, "nbformat_minor": 2 }