WFP-01-03-02 Build the data transformation application for the Production Centre - CHIRPS Rainfall Estimates (RFE) - Aggregations and Anomalies¶

The Production Centre uses Docker images and deploys them in a Cloud based environment for large scale production campaigns.

A few Python modules to do the job:

In [1]:

app = dict([('artifact_id', 'ewf-wfp-01-03-02'),
            ('repository', 'Gitlab Groups'),
            ('community', 'ec-better'),
            ('folder', 'wfp')])

In [2]:

import getpass
import jenkins
import time
import lxml.etree as etree
import sys
import requests
import os
import string
import hashlib
import urllib2
import pytz
from datetime import datetime
import time
import owslib
from owslib.wps import monitorExecution
import uuid
from owslib.wps import WebProcessingService
import getpass
from nbconvert.preprocessors import ExecutePreprocessor, CellExecutionError
import nbformat as nbf

username = raw_input("Provide the Continuous Integration user username? ")
api_key = getpass.getpass('And its associated API key:')

Read the data pipeline configuration information:

In [9]:

%store -r

nb_config = os.path.join('../operations', 'configuration.ipynb')

nb = nbf.read(nb_config, 4)

exec(nb['cells'][1]['source']) in globals(), locals()

app = dict([('artifact_id', app_artifact_id),
            ('folder', folder),
            ('repository', repository),
            ('community', community)])

Create two simple functions to submit the build request and to monitor it:

In [10]:

def submit_build_job(server, job, sleepSecs=3):

    last_build = server.get_job_info(job)['lastCompletedBuild']['number']

    server.build_job(job)

    while server.get_job_info(job)['lastBuild']['number'] == last_build:

        time.sleep(sleepSecs)

    return server.get_job_info(job)['lastBuild']['number']

In [11]:

def monitor_build_job(server, job, sleepSecs=15):

    last_build_number = server.get_job_info(job)['lastBuild']['number']

    while server.get_build_info(job, last_build_number)['building']:

        time.sleep(sleepSecs)

    if str(server.get_build_info(job, last_build_number)['result']) == 'SUCCESS':

        return True

    if str(server.get_build_info(job, last_build_number)['result']) == 'FAILURE':

        return False

Establish a connection with the build component of the Ellip Plataform Continuous Integration:

In [12]:

server = jenkins.Jenkins(url='https://build.terradue.com',
                        username=username,
                        password=api_key)

job = 'communities/%s/%s/%s/applications/%s/docker' % (app['repository'],
                                                       app['community'],
                                                       app['folder'],
                                                       app['artifact_id'])
print(job)

communities/Gitlab Groups/ec-better/wfp/applications/ewf-wfp-01-03-02/docker

Submit the build job:

In [13]:

submit_build_job(server, job)

Out[13]:

Monitor the build request:

In [14]:

if monitor_build_job(server, job):
    print 'Job build completed'
else:
    print 'Job build failed'

Job build completed