Build the data transformation application for the Production Centre¶

The Production Centre uses Docker images and deploys them in a Cloud based environment for large scale production campaigns.

This notebook will trigger the build of a Docker image containing the data transformation application.

A few Python modules to do the job:

In [1]:

import os
import time
import jenkins
import getpass
from nbconvert.preprocessors import ExecutePreprocessor, CellExecutionError
import nbformat as nbf

Read the data pipeline configuration information:

In [2]:

%store -r

nb_config = os.path.join('../operations', 'configuration.ipynb')

nb = nbf.read(nb_config, 4)

exec(nb['cells'][1]['source']) in globals(), locals()

app = dict([('artifact_id', app_artifact_id),
            ('folder', folder),
            ('repository', repository),
            ('community', community)])

Create two simple functions to submit the build request and to monitor it:

In [3]:

def submit_build_job(server, job, sleep_secs=3):

    last_build = server.get_job_info(job)['lastCompletedBuild']['number']

    server.build_job(job)

    while server.get_job_info(job)['lastBuild']['number'] == last_build:

        time.sleep(sleep_secs)

    return server.get_job_info(job)['lastBuild']['number']

In [4]:

def monitor_build_job(server, job, sleep_secs=15):

    last_build_number = server.get_job_info(job)['lastBuild']['number']

    while server.get_build_info(job, last_build_number)['building']:

        time.sleep(sleep_secs)

    if str(server.get_build_info(job, last_build_number)['result']) == 'SUCCESS':

        return True

    if str(server.get_build_info(job, last_build_number)['result']) == 'FAILURE':

        return False

Establish a connection with the build component of the Ellip Platform Continuous Integration:

In [5]:

server = jenkins.Jenkins(url=jenkins_url,
                         username=jenkins_username,
                         password=jenkins_api_key)

Create the build request path:

In [6]:

job = 'communities/%s/%s/%s/applications/%s/docker' % (app['repository'],
                                                       app['community'],
                                                       app['folder'],
                                                       app['artifact_id'])

In [7]:

print job

communities/Gitlab Groups/ec-better/wfp/applications/ewf-wfp-01-01-02/docker

Submit the build job:

In [8]:

submit_build_job(server, job)

Out[8]:

Monitor the build request:

In [9]:

if monitor_build_job(server, job):
    print 'Job build completed'
else:
    raise Exception ('Job build failed')

Job build completed

At this stage, there is a Docker container with the data transformation ready to be deployed on a Production Centre. This is accomplished in the deploy step.