Build the data transformation application for the Production CentreΒΆ
The Production Centre uses Docker images and deploys them in a Cloud based environment for large scale production campaigns.
This notebook will trigger the build of a Docker image containing the data transformation application.
- A few Python modules to do the job:
In [ ]:
import os
import time
import jenkins
import getpass
import nbformat as nbf
- Read the data pipeline configuration information:
In [ ]:
%store -r
nb_config = os.path.join('..', 'configuration.ipynb')
nb = nbf.read(nb_config, 4)
exec(nb['cells'][1]['source']) in globals(), locals()
app = dict([('artifact_id', app_artifact_id),
('folder', folder),
('repository', repository),
('community', community)])
- Create two simple functions to submit the build request and to monitor it:
In [ ]:
def submit_build_job(server, job, sleep_secs=3):
last_build = server.get_job_info(job)['lastCompletedBuild']['number']
server.build_job(job)
while server.get_job_info(job)['lastBuild']['number'] == last_build:
time.sleep(sleep_secs)
return server.get_job_info(job)['lastBuild']['number']
In [ ]:
def monitor_build_job(server, job, sleep_secs=15):
last_build_number = server.get_job_info(job)['lastBuild']['number']
while server.get_build_info(job, last_build_number)['building']:
time.sleep(sleep_secs)
if str(server.get_build_info(job, last_build_number)['result']) == 'SUCCESS':
return True
if str(server.get_build_info(job, last_build_number)['result']) == 'FAILURE':
return False
- Establish a connection with the build component of the Ellip Platform Continuous Integration:
In [ ]:
server = jenkins.Jenkins(url=jenkins_url,
username=jenkins_username,
password=jenkins_api_key)
- Create the build request path:
In [ ]:
job = 'communities/%s/%s/%s/applications/%s/docker' % (app['repository'],
app['community'],
app['folder'],
app['artifact_id'])
In [ ]:
print job
- Submit the build job:
In [ ]:
submit_build_job(server, job)
- Monitor the build request:
In [ ]:
if monitor_build_job(server, job):
print 'Job build completed'
else:
raise Exception ('Job build failed')
At this stage, there is a Docker container with the data transformation ready to be deployed on a Production Centre. This is accomplished in the deploy step.