Export Python Code Coverage from a Docker Container to Jenkinsby Sebastien Mirolo on Wed, 17 May 2017
The full tests workflow is presented in the diagram below. We presented the steps to build a Docker container and upload it to ECR in another post. Here we will focus on generating Python coverage results and getting the files to be presented into Jenkins.
Actively running the casperjs tests flow against a fully provisioned WebApp will require to
- Bring-up an ECS instance with the Docker container.
- Run the tests from the Jenkins Server against the WebApp HTTP end-point
- Shutdown the container which will have for effect to copy the coverage files on disk.
- Copy the coverage files back to the Jenkins server at an appropriate location and let Jenkins handle presentation of results.
Bring-up the Dockerized WebApp
The major features released in Jenkins v2 are Jenkinsfile and Pipelines. Unfortunately after much tinkering, we found out that these features rely on a new API for plug-ins and Cobertura does not support it yet (Jenkins version 2.32.2, #352, #50). We reverted back to Old-style Jenkins projects and good old fashion Python scripts to drive provisioning (calling on boto3), running tests and gathering results.
The steps to provision an ECS cluster and deploy a Docker container inside it are somewhat generic (see drundocker.py for details). What is most interesting is how to setup our Docker container and web application to make it all work.
Configuration for Python coverage
The whole point of building a Docker container is to have the same container
used during testing and deployed in production. We are thus very careful to
only install packages required for production use.
For example, we purposely insure that
django-extensions are not
installed in the container.
An exception to that rule is
On a side note, we also don't want to
(statement in the Dockerfile) a module that requires a native module and ends up
gcc. Ideally when building a Docker container, we would
want all packages (Python or otherwise)
installed through the system package manager.
wsgi.py such that code coverage is recorded and saved
import os import signal def save_coverage(*args, **kwargs): sys.stderr.write("saving coverage\n") cov.stop() cov.save() if os.getenv('DJANGO_COVERAGE'): import atexit, sys import coverage cov = coverage.coverage(data_file=os.path.join(os.getenv('DJANGO_COVERAGE'), ".coverage.%d" % os.getpid())) cov.start() atexit.register(save_coverage) try: signal.signal(signal.SIGTERM, save_coverage) except ValueError as e: # trapping signals does not work with manage.py # trying to do so fails with ValueError. # Signal only works in main thread. pass
Flexible config files
We need a way to start a Docker container with either tests or production config
files. Extending on the environment variables approach described in
Docker on Elastic Beanstalk,
we add the following code in
from deployutils import load_config, update_settings APP_NAME = os.path.basename(BASE_DIR) update_settings(sys.modules[__name__], load_config(APP_NAME, 'credentials', 'site.conf', location=os.getenv("SETTINGS_LOCATION", None), passphrase=os.getenv("SETTINGS_CRYPT_KEY", None)))
This way we pass two environment variables to the Docker container which
are the location of the config files and a key to decrypt them.
In practice we use a
credentials file that holds secrets such as
API keys, e-mail passwords, etc. and a
site.conf file that holds
database names, service end points, etc.
Gathering the coverage files
When running the container for testing, we mount a directory on the host filesystem inside the container such that coverage files are written there. Later on it is only a matter of copying the files back from the host machine to the Jenkins server.
There was a bit of trial and error to get coverage files written out when the gunicorn daemon is shutdown but we finally got it to work as is reflected in the wsgi.py code above.
We still had to massage the coverage output files a bit because of Cobertura / Python Coverage issues but otherwise we are done. We have code coverage displayed in Jenkins for the Django WebApp running inside a Docker container.
More to read
If you are looking for related posts, Jenkins, Docker build and Amazon EC2 Container Registry and Docker on Elastic Beanstalk are good reads.