GitLab CI template for Python
This project implements a generic GitLab CI template for Python.
It provides several features, usable in different modes (by configuration) following those recommendations
Usage
In order to include this template in your project, add the following to your gitlab-ci.yml
:
include:
- project: 'to-be-continuous/python'
ref: '3.2.1'
file: '/templates/gitlab-ci-python.yml'
Global configuration
The Python template uses some global configuration used throughout all jobs.
Name | description | default value |
---|---|---|
PYTHON_IMAGE |
The Docker image used to run Python ⚠️ set the version required by your project |
python:3 |
PYTHON_PROJECT_DIR |
Python project root directory | . |
PYTHON_BUILD_SYSTEM |
Python build-system to use to install dependencies, build and package the project (see below) | none (auto-detect) |
PIP_INDEX_URL |
Python repository url | none |
PIP_OPTS |
pip extra options | none |
PYTHON_EXTRA_DEPS |
Python extra sets of dependencies to install For Setuptools or Poetry only |
none |
REQUIREMENTS_FILE |
Name of requirements file (relative to $PYTHON_PROJECT_DIR )For Requirements Files build-system only |
requirements.txt |
TEST_REQUIREMENTS_FILE |
Name of dev/test requirements file (relative to $PYTHON_PROJECT_DIR )
|
test-requirements.txt |
The cache policy also makes the necessary to manage pip cache (not to download Python dependencies over and over again).
Multi build-system support
The Python template supports 3 popular dependency management & build systems:
- Setuptools,
- Poetry,
- Requirements Files (dependency management only).
By default the template tries to auto-detect the build system used by the project (based on presence of pyproject.toml
and/or setup.py
and/or requirements.txt
), but the build system might also be explicitly set using the $PYTHON_BUILD_SYSTEM
variable.
Supported values of $PYTHON_BUILD_SYSTEM
:
Value | Description |
---|---|
none (default) | The template tries to auto-detect the actual build system, based of the presence of some key files |
setuptools |
Setuptools will be used to install dependencies, build and package the project |
poetry |
Poetry will be used to install dependencies, build, test and package the project |
reqfile |
Requirements Files will be used to install dependencies |
Jobs
Lint jobs
py-pylint
job
This job is disabled by default and performs code analysis based on pylint Python lib.
It is activated by setting $PYLINT_ENABLED
to true
.
It is bound to the build
stage, and uses the following variables:
Name | description | default value |
---|---|---|
PYLINT_ARGS |
Additional pylint CLI options | none |
PYLINT_FILES |
Files or directories to analyse | none (by default analyses all found python source files) |
This job produces the following artifacts, kept for one day:
- Code quality json report in code climate format.
Test jobs
The Python template features four alternative test jobs:
-
py-unittest
that performs tests based on unittest Python lib, - or
py-pytest
that performs tests based on pytest Python lib, - or
py-nosetest
that performs tests based on nose Python lib, - or
py-compile
that performs byte code generation to check syntax if not tests are available.
py-unittest
job
This job is disabled by default and performs tests based on unittest Python lib.
It is activated by setting $UNITTEST_ENABLED
to true
.
In order to produce JUnit test reports, the tests are executed with the xmlrunner module.
It is bound to the build
stage, and uses the following variables:
Name | description | default value |
---|---|---|
UNITTEST_ARGS |
Additional xmlrunner/unittest CLI options | none |
This job produces the following artifacts, kept for one day:
- JUnit test report (using the xmlrunner module)
- code coverage report (cobertura xml format).
⚠️ create a .coveragerc
file at the root of your Python project to control the coverage settings.
Example:
[run]
# enables branch coverage
branch = True
# list of directories/packages to cover
source =
module_1
module_2
py-pytest
job
This job is disabled by default and performs tests based on pytest Python lib.
It is activated by setting $PYTEST_ENABLED
to true
.
It is bound to the build
stage, and uses the following variables:
Name | description | default value |
---|---|---|
PYTEST_ARGS |
Additional pytest or pytest-cov CLI options | none |
This job produces the following artifacts, kept for one day:
- JUnit test report (with the
--junit-xml
argument) - code coverage report (cobertura xml format).
⚠️ create a .coveragerc
file at the root of your Python project to control the coverage settings.
Example:
[run]
# enables branch coverage
branch = True
# list of directories/packages to cover
source =
module_1
module_2
py-nosetest
job
This job is disabled by default and performs tests based on nose Python lib.
It is activated by setting $NOSETESTS_ENABLED
to true
.
It is bound to the build
stage, and uses the following variables:
Name | description | default value |
---|---|---|
NOSETESTS_ARGS |
Additional nose CLI options | none |
By default coverage will be run on all the directory. You can restrict it to your packages by setting NOSE_COVER_PACKAGE variable. More info
This job produces the following artifacts, kept for one day:
- JUnit test report (with the
--with-xunit
argument) - code coverage report (cobertura xml format + html report).
⚠️ create a .coveragerc
file at the root of your Python project or use nose CLI options to control the coverage settings.
py-compile
job
This job is a fallback if no unit test has been setup ($UNITTEST_ENABLED
and $PYTEST_ENABLED
and $NOSETEST_ENABLED
are not set), and performs a compileall
.
It is bound to the build
stage, and uses the following variables:
Name | description | default value |
---|---|---|
PYTHON_COMPILE_ARGS |
compileall CLI options |
* |
SonarQube analysis
If you're using the SonarQube template to analyse your Python code, here is a sample sonar-project.properties
file:
# see: https://docs.sonarqube.org/latest/analysis/languages/python/
# set your source directory(ies) here (relative to the sonar-project.properties file)
sonar.sources=.
# exclude unwanted directories and files from being analysed
sonar.exclusions=**/test_*.py
# set your tests directory(ies) here (relative to the sonar-project.properties file)
sonar.tests=.
sonar.test.inclusions=**/test_*.py
# tests report: generic format
sonar.python.xunit.reportPath=reports/unittest/TEST-*.xml
# coverage report: XUnit format
sonar.python.coverage.reportPaths=reports/coverage.xml
More info:
py-bandit
job (SAST)
This job is disabled by default and performs a Bandit analysis.
It is bound to the test
stage, and uses the following variables:
Name | description | default value |
---|---|---|
BANDIT_ENABLED |
Set to true to enable Bandit analysis |
none (disabled) |
BANDIT_ARGS |
Additional Bandit CLI options | --recursive . |
This job outputs a textual report in the console, and in case of failure also exports a JSON report in the reports/
directory (relative to project root dir).
py-safety
job (dependency check)
This job is disabled by default and performs a dependency check analysis using Safety.
It is bound to the test
stage, and uses the following variables:
Name | description | default value |
---|---|---|
SAFETY_ENABLED |
Set to true to enable Safety job |
none (disabled) |
SAFETY_ARGS |
Additional Safety CLI options | --full-report |
This job outputs a textual report in the console, and in case of failure also exports a JSON report in the reports/
directory (relative to project root dir).
py-trivy
job (dependency check)
This job is disabled by default and performs a dependency check analysis using Trivy.
It is bound to the test
stage, and uses the following variables:
Name | description | default value |
---|---|---|
PYTHON_TRIVY_ENABLED |
Set to true to enable Trivy job |
none (disabled) |
PYTHON_TRIVY_ARGS |
Additional Trivy CLI options | --vuln-type library |
This job outputs a textual report in the console, and in case of failure also exports a JSON report in the reports/
directory (relative to project root dir).
Package jobs
py-package
job
This job is disabled by default and performs a packaging of your Python code.
It is bound to the package-build
stage, applies only on git tags and uses the following variables:
Name | description | default value |
---|---|---|
PYTHON_FORCE_PACKAGE |
Set to true to force the packaging even if not on tag related event |
none (disabled) |
Publish jobs
py-release
job
This job is disabled by default and performs an automatic tagging of your Python code.
- Bumpversion Python library is used for version management.
- Looks for an existing
.bumpversion.cfg
at the project root. If found, it will be the configuration used by bumpversion. If not, the$RELEASE_VERSION_PART
variable andsetup.py
will be used instead. - Creating a Git tag involves an authenticated and authorized Git user.
Don't use your personal password !!! Use an access token with write_repository rights. If you have a generic account, add it to the project and generate access token from this account.
It is bound to the publish
stage, applies only on master branch and uses the following variables:
Name | description | default value |
---|---|---|
RELEASE_VERSION_PART |
The part of the version to increase (one of: major , minor , patch ) |
minor |
RELEASE_USERNAME |
Username credential for git push | none (disabled) |
RELEASE_ACCESS_TOKEN |
Password credential for git push | none |
py-publish
job
This job is disabled by default and performs a publication of your Python code.
It is bound to the publish
stage, applies only on git tags and uses the following variables:
Name | description | default value |
---|---|---|
PYTHON_PUBLISH_ENABLED |
Set to true to enable the publish job |
none (disabled) |
TWINE_REPOSITORY_URL |
Where to publish your Python project | GitLab Project's Pypi Packages registry |
TWINE_USERNAME |
Username credential to publish to $TWINE_REPOSITORY_URL | gitlab-ci-token |
TWINE_PASSWORD |
Password credential to publish to $TWINE_REPOSITORY_URL | $CI_JOB_TOKEN |
More info:
If you want to automatically create tag and publish your Python package, please have a look here
GitLab compatibility
ℹ️ This template is actually tested and validated on GitLab Community Edition instance version 13.12.11