# GitLab CI template for Python
This project implements a generic GitLab CI template for [Python](https://www.python.org/).
It provides several features, usable in different modes (by configuration).
## Usage
In order to include this template in your project, add the following to your `gitlab-ci.yml`:
```yaml
include:
- project: 'to-be-continuous/python'
ref: '5.0.0'
file: '/templates/gitlab-ci-python.yml'
```
## Global configuration
The Python template uses some global configuration used throughout all jobs.
| Name | description | default value |
| -------------------- | ------------------------------------------------------------------------------------- | ------------------ |
| `PYTHON_IMAGE` | The Docker image used to run Python <br/>:warning: **set the version required by your project** | `python:3` |
| `PYTHON_PROJECT_DIR` | Python project root directory | `.` |
| `PYTHON_BUILD_SYSTEM`| Python build-system to use to install dependencies, build and package the project (see below) | _none_ (auto-detect) |
| `PIP_INDEX_URL` | Python repository url | _none_ |
| `PIP_OPTS` | pip [extra options](https://pip.pypa.io/en/stable/reference/pip/#general-options) | _none_ |
| `PYTHON_EXTRA_DEPS` | Python extra sets of dependencies to install<br/>For [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies) or [Poetry](https://python-poetry.org/docs/pyproject/#extras) only | _none_ |
| `PYTHON_REQS_FILE` | Main requirements file _(relative to `$PYTHON_PROJECT_DIR`)_<br/>For [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) build-system only | `requirements.txt` |
| `PYTHON_EXTRA_REQS_FILES` | Extra dev requirements file(s) to install _(relative to `$PYTHON_PROJECT_DIR`)_ | `requirements-dev.txt` |
The cache policy also makes the necessary to manage pip cache (not to download Python dependencies over and over again).
## Multi build-system support
The Python template supports the most popular dependency management & build systems.
By default it tries to auto-detect the build system used by the project (based on the presence of `pyproject.toml`
and/or `setup.py` and/or `requirements.txt`), but the build system might also be set explicitly using the
`$PYTHON_BUILD_SYSTEM` variable:
| Value | Build System (scope) |
| ---------------- | ---------------------------------------------------------- |
| _none_ (default) or `auto` | The template tries to **auto-detect** the actual build system |
| `setuptools` | [Setuptools](https://setuptools.pypa.io/) (dependencies, build & packaging) |
| `poetry` | [Poetry](https://python-poetry.org/) (dependencies, build, test & packaging) |
| `pipenv` | [Pipenv](https://pipenv.pypa.io/) (dependencies only) |
| `reqfile` | [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) (dependencies only) |
:warning: You can explicitly set the build tool version by setting `$PYTHON_BUILD_SYSTEM` variable including a [version identification](https://peps.python.org/pep-0440/). For example `PYTHON_BUILD_SYSTEM="poetry==1.1.15"`
## Jobs
### `py-package` job
This job allows building your Python project [distribution packages](https://packaging.python.org/en/latest/glossary/#term-Distribution-Package).
It is bound to the `build` stage, it is **disabled by default** and can be enabled by setting `$PYTHON_PACKAGE_ENABLED` to `true`.
### Lint jobs
#### `py-pylint` job
This job is **disabled by default** and performs code analysis based on [pylint](http://pylint.pycqa.org/en/latest/) Python lib.
It is activated by setting `$PYLINT_ENABLED` to `true`.
It is bound to the `build` stage, and uses the following variables:
| Name | description | default value |
| ------------------------ | ---------------------------------- | ----------------- |
| `PYLINT_ARGS` | Additional [pylint CLI options](http://pylint.pycqa.org/en/latest/user_guide/run.html#command-line-options) | _none_ |
| `PYLINT_FILES` | Files or directories to analyse | _none_ (by default analyses all found python source files) |
This job produces the following artifacts, kept for one day:
* Code quality json report in code climate format.
* Pylint report for SonarQube (if `SONAR_URL` is defined).
### Test jobs
The Python template features four alternative test jobs:
* `py-unittest` that performs tests based on [unittest](https://docs.python.org/3/library/unittest.html) Python lib,
* or `py-pytest` that performs tests based on [pytest](https://docs.pytest.org/en/latest/) Python lib,
* or `py-nosetest` that performs tests based on [nose](https://nose.readthedocs.io/en/latest/) Python lib,
* or `py-compile` that performs byte code generation to check syntax if not tests are available.
#### `py-unittest` job
This job is **disabled by default** and performs tests based on [unittest](https://docs.python.org/3/library/unittest.html) Python lib.
It is activated by setting `$UNITTEST_ENABLED` to `true`.
In order to produce JUnit test reports, the tests are executed with the [xmlrunner](https://github.com/xmlrunner/unittest-xml-reporting) module.
It is bound to the `build` stage, and uses the following variables:
| Name | description | default value |
| ------------------------ | -------------------------------------------------------------------- | ----------------------- |
| `UNITTEST_ARGS` | Additional xmlrunner/unittest CLI options | _none_ |
This job produces the following artifacts, kept for one day:
* JUnit test report (using the [xmlrunner](https://github.com/xmlrunner/unittest-xml-reporting) module)
* code coverage report (cobertura xml format).
:warning: code coverage report artifact is disabled, due to a deprecated syntax, see [Activate code coverage report artifact](#activate-code-coverage-report-artifact)
:warning: create a `.coveragerc` file at the root of your Python project to control the coverage settings.
Example:
```conf
[run]
# enables branch coverage
branch = True
# list of directories/packages to cover
source =
module_1
module_2
```
#### `py-pytest` job
This job is **disabled by default** and performs tests based on [pytest](https://docs.pytest.org/en/latest/) Python lib.
It is activated by setting `$PYTEST_ENABLED` to `true`.
It is bound to the `build` stage, and uses the following variables:
| Name | description | default value |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
| `PYTEST_ARGS` | Additional [pytest](https://docs.pytest.org/en/stable/usage.html) or [pytest-cov](https://github.com/pytest-dev/pytest-cov#usage) CLI options | _none_ |
This job produces the following artifacts, kept for one day:
* JUnit test report (with the [`--junit-xml`](http://doc.pytest.org/en/latest/usage.html#creating-junitxml-format-files) argument)
* code coverage report (cobertura xml format).
:warning: code coverage report artifact is disabled, due to a deprecated syntax, see [Activate code coverage report artifact](#activate-code-coverage-report-artifact)
:warning: create a `.coveragerc` file at the root of your Python project to control the coverage settings.
Example:
```conf
[run]
# enables branch coverage
branch = True
# list of directories/packages to cover
source =
module_1
module_2
```
#### `py-nosetest` job
This job is **disabled by default** and performs tests based on [nose](https://nose.readthedocs.io/en/latest/) Python lib.
It is activated by setting `$NOSETESTS_ENABLED` to `true`.
It is bound to the `build` stage, and uses the following variables:
| Name | description | default value |
| ------------------------ | --------------------------------------------------------------------------------------- | ----------------------- |
| `NOSETESTS_ARGS` | Additional [nose CLI options](https://nose.readthedocs.io/en/latest/usage.html#options) | _none_ |
By default coverage will be run on all the directory. You can restrict it to your packages by setting NOSE_COVER_PACKAGE variable.
More [info](https://nose.readthedocs.io/en/latest/plugins/cover.html)
This job produces the following artifacts, kept for one day:
* JUnit test report (with the [`--with-xunit`](https://nose.readthedocs.io/en/latest/plugins/xunit.html) argument)
* code coverage report (cobertura xml format + html report).
:warning: code coverage report artifact is disabled, due to a deprecated syntax, see [Activate code coverage report artifact](#activate-code-coverage-report-artifact)
:warning: create a `.coveragerc` file at the root of your Python project or use [nose CLI options](https://nose.readthedocs.io/en/latest/plugins/cover.html#options) to control the coverage settings.
#### `py-compile` job
This job is a fallback if no unit test has been setup (`$UNITTEST_ENABLED` and `$PYTEST_ENABLED` and `$NOSETEST_ENABLED`
are not set), and performs a [`compileall`](https://docs.python.org/3/library/compileall.html).
It is bound to the `build` stage, and uses the following variables:
| Name | description | default value |
| --------------------- | ----------------------------------------------------------------------------- | ------------- |
| `PYTHON_COMPILE_ARGS` | [`compileall` CLI options](https://docs.python.org/3/library/compileall.html) | `*` |
#### Activate code coverage report artifact
Code coverage report artifact is disabled, due to a [deprecated syntax](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78132).
In order to activate code coverage report artifact, you need to override your actual unit test job depending on our GitLab version.
Here is an example with `py-pytest` job (change to `py-unittest` or `py-nosetests` depending on your unit tests library):
* for GitLab < 14.10:
```yaml
py-pytest:
artifacts:
reports:
cobertura: $PYTHON_PROJECT_DIR/reports/coverage.xml
```
* for GitLab >= 14.10:
```yaml
py-pytest:
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: $PYTHON_PROJECT_DIR/reports/coverage.xml
```
### SonarQube analysis
If you're using the SonarQube template to analyse your Python code, here is a sample `sonar-project.properties` file:
```properties
# see: https://docs.sonarqube.org/latest/analysis/languages/python/
# set your source directory(ies) here (relative to the sonar-project.properties file)
sonar.sources=.
# exclude unwanted directories and files from being analysed
sonar.exclusions=**/test_*.py
# set your tests directory(ies) here (relative to the sonar-project.properties file)
sonar.tests=.
sonar.test.inclusions=**/test_*.py
# tests report: generic format
sonar.python.xunit.reportPath=reports/unittest/TEST-*.xml
# coverage report: XUnit format
sonar.python.coverage.reportPaths=reports/coverage.xml
```
More info:
* [Python language support](https://docs.sonarqube.org/latest/analysis/languages/python/)
* [test coverage & execution parameters](https://docs.sonarqube.org/latest/analysis/coverage/)
* [third-party issues](https://docs.sonarqube.org/latest/analysis/external-issues/)
### `py-bandit` job (SAST)
This job is **disabled by default** and performs a [Bandit](https://pypi.org/project/bandit/) analysis.
It is bound to the `test` stage, and uses the following variables:
| Name | description | default value |
| ---------------- | ---------------------------------------------------------------------- | ----------------- |
| `BANDIT_ENABLED` | Set to `true` to enable Bandit analysis | _none_ (disabled) |
| `BANDIT_ARGS` | Additional [Bandit CLI options](https://github.com/PyCQA/bandit#usage) | `--recursive .` |
This job outputs a **textual report** in the console, and in case of failure also exports a JSON report in the `reports/`
directory _(relative to project root dir)_.
### `py-safety` job (dependency check)
This job is **disabled by default** and performs a dependency check analysis using [Safety](https://pypi.org/project/safety/).
It is bound to the `test` stage, and uses the following variables:
| Name | description | default value |
| ---------------- | ----------------------------------------------------------------------- | ----------------- |
| `SAFETY_ENABLED` | Set to `true` to enable Safety job | _none_ (disabled) |
| `SAFETY_ARGS` | Additional [Safety CLI options](https://github.com/pyupio/safety#usage) | `--full-report` |
This job outputs a **textual report** in the console, and in case of failure also exports a JSON report in the `reports/`
directory _(relative to project root dir)_.
### `py-trivy` job (dependency check)
This job is **disabled by default** and performs a dependency check analysis using [Trivy](https://github.com/aquasecurity/trivy/).
It is bound to the `test` stage, and uses the following variables:
| Name | description | default value |
| ---------------- | ----------------------------------------------------------------------- | ----------------- |
| `PYTHON_TRIVY_ENABLED` | Set to `true` to enable Trivy job | _none_ (disabled) |
| `PYTHON_TRIVY_ARGS` | Additional [Trivy CLI options](https://aquasecurity.github.io/trivy/v0.21.1/getting-started/cli/fs/) | `--vuln-type library` |
This job outputs a **textual report** in the console, and in case of failure also exports a JSON report in the `reports/`
directory _(relative to project root dir)_.
### `py-release` job
This job is **disabled by default** and allows to perform a complete release of your Python code:
1. increase the Python project version,
2. Git commit changes and create a Git tag with the new version number,
3. build the [Python packages](https://packaging.python.org/),
4. publish the built packages to a PyPI compatible repository ([GitLab packages](https://docs.gitlab.com/ee/user/packages/pypi_repository/) by default).
The Python template supports two packaging systems:
* [Poetry](https://python-poetry.org/): uses Poetry-specific [version](https://python-poetry.org/docs/cli/#version), [build](https://python-poetry.org/docs/cli/#build) and [publish](https://python-poetry.org/docs/cli/#publish) commands.
* [Setuptools](https://setuptools.pypa.io/): uses [Bumpversion](https://github.com/peritus/bumpversion) as version management, [build](https://pypa-build.readthedocs.io/) as package builder and [Twine](https://twine.readthedocs.io/) to publish.
The release job is bound to the `publish` stage, appears only on production and integration branches and uses the following variables:
| Name | description | default value |
| ----------------------- | ----------------------------------------------------------------------- | ----------------- |
| `PYTHON_RELEASE_ENABLED`| Set to `true` to enable the release job | _none_ (disabled) |
| `PYTHON_RELEASE_NEXT` | The part of the version to increase (one of: `major`, `minor`, `patch`) | `minor` |
| `PYTHON_SEMREL_RELEASE_DISABLED`| Set to `true` to disable [semantic-release integration](#semantic-release-integration) | _none_ (disabled) |
| `GIT_USERNAME` | Git username for Git push operations (see below) | _none_ |
| :lock: `GIT_PASSWORD` | Git password for Git push operations (see below) | _none_ |
| :lock: `GIT_PRIVATE_KEY`| SSH key for Git push operations (see below) | _none_ |
| `PYTHON_REPOSITORY_URL`| Target PyPI repository to publish packages | _[GitLab project's PyPI packages repository](https://docs.gitlab.com/ee/user/packages/pypi_repository/)_ |
| `PYTHON_REPOSITORY_USERNAME`| Target PyPI repository username credential | `gitlab-ci-token` |
| :lock: `PYTHON_REPOSITORY_PASSWORD`| Target PyPI repository password credential | `$CI_JOB_TOKEN` |
#### Setuptools tip
If you're using a `setup.cfg` declarative file for your project Setuptools configuration, then you will have to write a
`.bumpversion.cfg` file to workaround a bug that prevents Bumpversion from updating the project version in your `setup.cfg` file.
Example of `.bumpversion.cfg` file:
```ini
[bumpversion]
# same version as in your setup.cfg
current_version = 0.5.0
[bumpversion:file:setup.cfg]
# any additional config here
# see: https://github.com/peritus/bumpversion#file-specific-configuration
```
#### `semantic-release` integration
If you activate the [`semantic-release-info` job from the `semantic-release` template](https://gitlab.com/to-be-continuous/semantic-release/#semantic-release-info-job), the `py-release` job will rely on the generated next version info.
Thus, a release will be performed only if a next semantic release is present.
You should disable the `semantic-release` job (as it's the `py-release` job that will perform the release and so we only need the `semantic-release-info` job) by setting `SEMREL_RELEASE_DISABLED` to `true`.
Finally, the semantic-release integration can be disabled with the `PYTHON_SEMREL_RELEASE_DISABLED` variable.
#### Git authentication
A Python release involves some Git push operations.
You can either use a SSH key or user/password credentials.
##### Using a SSH key
We recommend you to use a [project deploy key](https://docs.gitlab.com/ee/user/project/deploy_keys/#project-deploy-keys) with write access to your project.
The key should not have a passphrase (see [how to generate a new SSH key pair](https://docs.gitlab.com/ce/ssh/README.html#generating-a-new-ssh-key-pair)).
Specify :lock: `$GIT_PRIVATE_KEY` as secret project variable with the private part of the deploy key.
```PEM
-----BEGIN 0PENSSH PRIVATE KEY-----
blablabla
-----END OPENSSH PRIVATE KEY-----
```
The template handles both classic variable and file variable.
##### Using user/password credentials
Simply specify :lock: `$GIT_USERNAME` and :lock: `$GIT_PASSWORD` as secret project variables.
Note that the password should be an access token (preferably a [project](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) or [group](https://docs.gitlab.com/ee/user/group/settings/group_access_tokens.html) access token) with `read_repository` and `write_repository` scopes.