# GitLab CI template for Python

This project implements a generic GitLab CI template for [Python](https://www.python.org/).

It provides several features, usable in different modes (by configuration) following those [recommendations](to-be-continuous.gitlab.io/doc/usage/)

## Usage

In order to include this template in your project, add the following to your `gitlab-ci.yml`:

```yaml
include:
  - project: 'to-be-continuous/python'
    ref: '3.2.1'
    file: '/templates/gitlab-ci-python.yml'
```

## Global configuration

The Python template uses some global configuration used throughout all jobs.

| Name                 | description                                                                           | default value      |
| -------------------- | ------------------------------------------------------------------------------------- | ------------------ |
| `PYTHON_IMAGE`       | The Docker image used to run Python <br/>:warning: **set the version required by your project** | `python:3` |
| `PYTHON_PROJECT_DIR` | Python project root directory                                                         | `.`                |
| `PYTHON_BUILD_SYSTEM`| Python build-system to use to install dependencies, build and package the project (see below) | _none_ (auto-detect) |
| `PIP_INDEX_URL`      | Python repository url                                                                 | _none_             |
| `PIP_OPTS`           | pip [extra options](https://pip.pypa.io/en/stable/reference/pip/#general-options)     | _none_             |
| `PYTHON_EXTRA_DEPS`  | Python extra sets of dependencies to install<br/>For [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies) or [Poetry](https://python-poetry.org/docs/pyproject/#extras) only | _none_ |
| `REQUIREMENTS_FILE`  | Name of requirements file _(relative to `$PYTHON_PROJECT_DIR`)_<br/>For [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) build-system only | `requirements.txt` |
| `TEST_REQUIREMENTS_FILE` | Name of dev/test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ | `test-requirements.txt` |

The cache policy also makes the necessary to manage pip cache (not to download Python dependencies over and over again).

## Multi build-system support

The Python template supports 3 popular dependency management & build systems:

* [Setuptools](https://setuptools.pypa.io/),
* [Poetry](https://python-poetry.org/),
* [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) (dependency management only).

By default the template tries to auto-detect the build system used by the project (based on presence of `pyproject.toml` 
and/or `setup.py` and/or `requirements.txt`), but the build system might also be explicitly set using the `$PYTHON_BUILD_SYSTEM` variable.

Supported values of `$PYTHON_BUILD_SYSTEM`:

| Value            | Description                                                |
| ---------------- | ---------------------------------------------------------- |
| _none_ (default) | The template tries to auto-detect the actual build system, based of the presence of some key files |
| `setuptools`     | [Setuptools](https://setuptools.pypa.io/) will be used to install dependencies, build and package the project |
| `poetry`         | [Poetry](https://python-poetry.org/) will be used to install dependencies, build, test and package the project |
| `reqfile`        | [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) will be used to install dependencies |

## Jobs

### Lint jobs

#### `py-pylint` job

This job is **disabled by default** and performs code analysis based on [pylint](http://pylint.pycqa.org/en/latest/) Python lib.
It is activated by setting `$PYLINT_ENABLED` to `true`.

It is bound to the `build` stage, and uses the following variables:

| Name                     | description                        | default value     |
| ------------------------ | ---------------------------------- | ----------------- |
| `PYLINT_ARGS`            | Additional [pylint CLI options](http://pylint.pycqa.org/en/latest/user_guide/run.html#command-line-options)      |  _none_           |
| `PYLINT_FILES`           | Files or directories to analyse   | _none_ (by default analyses all found python source files) |

This job produces the following artifacts, kept for one day:

* Code quality json report in code climate format.

### Test jobs

The Python template features four alternative test jobs:

* `py-unittest` that performs tests based on [unittest](https://docs.python.org/3/library/unittest.html) Python lib,
* or `py-pytest` that performs tests based on [pytest](https://docs.pytest.org/en/latest/) Python lib,
* or `py-nosetest` that performs tests based on [nose](https://nose.readthedocs.io/en/latest/) Python lib,
* or `py-compile` that performs byte code generation to check syntax if not tests are available.

#### `py-unittest` job

This job is **disabled by default** and performs tests based on [unittest](https://docs.python.org/3/library/unittest.html) Python lib.
It is activated by setting `$UNITTEST_ENABLED` to `true`.

In order to produce JUnit test reports, the tests are executed with the [xmlrunner](https://github.com/xmlrunner/unittest-xml-reporting) module.

It is bound to the `build` stage, and uses the following variables:

| Name                     | description                                                          | default value           |
| ------------------------ | -------------------------------------------------------------------- | ----------------------- |
| `UNITTEST_ARGS`          | Additional xmlrunner/unittest CLI options                            | _none_                  |

This job produces the following artifacts, kept for one day:

* JUnit test report (using the [xmlrunner](https://github.com/xmlrunner/unittest-xml-reporting) module)
* code coverage report (cobertura xml format).

:warning: create a `.coveragerc` file at the root of your Python project to control the coverage settings.

Example:

```conf
[run]
# enables branch coverage
branch = True
# list of directories/packages to cover
source = 
    module_1
    module_2
```

#### `py-pytest` job

This job is **disabled by default** and performs tests based on [pytest](https://docs.pytest.org/en/latest/) Python lib.
It is activated by setting `$PYTEST_ENABLED` to `true`.

It is bound to the `build` stage, and uses the following variables:

| Name                     | description                                                                                                                                   | default value           |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
| `PYTEST_ARGS`            | Additional [pytest](https://docs.pytest.org/en/stable/usage.html) or [pytest-cov](https://github.com/pytest-dev/pytest-cov#usage) CLI options | _none_                  |

This job produces the following artifacts, kept for one day:

* JUnit test report (with the [`--junit-xml`](http://doc.pytest.org/en/latest/usage.html#creating-junitxml-format-files) argument)
* code coverage report (cobertura xml format).

:warning: create a `.coveragerc` file at the root of your Python project to control the coverage settings.

Example:

```conf
[run]
# enables branch coverage
branch = True
# list of directories/packages to cover
source = 
    module_1
    module_2
```

#### `py-nosetest` job

This job is **disabled by default** and performs tests based on [nose](https://nose.readthedocs.io/en/latest/) Python lib.
It is activated by setting `$NOSETESTS_ENABLED` to `true`.

It is bound to the `build` stage, and uses the following variables:

| Name                     | description                                                                             | default value           |
| ------------------------ | --------------------------------------------------------------------------------------- | ----------------------- |
| `NOSETESTS_ARGS`         | Additional [nose CLI options](https://nose.readthedocs.io/en/latest/usage.html#options) | _none_                  |

By default coverage will be run on all the directory. You can restrict it to your packages by setting NOSE_COVER_PACKAGE variable.
More [info](https://nose.readthedocs.io/en/latest/plugins/cover.html)

This job produces the following artifacts, kept for one day:

* JUnit test report (with the [`--with-xunit`](https://nose.readthedocs.io/en/latest/plugins/xunit.html) argument)
* code coverage report (cobertura xml format + html report).

:warning: create a `.coveragerc` file at the root of your Python project or use [nose CLI options](https://nose.readthedocs.io/en/latest/plugins/cover.html#options) to control the coverage settings.

#### `py-compile` job

This job is a fallback if no unit test has been setup (`$UNITTEST_ENABLED` and `$PYTEST_ENABLED` and `$NOSETEST_ENABLED`
are not set), and performs a [`compileall`](https://docs.python.org/3/library/compileall.html).

It is bound to the `build` stage, and uses the following variables:

| Name                  | description                                                                   | default value |
| --------------------- | ----------------------------------------------------------------------------- | ------------- |
| `PYTHON_COMPILE_ARGS` | [`compileall` CLI options](https://docs.python.org/3/library/compileall.html) | `*`           |

### SonarQube analysis

If you're using the SonarQube template to analyse your Python code, here is a sample `sonar-project.properties` file:

```properties
# see: https://docs.sonarqube.org/latest/analysis/languages/python/
# set your source directory(ies) here (relative to the sonar-project.properties file)
sonar.sources=.
# exclude unwanted directories and files from being analysed
sonar.exclusions=**/test_*.py

# set your tests directory(ies) here (relative to the sonar-project.properties file)
sonar.tests=.
sonar.test.inclusions=**/test_*.py

# tests report: generic format
sonar.python.xunit.reportPath=reports/unittest/TEST-*.xml
# coverage report: XUnit format
sonar.python.coverage.reportPaths=reports/coverage.xml
```

More info:

* [Python language support](https://docs.sonarqube.org/latest/analysis/languages/python/)
* [test coverage & execution parameters](https://docs.sonarqube.org/latest/analysis/coverage/)
* [third-party issues](https://docs.sonarqube.org/latest/analysis/external-issues/)

### `py-bandit` job (SAST)

This job is **disabled by default** and performs a [Bandit](https://pypi.org/project/bandit/) analysis.

It is bound to the `test` stage, and uses the following variables:

| Name             | description                                                            | default value     |
| ---------------- | ---------------------------------------------------------------------- | ----------------- |
| `BANDIT_ENABLED` | Set to `true` to enable Bandit analysis                                     | _none_ (disabled) |
| `BANDIT_ARGS`    | Additional [Bandit CLI options](https://github.com/PyCQA/bandit#usage) | `--recursive .`   |

This job outputs a **textual report** in the console, and in case of failure also exports a JSON report in the `reports/`
directory _(relative to project root dir)_.

### `py-safety` job (dependency check)

This job is **disabled by default** and performs a dependency check analysis using [Safety](https://pypi.org/project/safety/).

It is bound to the `test` stage, and uses the following variables:

| Name             | description                                                             | default value     |
| ---------------- | ----------------------------------------------------------------------- | ----------------- |
| `SAFETY_ENABLED` | Set to `true` to enable Safety job                                           | _none_ (disabled) |
| `SAFETY_ARGS`    | Additional [Safety CLI options](https://github.com/pyupio/safety#usage) | `--full-report`   |

This job outputs a **textual report** in the console, and in case of failure also exports a JSON report in the `reports/`
directory _(relative to project root dir)_.

### `py-trivy` job (dependency check)

This job is **disabled by default** and performs a dependency check analysis using [Trivy](https://github.com/aquasecurity/trivy/).

It is bound to the `test` stage, and uses the following variables:

| Name             | description                                                             | default value     |
| ---------------- | ----------------------------------------------------------------------- | ----------------- |
| `PYTHON_TRIVY_ENABLED` | Set to `true` to enable Trivy job                                           | _none_ (disabled) |
| `PYTHON_TRIVY_ARGS`    | Additional [Trivy CLI options](https://aquasecurity.github.io/trivy/v0.21.1/getting-started/cli/fs/) | `--vuln-type library`   |

This job outputs a **textual report** in the console, and in case of failure also exports a JSON report in the `reports/`
directory _(relative to project root dir)_.

### Package jobs

#### `py-package` job

This job is **disabled by default** and performs a packaging of your Python code.

It is bound to the `package-build` stage, applies only on git tags and uses the following variables:

| Name            | description                                          | default value |
| --------------- | ---------------------------------------------------- | ------------- |
| `PYTHON_FORCE_PACKAGE` | Set to `true` to force the packaging even if not on tag related event | _none_ (disabled) |

### Publish jobs

#### `py-release` job

This job is **disabled by default** and performs an automatic tagging of your Python code.

* [Bumpversion](https://github.com/peritus/bumpversion) Python library is used for version management.
* Looks for an existing `.bumpversion.cfg` at the project root. If found, it will be the configuration used by bumpversion. If not, the `$RELEASE_VERSION_PART` variable and `setup.py` will be used instead.
* Creating a Git tag involves an authenticated and authorized Git user.

**Don't use your personal password !!!
Use an [access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) with write_repository rights.
If you have a generic account, add it to the project and generate access token from this account.**

It is bound to the `publish` stage, applies only on master branch and uses the following variables:

| Name                   | description                                                             | default value     |
| ---------------------- | ----------------------------------------------------------------------- | ----------------- |
| `RELEASE_VERSION_PART` | The part of the version to increase (one of: `major`, `minor`, `patch`) | `minor`           |
| `RELEASE_USERNAME`     | Username credential for git push                                        | _none_ (disabled) |
| `RELEASE_ACCESS_TOKEN` | Password credential for git push                                        | _none_            |

#### `py-publish` job

This job is **disabled by default** and performs a publication of your Python code.

It is bound to the `publish` stage, applies only on git tags and uses the following variables:

| Name                   | description                                              | default value     |
| ---------------------- | -------------------------------------------------------- | ----------------- |
| `PYTHON_PUBLISH_ENABLED`| Set to `true` to enable the publish job                 | _none_ (disabled) |
| `TWINE_REPOSITORY_URL` | Where to publish your Python project                     | GitLab Project's Pypi Packages registry |
| `TWINE_USERNAME`       | Username credential to publish to \$TWINE_REPOSITORY_URL | `gitlab-ci-token` |
| `TWINE_PASSWORD`       | Password credential to publish to \$TWINE_REPOSITORY_URL | `$CI_JOB_TOKEN` |

More info:

* [Python Packaging User Guide](https://packaging.python.org/)
* [PyPI packages in the Package Registry](https://docs.gitlab.com/ee/user/packages/pypi_repository/)

If you want to automatically create tag and publish your Python package, please have a look [here](#release-python)

## GitLab compatibility

:information_source: This template is actually tested and validated on GitLab Community Edition instance version 13.12.11