Skip to content
Snippets Groups Projects
Commit dda82d21 authored by Pierre Smeyers's avatar Pierre Smeyers
Browse files

Merge branch 'feat/poetry-improvments' into 'master' 

Python improvements

See merge request to-be-continuous/python!32
parents 8a3c4f34 ff8b9856
Branches
Tags
No related merge requests found
Expand all Hide whitespace changes
Inline Side-by-side
README.md
+ 89
− 64
View file @ dda82d21
......@@ -2,7 +2,7 @@
This project implements a generic GitLab CI template for [Python](https://www.python.org/).
It provides several features, usable in different modes (by configuration) following those [recommendations](to-be-continuous.gitlab.io/doc/usage/)
It provides several features, usable in different modes (by configuration).
## Usage
......@@ -21,34 +21,40 @@ The Python template uses some global configuration used throughout all jobs.
| Name | description | default value |
| -------------------- | ------------------------------------------------------------------------------------- | ------------------ |
| `PYTHON_IMAGE` | The Docker image used to run Python <br/>:warning: **set the version required by your project** | `python:3` |
| `PIP_INDEX_URL` | Python repository url | _none_ |
| `PYTHON_IMAGE` | The Docker image used to run Python <br/>:warning: **set the version required by your project** | `python:3` |
| `PYTHON_PROJECT_DIR` | Python project root directory | `.` |
| `REQUIREMENTS_FILE` | Path to requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ | `requirements.txt` |
| `PIP_OPTS` | pip extra [options](https://pip.pypa.io/en/stable/reference/pip/#general-options) | _none_ |
| `PYTHON_BUILD_SYSTEM`| Python build-system to use to install dependencies, build and package the project (see below) | _none_ (auto-detect) |
| `PIP_INDEX_URL` | Python repository url | _none_ |
| `PIP_OPTS` | pip [extra options](https://pip.pypa.io/en/stable/reference/pip/#general-options) | _none_ |
| `PYTHON_EXTRA_DEPS` | Python extra sets of dependencies to install<br/>For [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies) or [Poetry](https://python-poetry.org/docs/pyproject/#extras) only | _none_ |
| `PYTHON_REQS_FILE` | Main requirements file _(relative to `$PYTHON_PROJECT_DIR`)_<br/>For [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) build-system only | `requirements.txt` |
| `PYTHON_EXTRA_REQS_FILES` | Extra dev requirements file(s) to install _(relative to `$PYTHON_PROJECT_DIR`)_ | `requirements-dev.txt` |
The cache policy also declares the `.cache/pip` directory as cached (not to download Python dependencies over and over again).
The cache policy also makes the necessary to manage pip cache (not to download Python dependencies over and over again).
Default configuration follows [this Python project structure](https://docs.python-guide.org/writing/structure/)
## Multi build-system support
### Poetry support
The Python template supports the most popular dependency management & build systems.
The Python template supports [Poetry](https://python-poetry.org/) as packaging and dependency management tool.
By default it tries to auto-detect the build system used by the project (based on the presence of `pyproject.toml`
and/or `setup.py` and/or `requirements.txt`), but the build system might also be set explicitly using the
`$PYTHON_BUILD_SYSTEM` variable:
If a `pyproject.toml` file is detected at the root of your Python project, requirements will automatically be generated from Poetry.
Poetry support can be explicitly disabled by setting `PYTHON_POETRY_DISABLED` to `true`.
| Value | Build System (scope) |
| ---------------- | ---------------------------------------------------------- |
| _none_ (default) or `auto` | The template tries to **auto-detect** the actual build system |
| `setuptools` | [Setuptools](https://setuptools.pypa.io/) (dependencies, build & packaging) |
| `poetry` | [Poetry](https://python-poetry.org/) (dependencies, build, test & packaging) |
| `pipenv` | [Pipenv](https://pipenv.pypa.io/) (dependencies only) |
| `reqfile` | [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) (dependencies only) |
:warning: If no `poetry.lock` file is found, the template will emit a (non-blocking) warning message, to enforce [Poetry recommendation](https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control):
## Jobs
> You should commit the `poetry.lock` file to your project repo so that all people working on the project are locked to the same versions of dependencies.
### `py-package` job
Poetry support uses the following variables:
This job allows building your Python project [distribution packages](https://packaging.python.org/en/latest/glossary/#term-Distribution-Package).
| Name | description | default value |
| ------------------------ | ---------------------------------------------------------- | ----------------- |
| `PYTHON_POETRY_EXTRAS` | Poetry [extra sets of dependencies](https://python-poetry.org/docs/pyproject/#extras) to include, space separated | _none_ |
## Jobs
It is bound to the `build` stage, it is **disabled by default** and can be enabled by setting `$PYTHON_PACKAGE_ENABLED` to `true`.
### Lint jobs
......@@ -88,7 +94,6 @@ It is bound to the `build` stage, and uses the following variables:
| Name | description | default value |
| ------------------------ | -------------------------------------------------------------------- | ----------------------- |
| `TEST_REQUIREMENTS_FILE` | Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ | `test-requirements.txt` |
| `UNITTEST_ARGS` | Additional xmlrunner/unittest CLI options | _none_ |
This job produces the following artifacts, kept for one day:
......@@ -119,7 +124,6 @@ It is bound to the `build` stage, and uses the following variables:
| Name | description | default value |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
| `TEST_REQUIREMENTS_FILE` | Path to test requirements file _(relative `$PYTHON_PROJECT_DIR`)_ | `test-requirements.txt` |
| `PYTEST_ARGS` | Additional [pytest](https://docs.pytest.org/en/stable/usage.html) or [pytest-cov](https://github.com/pytest-dev/pytest-cov#usage) CLI options | _none_ |
This job produces the following artifacts, kept for one day:
......@@ -150,7 +154,6 @@ It is bound to the `build` stage, and uses the following variables:
| Name | description | default value |
| ------------------------ | --------------------------------------------------------------------------------------- | ----------------------- |
| `TEST_REQUIREMENTS_FILE` | Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ | `test-requirements.txt` |
| `NOSETESTS_ARGS` | Additional [nose CLI options](https://nose.readthedocs.io/en/latest/usage.html#options) | _none_ |
By default coverage will be run on all the directory. You can restrict it to your packages by setting NOSE_COVER_PACKAGE variable.
......@@ -209,7 +212,7 @@ It is bound to the `test` stage, and uses the following variables:
| Name | description | default value |
| ---------------- | ---------------------------------------------------------------------- | ----------------- |
| `BANDIT_ENABLED` | Set to `true` to enable Bandit analysis | _none_ (disabled) |
| `BANDIT_ENABLED` | Set to `true` to enable Bandit analysis | _none_ (disabled) |
| `BANDIT_ARGS` | Additional [Bandit CLI options](https://github.com/PyCQA/bandit#usage) | `--recursive .` |
This job outputs a **textual report** in the console, and in case of failure also exports a JSON report in the `reports/`
......@@ -223,7 +226,7 @@ It is bound to the `test` stage, and uses the following variables:
| Name | description | default value |
| ---------------- | ----------------------------------------------------------------------- | ----------------- |
| `SAFETY_ENABLED` | Set to `true` to enable Safety job | _none_ (disabled) |
| `SAFETY_ENABLED` | Set to `true` to enable Safety job | _none_ (disabled) |
| `SAFETY_ARGS` | Additional [Safety CLI options](https://github.com/pyupio/safety#usage) | `--full-report` |
This job outputs a **textual report** in the console, and in case of failure also exports a JSON report in the `reports/`
......@@ -237,70 +240,92 @@ It is bound to the `test` stage, and uses the following variables:
| Name | description | default value |
| ---------------- | ----------------------------------------------------------------------- | ----------------- |
| `PYTHON_TRIVY_ENABLED` | Set to `true` to enable Trivy job | _none_ (disabled) |
| `PYTHON_TRIVY_ENABLED` | Set to `true` to enable Trivy job | _none_ (disabled) |
| `PYTHON_TRIVY_ARGS` | Additional [Trivy CLI options](https://aquasecurity.github.io/trivy/v0.21.1/getting-started/cli/fs/) | `--vuln-type library` |
This job outputs a **textual report** in the console, and in case of failure also exports a JSON report in the `reports/`
directory _(relative to project root dir)_.
### Package jobs
### `py-release` job
#### `py-package` job
This job is **disabled by default** and allows to perform a complete release of your Python code:
This job is **disabled by default** and performs a packaging of your Python code.
1. increase the Python project version,
2. Git commit changes and create a Git tag with the new version number,
3. build the [Python packages](https://packaging.python.org/),
4. publish the built packages to a PyPI compatible repository ([GitLab packages](https://docs.gitlab.com/ee/user/packages/pypi_repository/) by default).
It is bound to the `package-build` stage, applies only on git tags and uses the following variables:
The Python template supports two packaging systems:
| Name | description | default value |
| --------------- | ---------------------------------------------------- | ------------- |
| `PYTHON_FORCE_PACKAGE` | Set to `true` to force the packaging even if not on tag related event | _none_ (disabled) |
* [Poetry](https://python-poetry.org/): uses Poetry-specific [version](https://python-poetry.org/docs/cli/#version), [build](https://python-poetry.org/docs/cli/#build) and [publish](https://python-poetry.org/docs/cli/#publish) commands.
* [Setuptools](https://setuptools.pypa.io/): uses [Bumpversion](https://github.com/peritus/bumpversion) as version management, [build](https://pypa-build.readthedocs.io/) as package builder and [Twine](https://twine.readthedocs.io/) to publish.
### Publish jobs
The release job is bound to the `publish` stage, appears only on production and integration branches and uses the following variables:
#### `py-release` job
| Name | description | default value |
| ----------------------- | ----------------------------------------------------------------------- | ----------------- |
| `PYTHON_RELEASE_ENABLED`| Set to `true` to enable the release job | _none_ (disabled) |
| `PYTHON_RELEASE_NEXT` | The part of the version to increase (one of: `major`, `minor`, `patch`) | `minor` |
| `PYTHON_SEMREL_RELEASE_DISABLED`| Set to `true` to disable [semantic-release integration](#semantic-release-integration) | _none_ (disabled) |
| `GIT_USERNAME` | Git username for Git push operations (see below) | _none_ |
| :lock: `GIT_PASSWORD` | Git password for Git push operations (see below) | _none_ |
| :lock: `GIT_PRIVATE_KEY`| SSH key for Git push operations (see below) | _none_ |
| `PYTHON_REPOSITORY_URL`| Target PyPI repository to publish packages | _[GitLab project's PyPI packages repository](https://docs.gitlab.com/ee/user/packages/pypi_repository/)_ |
| `PYTHON_REPOSITORY_USERNAME`| Target PyPI repository username credential | `gitlab-ci-token` |
| :lock: `PYTHON_REPOSITORY_PASSWORD`| Target PyPI repository password credential | `$CI_JOB_TOKEN` |
This job is **disabled by default** and performs an automatic tagging of your Python code.
#### Setuptools tip
* [Bumpversion](https://github.com/peritus/bumpversion) Python library is used for version management.
* Looks for an existing `.bumpversion.cfg` at the project root. If found, it will be the configuration used by bumpversion. If not, the `$RELEASE_VERSION_PART` variable and `setup.py` will be used instead.
* Creating a Git tag involves an authenticated and authorized Git user.
If you're using a `setup.cfg` declarative file for your project Setuptools configuration, then you will have to write a
`.bumpversion.cfg` file to workaround a bug that prevents Bumpversion from updating the project version in your `setup.cfg` file.
**Don't use your personal password !!!
Use an [access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) with write_repository rights.
If you have a generic account, add it to the project and generate access token from this account.**
Example of `.bumpversion.cfg` file:
It is bound to the `publish` stage, applies only on master branch and uses the following variables:
```ini
[bumpversion]
# same version as in your setup.cfg
current_version = 0.5.0
| Name | description | default value |
| ---------------------- | ----------------------------------------------------------------------- | ----------------- |
| `RELEASE_VERSION_PART` | The part of the version to increase (one of: `major`, `minor`, `patch`) | `minor` |
| `RELEASE_USERNAME` | Username credential for git push | _none_ (disabled) |
| `RELEASE_ACCESS_TOKEN` | Password credential for git push | _none_ |
[bumpversion:file:setup.cfg]
# any additional config here
# see: https://github.com/peritus/bumpversion#file-specific-configuration
```
#### `py-publish` job
#### `semantic-release` integration
This job is **disabled by default** and performs a publication of your Python code.
If you activate the [`semantic-release-info` job from the `semantic-release` template](https://gitlab.com/to-be-continuous/semantic-release/#semantic-release-info-job), the `py-release` job will rely on the generated next version info.
Thus, a release will be performed only if a next semantic release is present.
It is bound to the `publish` stage, applies only on git tags and uses the following variables:
You should disable the `semantic-release` job (as it's the `py-release` job that will perform the release and so we only need the `semantic-release-info` job) by setting `SEMREL_RELEASE_DISABLED` to `true`.
| Name | description | default value |
| ---------------------- | -------------------------------------------------------- | ----------------- |
| `PYTHON_PUBLISH_ENABLED`| Set to `true` to enable the publish job | _none_ (disabled) |
| `TWINE_REPOSITORY_URL` | Where to publish your Python project | GitLab Project's Pypi Packages registry |
| `TWINE_USERNAME` | Username credential to publish to \$TWINE_REPOSITORY_URL | `gitlab-ci-token` |
| `TWINE_PASSWORD` | Password credential to publish to \$TWINE_REPOSITORY_URL | `$CI_JOB_TOKEN` |
Finally, the semantic-release integration can be disabled with the `PYTHON_SEMREL_RELEASE_DISABLED` variable.
More info:
####
* [Python Packaging User Guide](https://packaging.python.org/)
* [PyPI packages in the Package Registry](https://docs.gitlab.com/ee/user/packages/pypi_repository/)
#### Git authentication
If you want to automatically create tag and publish your Python package, please have a look [here](#release-python)
A Python release involves some Git push operations.
You can either use a SSH key or user/password credentials.
##### Using a SSH key
We recommend you to use a [project deploy key](https://docs.gitlab.com/ee/user/project/deploy_keys/#project-deploy-keys) with write access to your project.
The key should not have a passphrase (see [how to generate a new SSH key pair](https://docs.gitlab.com/ce/ssh/README.html#generating-a-new-ssh-key-pair)).
Specify :lock: `$GIT_PRIVATE_KEY` as secret project variable with the private part of the deploy key.
```PEM
-----BEGIN OPENSSH PRIVATE KEY-----
blablabla
-----END OPENSSH PRIVATE KEY-----
```
#### `py-docs` job
The template handles both classic variable and file variable.
This job is no longer supported in this version of the template. It might come back later on with a more generic & configurable implementation.
##### Using user/password credentials
## GitLab compatibility
Simply specify :lock: `$GIT_USERNAME` and :lock: `$GIT_PASSWORD` as secret project variables.
:information_source: This template is actually tested and validated on GitLab Community Edition instance version 13.12.11
Note that the password should be an access token (preferably a [project](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) or [group](https://docs.gitlab.com/ee/user/group/settings/group_access_tokens.html) access token) with `read_repository` and `write_repository` scopes.
kicker.json
+ 53
− 74
View file @ dda82d21
......@@ -15,11 +15,25 @@
"default": "."
},
{
"name": "REQUIREMENTS_FILE",
"description": "Full path to `requirements.txt` file _(relative to `$PYTHON_PROJECT_DIR`)_",
"name": "PYTHON_BUILD_SYSTEM",
"description": "Python build-system to use to install dependencies, build and package the project",
"type": "enum",
"values": ["auto", "setuptools", "poetry", "pipenv", "reqfile"],
"default": "auto",
"advanced": true
},
{
"name": "PYTHON_REQS_FILE",
"description": "Main requirements file _(relative to `$PYTHON_PROJECT_DIR`)_\n\nFor [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) build-system only",
"default": "requirements.txt",
"advanced": true
},
{
"name": "PYTHON_EXTRA_REQS_FILES",
"description": "Extra dev requirements file(s) to install _(relative to `$PYTHON_PROJECT_DIR`)_\n\nFor [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) build-system only",
"default": "requirements-dev.txt",
"advanced": true
},
{
"name": "PYTHON_COMPILE_ARGS",
"description": "[`compileall` CLI options](https://docs.python.org/3/library/compileall.html)",
......@@ -32,15 +46,8 @@
"advanced": true
},
{
"name": "PYTHON_POETRY_DISABLED",
"description": "Disable poetry support",
"type": "boolean",
"advanced": true
},
{
"name": "PYTHON_POETRY_EXTRAS",
"description": "Poetry [extra sets of dependencies](https://python-poetry.org/docs/pyproject/#extras) to include, space separated",
"advanced": true
"name": "PYTHON_EXTRA_DEPS",
"description": "Extra sets of dependencies to install\n\nFor [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies) or [Poetry](https://python-poetry.org/docs/pyproject/#extras) only"
}
],
"features": [
......@@ -68,12 +75,6 @@
"description": "Unit tests based on [unittest](https://docs.python.org/3/library/unittest.html) framework",
"enable_with": "UNITTEST_ENABLED",
"variables": [
{
"name": "TEST_REQUIREMENTS_FILE",
"description": "Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_",
"default": "test-requirements.txt",
"advanced": true
},
{
"name": "UNITTEST_ARGS",
"description": "Additional xmlrunner/unittest CLI options",
......@@ -87,12 +88,6 @@
"description": "Unit tests based on [pytest](https://docs.pytest.org/) framework",
"enable_with": "PYTEST_ENABLED",
"variables": [
{
"name": "TEST_REQUIREMENTS_FILE",
"description": "Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_",
"default": "test-requirements.txt",
"advanced": true
},
{
"name": "PYTEST_ARGS",
"description": "Additional [pytest](https://docs.pytest.org/en/stable/usage.html) or [pytest-cov](https://github.com/pytest-dev/pytest-cov#usage) CLI options",
......@@ -106,12 +101,6 @@
"description": "Unit tests based on [nose](https://nose.readthedocs.io/) framework",
"enable_with": "NOSETESTS_ENABLED",
"variables": [
{
"name": "TEST_REQUIREMENTS_FILE",
"description": "Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_",
"default": "test-requirements.txt",
"advanced": true
},
{
"name": "NOSETESTS_ARGS",
"description": "Additional [nose CLI options](https://nose.readthedocs.io/en/latest/usage.html#options)",
......@@ -161,51 +150,14 @@
}
]
},
{
"id": "package",
"name": "package",
"description": "Packaging of your Python code",
"variables": [
{
"name": "PYTHON_FORCE_PACKAGE",
"description": "Force the packaging even if not on tag related event",
"type": "boolean"
}
]
},
{
"id": "publish",
"name": "Publish",
"description": "Publish your code to a [Twine](https://pypi.org/project/twine/) repository",
"enable_with": "PYTHON_PUBLISH_ENABLED",
"variables": [
{
"name": "TWINE_REPOSITORY_URL",
"type": "url",
"description": "Twine repository url to publish you python project",
"default": "${CI_SERVER_URL}/api/v4/projects/${CI_PROJECT_ID}/packages/pypi"
},
{
"name": "TWINE_USERNAME",
"description": "Twine repository username credential",
"secret": true,
"default": "gitlab-ci-token"
},
{
"name": "TWINE_PASSWORD",
"description": "Twine repository password credential",
"secret": true,
"default": "$CI_JOB_TOKEN"
}
]
},
{
"id": "release",
"name": "Release",
"description": "Manually trigger a release of your code (uses [bumpversion](https://pypi.org/project/bumpversion/))",
"enable_with": "PYTHON_RELEASE_ENABLED",
"variables": [
{
"name": "RELEASE_VERSION_PART",
"name": "PYTHON_RELEASE_NEXT",
"type": "enum",
"values": [
"",
......@@ -218,16 +170,43 @@
"advanced": true
},
{
"name": "RELEASE_USERNAME",
"description": "Username credential for Git push",
"name": "PYTHON_SEMREL_RELEASE_DISABLED",
"description": "Disable semantic-release integration",
"type": "boolean",
"advanced": true
},
{
"name": "GIT_USERNAME",
"description": "Git username for Git push operations",
"secret": true
},
{
"name": "GIT_PASSWORD",
"description": "Git password for Git push operations",
"secret": true
},
{
"name": "GIT_PRIVATE_KEY",
"description": "SSH key for Git push operations",
"secret": true
},
{
"name": "PYTHON_REPOSITORY_URL",
"type": "url",
"description": "Target PyPI repository to publish packages.\n\n_defaults to [GitLab project's packages repository](https://docs.gitlab.com/ee/user/packages/pypi_repository/)_",
"default": "${CI_SERVER_URL}/api/v4/projects/${CI_PROJECT_ID}/packages/pypi"
},
{
"name": "PYTHON_REPOSITORY_USERNAME",
"description": "Target PyPI repository username credential",
"secret": true,
"mandatory": true
"default": "gitlab-ci-token"
},
{
"name": "RELEASE_ACCESS_TOKEN",
"description": "Password credential for Git push",
"name": "PYTHON_REPOSITORY_PASSWORD",
"description": "Target PyPI repository password credential",
"secret": true,
"mandatory": true
"default": "$CI_JOB_TOKEN"
}
]
}
......
templates/gitlab-ci-python.yml
+ 320
− 138
View file @ dda82d21
This diff is collapsed.
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment