From 130e2102af56dc8719ba5c87a7e31902fb9fe228 Mon Sep 17 00:00:00 2001 From: Pierre Smeyers <pierre.smeyers@gmail.com> Date: Thu, 3 Feb 2022 15:47:03 +0100 Subject: [PATCH] feat: add multi build-system support (Poetry, Setuptools or requirements file) BREAKING CHANGE: removed $PYTHON_POETRY_DISABLED with $PYTHON_BUILD_SYSTEM (see doc) --- README.md | 47 ++++--- kicker.json | 44 +++---- templates/gitlab-ci-python.yml | 233 ++++++++++++++++++++++----------- 3 files changed, 191 insertions(+), 133 deletions(-) diff --git a/README.md b/README.md index 852e98e..9ac3e06 100644 --- a/README.md +++ b/README.md @@ -21,32 +21,36 @@ The Python template uses some global configuration used throughout all jobs. | Name | description | default value | | -------------------- | ------------------------------------------------------------------------------------- | ------------------ | -| `PYTHON_IMAGE` | The Docker image used to run Python <br/>:warning: **set the version required by your project** | `python:3` | -| `PIP_INDEX_URL` | Python repository url | _none_ | +| `PYTHON_IMAGE` | The Docker image used to run Python <br/>:warning: **set the version required by your project** | `python:3` | | `PYTHON_PROJECT_DIR` | Python project root directory | `.` | -| `REQUIREMENTS_FILE` | Path to requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ | `requirements.txt` | -| `PIP_OPTS` | pip extra [options](https://pip.pypa.io/en/stable/reference/pip/#general-options) | _none_ | - -The cache policy also declares the `.cache/pip` directory as cached (not to download Python dependencies over and over again). - -Default configuration follows [this Python project structure](https://docs.python-guide.org/writing/structure/) +| `PYTHON_BUILD_SYSTEM`| Python build-system to use to install dependencies, build and package the project (see below) | _none_ (auto-detect) | +| `PIP_INDEX_URL` | Python repository url | _none_ | +| `PIP_OPTS` | pip [extra options](https://pip.pypa.io/en/stable/reference/pip/#general-options) | _none_ | +| `PYTHON_EXTRA_DEPS` | Python extra sets of dependencies to install<br/>For [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies) or [Poetry](https://python-poetry.org/docs/pyproject/#extras) only | _none_ | +| `REQUIREMENTS_FILE` | Name of requirements file _(relative to `$PYTHON_PROJECT_DIR`)_<br/>For [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) build-system only | `requirements.txt` | +| `TEST_REQUIREMENTS_FILE` | Name of dev/test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ | `test-requirements.txt` | -### Poetry support +The cache policy also makes the necessary to manage pip cache (not to download Python dependencies over and over again). -The Python template supports [Poetry](https://python-poetry.org/) as packaging and dependency management tool. +## Multi build-system support -If a `pyproject.toml` file is detected at the root of your Python project, requirements will automatically be generated from Poetry. -Poetry support can be explicitly disabled by setting `PYTHON_POETRY_DISABLED` to `true`. +The Python template supports 3 popular dependency management & build systems: -:warning: If no `poetry.lock` file is found, the template will emit a (non-blocking) warning message, to enforce [Poetry recommendation](https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control): +* [Setuptools](https://setuptools.pypa.io/), +* [Poetry](https://python-poetry.org/), +* [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) (dependency management only). -> You should commit the `poetry.lock` file to your project repo so that all people working on the project are locked to the same versions of dependencies. +By default the template tries to auto-detect the build system used by the project (based on presence of `pyproject.toml` +and/or `setup.py` and/or `requirements.txt`), but the build system might also be explicitly set using the `$PYTHON_BUILD_SYSTEM` variable. -Poetry support uses the following variables: +Supported values of `$PYTHON_BUILD_SYSTEM`: -| Name | description | default value | -| ------------------------ | ---------------------------------------------------------- | ----------------- | -| `PYTHON_POETRY_EXTRAS` | Poetry [extra sets of dependencies](https://python-poetry.org/docs/pyproject/#extras) to include, space separated | _none_ | +| Value | Description | +| ---------------- | ---------------------------------------------------------- | +| _none_ (default) | The template tries to auto-detect the actual build system, based of the presence of some key files | +| `setuptools` | [Setuptools](https://setuptools.pypa.io/) will be used to install dependencies, build and package the project | +| `poetry` | [Poetry](https://python-poetry.org/) will be used to install dependencies, build, test and package the project | +| `reqfile` | [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) will be used to install dependencies | ## Jobs @@ -88,7 +92,6 @@ It is bound to the `build` stage, and uses the following variables: | Name | description | default value | | ------------------------ | -------------------------------------------------------------------- | ----------------------- | -| `TEST_REQUIREMENTS_FILE` | Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ | `test-requirements.txt` | | `UNITTEST_ARGS` | Additional xmlrunner/unittest CLI options | _none_ | This job produces the following artifacts, kept for one day: @@ -119,7 +122,6 @@ It is bound to the `build` stage, and uses the following variables: | Name | description | default value | | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | -| `TEST_REQUIREMENTS_FILE` | Path to test requirements file _(relative `$PYTHON_PROJECT_DIR`)_ | `test-requirements.txt` | | `PYTEST_ARGS` | Additional [pytest](https://docs.pytest.org/en/stable/usage.html) or [pytest-cov](https://github.com/pytest-dev/pytest-cov#usage) CLI options | _none_ | This job produces the following artifacts, kept for one day: @@ -150,7 +152,6 @@ It is bound to the `build` stage, and uses the following variables: | Name | description | default value | | ------------------------ | --------------------------------------------------------------------------------------- | ----------------------- | -| `TEST_REQUIREMENTS_FILE` | Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_ | `test-requirements.txt` | | `NOSETESTS_ARGS` | Additional [nose CLI options](https://nose.readthedocs.io/en/latest/usage.html#options) | _none_ | By default coverage will be run on all the directory. You can restrict it to your packages by setting NOSE_COVER_PACKAGE variable. @@ -297,10 +298,6 @@ More info: If you want to automatically create tag and publish your Python package, please have a look [here](#release-python) -#### `py-docs` job - -This job is no longer supported in this version of the template. It might come back later on with a more generic & configurable implementation. - ## GitLab compatibility :information_source: This template is actually tested and validated on GitLab Community Edition instance version 13.12.11 diff --git a/kicker.json b/kicker.json index 5983e8b..702f023 100644 --- a/kicker.json +++ b/kicker.json @@ -14,12 +14,25 @@ "description": "Python project root directory", "default": "." }, + { + "name": "PYTHON_BUILD_SYSTEM", + "description": "Python build-system to use to install dependencies, build and package the project", + "type": "enum", + "values": ["", "setuptools", "poetry", "reqfile"], + "advanced": true + }, { "name": "REQUIREMENTS_FILE", - "description": "Full path to `requirements.txt` file _(relative to `$PYTHON_PROJECT_DIR`)_", + "description": "Name of requirements file _(relative to `$PYTHON_PROJECT_DIR`)_\n\nFor [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) build-system only", "default": "requirements.txt", "advanced": true }, + { + "name": "TEST_REQUIREMENTS_FILE", + "description": "Name of dev/test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_\n\nFor [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) build-system only", + "default": "test-requirements.txt", + "advanced": true + }, { "name": "PYTHON_COMPILE_ARGS", "description": "[`compileall` CLI options](https://docs.python.org/3/library/compileall.html)", @@ -32,15 +45,8 @@ "advanced": true }, { - "name": "PYTHON_POETRY_DISABLED", - "description": "Disable poetry support", - "type": "boolean", - "advanced": true - }, - { - "name": "PYTHON_POETRY_EXTRAS", - "description": "Poetry [extra sets of dependencies](https://python-poetry.org/docs/pyproject/#extras) to include, space separated", - "advanced": true + "name": "PYTHON_EXTRA_DEPS", + "description": "Extra sets of dependencies to install\n\nFor [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies) or [Poetry](https://python-poetry.org/docs/pyproject/#extras) only" } ], "features": [ @@ -68,12 +74,6 @@ "description": "Unit tests based on [unittest](https://docs.python.org/3/library/unittest.html) framework", "enable_with": "UNITTEST_ENABLED", "variables": [ - { - "name": "TEST_REQUIREMENTS_FILE", - "description": "Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_", - "default": "test-requirements.txt", - "advanced": true - }, { "name": "UNITTEST_ARGS", "description": "Additional xmlrunner/unittest CLI options", @@ -87,12 +87,6 @@ "description": "Unit tests based on [pytest](https://docs.pytest.org/) framework", "enable_with": "PYTEST_ENABLED", "variables": [ - { - "name": "TEST_REQUIREMENTS_FILE", - "description": "Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_", - "default": "test-requirements.txt", - "advanced": true - }, { "name": "PYTEST_ARGS", "description": "Additional [pytest](https://docs.pytest.org/en/stable/usage.html) or [pytest-cov](https://github.com/pytest-dev/pytest-cov#usage) CLI options", @@ -106,12 +100,6 @@ "description": "Unit tests based on [nose](https://nose.readthedocs.io/) framework", "enable_with": "NOSETESTS_ENABLED", "variables": [ - { - "name": "TEST_REQUIREMENTS_FILE", - "description": "Path to test requirements file _(relative to `$PYTHON_PROJECT_DIR`)_", - "default": "test-requirements.txt", - "advanced": true - }, { "name": "NOSETESTS_ARGS", "description": "Additional [nose CLI options](https://nose.readthedocs.io/en/latest/usage.html#options)", diff --git a/templates/gitlab-ci-python.yml b/templates/gitlab-ci-python.yml index 14c07da..26ef1f2 100644 --- a/templates/gitlab-ci-python.yml +++ b/templates/gitlab-ci-python.yml @@ -44,13 +44,6 @@ variables: PYTHON_TRIVY_IMAGE: aquasec/trivy:latest PYTHON_TRIVY_ARGS: "--vuln-type library" - - # Docs - DOCS_REQUIREMENTS_FILE: docs-requirements.txt - DOCS_DIRECTORY: docs - DOCS_BUILD_DIR: public - DOCS_MAKE_ARGS: html BUILDDIR=${DOCS_BUILD_DIR} - RELEASE_VERSION_PART: "minor" # By default, publish on the Packages registry of the project @@ -219,46 +212,107 @@ variables: log_info "... done" } + function guess_build_system() { + if [[ "$PYTHON_BUILD_SYSTEM" ]] + then + case "$PYTHON_BUILD_SYSTEM" in + poetry) + log_info "--- Build system explictly declared: Poetry" + return + ;; + setuptools) + log_info "--- Build system explictly declared: Setuptools" + return + ;; + reqfile) + log_info "--- Build system explictly declared: requirements file" + return + ;; + *) + log_warn "--- Unknown declared build system: \\e[33;1m${PYTHON_BUILD_SYSTEM}\\e[0m: please read template doc" + ;; + esac + fi + + if [[ -f "pyproject.toml" ]] + then + # that might be PEP 517 if a build-backend is specified + # otherwise it might be only used as configuration file for development tools... + build_backend=$(sed -rn 's/^build-backend *= *"([^"]*)".*/\1/p' pyproject.toml) + if [[ "$build_backend" ]] + then + case "$build_backend" in + poetry.core.masonry.api) + log_info "--- Build system auto-detected: PEP 517 with Poetry backend" + export PYTHON_BUILD_SYSTEM="poetry" + return + ;; + setuptools.build_meta) + log_info "--- Build system auto-detected: PEP 517 with Setuptools backend" + export PYTHON_BUILD_SYSTEM="setuptools" + return + ;; + *) + log_error "--- Build system auto-detected: PEP 517 with unsupported backend \\e[33;1m${build_backend}\\e[0m: please read template doc" + exit 1 + ;; + esac + fi + fi + + if [[ -f "setup.py" ]] + then + log_info "--- Build system auto-detected: Setuptools (legacy)" + export PYTHON_BUILD_SYSTEM="setuptools" + elif [[ -f "${REQUIREMENTS_FILE}" ]] + then + log_info "--- Build system auto-detected: requirements file" + export PYTHON_BUILD_SYSTEM="reqfile" + else + log_error "--- Build system auto-detect failed: please read template doc" + exit 1 + fi + } + # install requirements - # arg1: 'build' (build only) or 'test' (build + test) function install_requirements() { - target=$1 - if [[ -f "pyproject.toml" ]] && [[ "${PYTHON_POETRY_DISABLED}" != "true" ]]; then + case "$PYTHON_BUILD_SYSTEM" in + poetry) if [[ ! -f "poetry.lock" ]]; then - log_warn "Poetry detected but \\e[33;1mpoetry.lock\\e[0m file not found: you shall commit it with your project files" - fi - pip install poetry - if [[ "$target" == "build" ]]; then - log_info "--- Poetry detected: install build only requirements" - poetry install --no-dev ${PYTHON_POETRY_EXTRAS:+--extras "$PYTHON_POETRY_EXTRAS"} - else - log_info "--- Poetry detected: install build and dev requirements" - poetry install ${PYTHON_POETRY_EXTRAS:+--extras "$PYTHON_POETRY_EXTRAS"} + log_warn "Using Poetry but \\e[33;1mpoetry.lock\\e[0m file not found: you shall commit it with your project files" fi - elif [[ -f "${REQUIREMENTS_FILE}" ]]; then - log_info "--- installing build requirements from \\e[33;1m${REQUIREMENTS_FILE}\\e[0m" # shellcheck disable=SC2086 - pip install ${PIP_OPTS} -r "${REQUIREMENTS_FILE}" - if [[ "$target" == "test" ]] && [[ -f "${TEST_REQUIREMENTS_FILE}" ]]; then - log_info "--- installing test requirements from \\e[33;1m${TEST_REQUIREMENTS_FILE}\\e[0m" + pip install ${PIP_OPTS} poetry + poetry install ${PYTHON_EXTRA_DEPS:+--extras "$PYTHON_EXTRA_DEPS"} + ;; + setuptools) + # shellcheck disable=SC2086 + pip install ${PIP_OPTS} setuptools + # shellcheck disable=SC2086 + pip install ${PIP_OPTS} ".${PYTHON_EXTRA_DEPS:+[$PYTHON_EXTRA_DEPS]}" + ;; + reqfile) + if [[ -f "${REQUIREMENTS_FILE}" ]]; then + log_info "--- installing build requirements from \\e[33;1m${REQUIREMENTS_FILE}\\e[0m" # shellcheck disable=SC2086 - pip install ${PIP_OPTS} -r "${TEST_REQUIREMENTS_FILE}" + pip install ${PIP_OPTS} -r "${REQUIREMENTS_FILE}" + if [[ -f "${TEST_REQUIREMENTS_FILE}" ]]; then + log_info "--- installing test requirements from \\e[33;1m${TEST_REQUIREMENTS_FILE}\\e[0m" + # shellcheck disable=SC2086 + pip install ${PIP_OPTS} -r "${TEST_REQUIREMENTS_FILE}" + fi + else + log_warn "--- requirements build system defined, but no ${REQUIREMENTS_FILE} file found" fi - elif [[ -f "${SETUP_PY_DIR}/setup.py" ]]; then - log_info "--- installing requirements from \\e[33;1m${SETUP_PY_DIR}/setup.py\\e[0m" - # shellcheck disable=SC2086 - pip install ${PIP_OPTS} "${SETUP_PY_DIR}/" - else - log_info "--- no dependency management tool, nor requirements file nor setup.py file found: skip install dependencies" - fi + ;; + esac } function _run() { - if [[ -f "pyproject.toml" ]] && [[ "${PYTHON_POETRY_DISABLED}" != "true" ]]; then - if ! command -v poetry > /dev/null - then - pip install poetry - fi + if [[ "${PYTHON_BUILD_SYSTEM}" == "poetry" ]] + then + # shellcheck disable=SC2086 + if ! command -v poetry > /dev/null; then pip install ${PIP_OPTS} poetry; fi poetry run "$@" else "$@" @@ -270,62 +324,79 @@ variables: } function _pip() { - _run pip "$@" + # shellcheck disable=SC2086 + _run pip ${PIP_OPTS} "$@" } - function _package(){ - if [[ -f "pyproject.toml" ]] && [[ "${PYTHON_POETRY_DISABLED}" != "true" ]]; then - pip install poetry + function _package() { + case "$PYTHON_BUILD_SYSTEM" in + poetry) + # shellcheck disable=SC2086 + if ! command -v poetry > /dev/null; then pip install ${PIP_OPTS} poetry; fi poetry build - else - pip install setuptools - python setup.py sdist bdist_wheel - fi + ;; + setuptools) + # shellcheck disable=SC2086 + pip install ${PIP_OPTS} setuptools build + python -m build + ;; + reqfile) + log_error "--- packaging is unsupported with requirements build system: read template doc" + exit 1 + ;; + esac } + function _publish() { - if [[ -f "pyproject.toml" ]] && [[ "${PYTHON_POETRY_DISABLED}" != "true" ]]; then - pip install poetry + case "$PYTHON_BUILD_SYSTEM" in + poetry) + # shellcheck disable=SC2086 + if ! command -v poetry > /dev/null; then pip install ${PIP_OPTS} poetry; fi poetry config repositories.user_defined "$TWINE_REPOSITORY_URL" poetry publish --username "$TWINE_USERNAME" --password "$TWINE_PASSWORD" --repository user_defined - else - pip install twine - pip list - + ;; + setuptools) + # shellcheck disable=SC2086 + pip install ${PIP_OPTS} twine twine upload --verbose dist/*.tar.gz twine upload --verbose dist/*.whl - fi + ;; + reqfile) + log_error "--- publish is unsupported with requirements build system: read template doc" + exit 1 + ;; + esac } function _release() { - if [[ -f "pyproject.toml" ]] && [[ "${PYTHON_POETRY_DISABLED}" != "true" ]]; then - pip install poetry + if [[ "${PYTHON_BUILD_SYSTEM}" == "poetry" ]] + then + # shellcheck disable=SC2086 + if ! command -v poetry > /dev/null; then pip install ${PIP_OPTS} poetry; fi poetry version "${RELEASE_VERSION_PART}" else - pip install bumpversion - release_args - bumpversion "${bumpversion_args}" - fi - } - function release_args() { - if [[ -f ".bumpversion.cfg" ]]; then - log_info "--- .bumpversion.cfg file found " - export bumpversion_args="${RELEASE_VERSION_PART} --verbose" - else - log_info "--- No .bumpversion.cfg file found " - if [[ -f "setup.py" ]]; then - log_info "--- Getting current version of setup.py file " - current_version=$(python setup.py --version) - export bumpversion_args=" --verbose --current-version ${current_version} --tag --tag-name {new_version} --commit ${RELEASE_VERSION_PART} setup.py" + # shellcheck disable=SC2086 + pip install ${PIP_OPTS} bumpversion + + if [[ -f ".bumpversion.cfg" ]]; then + log_info "--- .bumpversion.cfg file found " + export bumpversion_args="${RELEASE_VERSION_PART} --verbose" else - log_warn "--- No setup.py file found. Cannot perform release." + log_info "--- No .bumpversion.cfg file found " + if [[ -f "setup.py" ]]; then + log_info "--- Getting current version of setup.py file " + current_version=$(python setup.py --version) + export bumpversion_args=" --verbose --current-version ${current_version} --tag --tag-name {new_version} --commit ${RELEASE_VERSION_PART} setup.py" + else + log_warn "--- No setup.py file found. Cannot perform release." + fi fi + log_info "--- Release args: ${bumpversion_args}" + + bumpversion "${bumpversion_args}" fi - log_info "--- Release args: ${bumpversion_args}" } - - - function get_latest_template_version() { tag_json=$(wget -T 5 -q -O - "$CI_API_V4_URL/projects/to-be-continuous%2F$1/repository/tags?per_page=1" || echo "") echo "$tag_json" | sed -rn 's/^.*"name":"([^"]*)".*$/\1/p' @@ -368,6 +439,7 @@ variables: - *python-scripts - install_ca_certs "${CUSTOM_CA_CERTS:-$DEFAULT_CA_CERTS}" - cd ${PYTHON_PROJECT_DIR} + - guess_build_system ############################################################################################### # stages definition # @@ -387,7 +459,7 @@ py-lint: script: - mkdir -p reports - chmod o+rwx reports - - install_requirements build + - install_requirements - _pip install pylint_gitlab - | if ! _run pylint --ignore=.cache --output-format=text ${PYLINT_ARGS} ${PYLINT_FILES:-$(find -type f -name "*.py")} @@ -423,7 +495,7 @@ py-compile: extends: .python-base stage: build script: - - install_requirements build + - install_requirements - _python -m compileall $PYTHON_COMPILE_ARGS rules: # exclude merge requests @@ -441,7 +513,7 @@ py-unittest: script: - mkdir -p reports - chmod o+rwx reports - - install_requirements test + - install_requirements # code coverage - _pip install coverage # JUnit XML report @@ -473,7 +545,7 @@ py-pytest: script: - mkdir -p reports - chmod o+rwx reports - - install_requirements test + - install_requirements - _pip install pytest pytest-cov coverage - _python -m pytest --junit-xml=reports/TEST-pytests.xml --cov --cov-report term --cov-report xml:reports/coverage.xml ${PYTEST_ARGS} coverage: /^TOTAL.+?(\d+\%)$/ @@ -500,7 +572,7 @@ py-nosetests: script: - mkdir -p reports - chmod o+rwx reports - - install_requirements test + - install_requirements - _run nosetests --with-xunit --xunit-file=reports/TEST-nosetests.xml --with-coverage --cover-erase --cover-xml --cover-xml-file=reports/coverage.xml --cover-html --cover-html-dir=reports/coverage ${NOSETESTS_ARGS} coverage: /^TOTAL.+?(\d+\%)$/ artifacts: @@ -529,6 +601,7 @@ py-bandit: script: - mkdir -p reports - chmod o+rwx reports + - install_requirements - _pip install bandit - | if ! _run bandit ${TRACE+--verbose} ${BANDIT_ARGS} @@ -565,8 +638,8 @@ py-safety: script: - mkdir -p reports - chmod o+rwx reports + - install_requirements - _pip install safety - - install_requirements build - | if ! _pip freeze | _run safety check --stdin ${SAFETY_ARGS} then -- GitLab