Skip to content
Snippets Groups Projects
Commit ee0ca2dc authored by Pierre Smeyers's avatar Pierre Smeyers
Browse files

Merge branch 'feat/hatch' into 'master'

feat(system-build): add hatch support as a new build system

Closes #103

See merge request to-be-continuous/python!144
parents 02bb8215 f684e634
No related branches found
No related tags found
No related merge requests found
......@@ -50,7 +50,7 @@ The Python template uses some global configuration used throughout all jobs.
| `PIP_INDEX_URL` | Python repository url | _none_ |
| `PIP_EXTRA_INDEX_URL` | Extra Python repository url | _none_ |
| `pip-opts` / `PIP_OPTS` | pip [extra options](https://pip.pypa.io/en/stable/cli/pip/#general-options) | _none_ |
| `extra-deps` / `PYTHON_EXTRA_DEPS` | Python extra sets of dependencies to install<br/>For [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies) or [Poetry](https://python-poetry.org/docs/pyproject/#extras) only | _none_ |
| `extra-deps` / `PYTHON_EXTRA_DEPS` | Python extra sets of dependencies to install<br/>For [Setuptools](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=extras#optional-dependencies) or [Poetry](https://python-poetry.org/docs/pyproject/#extras) or [uv](https://docs.astral.sh/uv/) only | _none_ |
| `reqs-file` / `PYTHON_REQS_FILE` | Main requirements file _(relative to `$PYTHON_PROJECT_DIR`)_<br/>For [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) build-system only | `requirements.txt` |
| `extra-reqs-files` / `PYTHON_EXTRA_REQS_FILES` | Extra dev requirements file(s) to install _(relative to `$PYTHON_PROJECT_DIR`)_ | `requirements-dev.txt` |
......@@ -66,15 +66,28 @@ and/or `setup.py` and/or `requirements.txt`), but the build system might also be
| Value | Build System (scope) |
| ---------------- |--------------------------------------------------------------------------------------------------------|
| _none_ (default) or `auto` | The template tries to **auto-detect** the actual build system |
| `setuptools` | [Setuptools](https://setuptools.pypa.io/) (dependencies, build & packaging) |
| `poetry` | [Poetry](https://python-poetry.org/) (dependencies, build, test & packaging) |
| `uv` | [uv](https://docs.astral.sh/uv/) (dependencies, build, test & packaging)|
| `pipenv` | [Pipenv](https://pipenv.pypa.io/) (dependencies only) |
| `reqfile` | [Requirements Files](https://pip.pypa.io/en/stable/user_guide/#requirements-files) (dependencies only) |
| _none_ (default) or `auto` | The template tries to **auto-detect** :sparkles: the actual build system |
| `setuptools` | [![setuptools](https://img.shields.io/badge/setuptools-grey)](https://setuptools.pypa.io/) ![setuptools](https://img.shields.io/badge/dependencies_%7C_build_%7C_packaging-blue) |
| `poetry` | [![Poetry](https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json)](https://python-poetry.org/) ![Poetry](https://img.shields.io/badge/dependencies_%7C_build_%7C_test_%7C_packaging-blue) |
| `uv` | [![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://docs.astral.sh/uv/) ![uv](https://img.shields.io/badge/dependencies_%7C_build_%7C_test_%7C_packaging-blue) |
| `hatch` | [![Hatch project](https://img.shields.io/badge/%F0%9F%A5%9A-Hatch-4051b5.svg)](https://hatch.pypa.io/latest/) ![Hatch](https://img.shields.io/badge/dependencies_%7C_build_%7C_test_%7C_packaging-blue) |
| `pipenv` | [![Pipenv](https://img.shields.io/badge/Pipenv-grey)](https://pipenv.pypa.io/) ![Pipenv](https://img.shields.io/badge/dependencies-blue) |
| `reqfile` | [![Requirements Files](https://img.shields.io/badge/Requirements_Files-grey)](https://pip.pypa.io/en/stable/user_guide/#requirements-files) ![Pipenv](https://img.shields.io/badge/dependencies-blue) |
:warning: You can explicitly set the build tool version by setting `$PYTHON_BUILD_SYSTEM` variable including a [version identification](https://peps.python.org/pep-0440/). For example `PYTHON_BUILD_SYSTEM="poetry==1.1.15"`
### Hatch
All template jobs use the `default` Hatch environment. So dev dependencies should defined in the `[tool.hatch.envs.default.dependencies]` section of `pyproject.toml`.
```toml
[tool.hatch.envs.default]
dependencies = [
"pytest>=8.0.0,<9",
...
]
````
## Jobs
### `py-package` job
......@@ -396,7 +409,8 @@ This job is **disabled by default** and allows to perform a complete release of
The Python template supports three packaging systems:
* [Poetry](https://python-poetry.org/): uses Poetry-specific [version](https://python-poetry.org/docs/cli/#version), [build](https://python-poetry.org/docs/cli/#build) and [publish](https://python-poetry.org/docs/cli/#publish) commands.
* [uv](https://docs.astral.sh/uv/): uses [uv](https://docs.astral.sh/uv/) as version management, [build](https://docs.astral.sh/uv/guides/publish/#building-your-package) as package builder and [publish](https://docs.astral.sh/uv/guides/publish/) to publish.
* [uv](https://docs.astral.sh/uv/): uses [bump-my-version](https://github.com/callowayproject/bump-my-version) as version management, [build](https://docs.astral.sh/uv/guides/publish/#building-your-package) as package builder and [publish](https://docs.astral.sh/uv/guides/publish/) to publish.
* [hatch](https://hatch.pypa.io/latest/): uses [bump-my-version](https://github.com/callowayproject/bump-my-version) as version management, [build](https://hatch.pypa.io/latest/build/) as package builder and [publish](https://hatch.pypa.io/latest/publish/) to publish.
* [Setuptools](https://setuptools.pypa.io/): uses [bump-my-version](https://github.com/callowayproject/bump-my-version) as version management, [build](https://pypa-build.readthedocs.io/) as package builder and [Twine](https://twine.readthedocs.io/) to publish.
The release job is bound to the `publish` stage, appears only on production and integration branches and uses the following variables:
......@@ -421,7 +435,8 @@ This job is **disabled by default** and allow to publish the built packages to a
The Python template supports three packaging systems:
* [Poetry](https://python-poetry.org/): uses Poetry-specific [version](https://python-poetry.org/docs/cli/#version), [build](https://python-poetry.org/docs/cli/#build) and [publish](https://python-poetry.org/docs/cli/#publish) commands.
* [uv](https://docs.astral.sh/uv/): uses [uv](https://docs.astral.sh/uv/) as version management, [build](https://docs.astral.sh/uv/guides/publish/#building-your-package) as package builder and [publish](https://docs.astral.sh/uv/guides/publish/) to publish.
* [uv](https://docs.astral.sh/uv/): uses [bump-my-version](https://github.com/callowayproject/bump-my-version) as version management, [build](https://docs.astral.sh/uv/guides/publish/#building-your-package) as package builder and [publish](https://docs.astral.sh/uv/guides/publish/) to publish.
* [hatch](https://hatch.pypa.io/latest//): uses [bump-my-version](https://github.com/callowayproject/bump-my-version) as version management, [build](https://hatch.pypa.io/latest/build/) as package builder and [publish](https://hatch.pypa.io/latest/publish/) to publish.
* [Setuptools](https://setuptools.pypa.io/): uses [bump-my-version](https://github.com/callowayproject/bump-my-version) as version management, [build](https://pypa-build.readthedocs.io/) as package builder and [Twine](https://twine.readthedocs.io/) to publish.
The publish job is bound to the `publish` stage, is executed on a Git tag matching [semantic versioning pattern](https://semver.org/) and uses the following variables:
......
......@@ -21,7 +21,7 @@
"name": "PYTHON_BUILD_SYSTEM",
"description": "Python build-system to use to install dependencies, build and package the project",
"type": "enum",
"values": ["auto", "setuptools", "poetry", "pipenv", "reqfile", "uv"],
"values": ["auto", "setuptools", "poetry", "pipenv", "reqfile", "uv", "hatch"],
"default": "auto",
"advanced": true
},
......
......@@ -30,6 +30,7 @@ spec:
- pipenv
- reqfile
- uv
- hatch
default: auto
reqs-file:
description: |-
......@@ -332,6 +333,17 @@ variables:
echo -e "[\\e[1;91mERROR\\e[0m] $*"
}
function log_elapsed_time() {
_end_time=$(get_current_ts_ms)
_named=$1
_start_time=$2
_diff_ms=$((_end_time - _start_time))
_elapsed_sec="$((_diff_ms / 1000)).$(((_diff_ms % 1000) / 100))"
log_info " *** ${_named} took \\e[32m$_elapsed_sec\\e[0m seconds"
}
function fail() {
log_error "$*"
exit 1
......@@ -345,6 +357,16 @@ variables:
fi
}
function get_current_ts_ms() {
if command -v busybox > /dev/null
then
# no easy way to retrieve ms in BusyBox
echo $(($(date +%s) * 1000))
else
echo $(($(date +%s%N) / 1000000))
fi
}
function install_ca_certs() {
certs=$1
if [[ -z "$certs" ]]
......@@ -586,11 +608,14 @@ variables:
}
function guess_build_system() {
_start_time=$(get_current_ts_ms)
case "${PYTHON_BUILD_SYSTEM:-auto}" in
auto)
;;
poetry*|setuptools*|pipenv*|uv*)
log_info "--- Build system explicitly declared: ${PYTHON_BUILD_SYSTEM}"
poetry*|setuptools*|pipenv*|uv*|hatch*)
export PYTHON_BUILD_SYSTEM_CMD="${PYTHON_BUILD_SYSTEM%%[=<>]*}"
log_info "--- Build system explicitly declared: ${PYTHON_BUILD_SYSTEM} cmd=\\e[33m$PYTHON_BUILD_SYSTEM_CMD\\e[0m"
return
;;
reqfile)
......@@ -615,6 +640,7 @@ variables:
then
log_info "--- Build system auto-detected: uv (uv.lock and pyproject.toml)"
export PYTHON_BUILD_SYSTEM="uv"
export PYTHON_BUILD_SYSTEM_CMD="uv"
return
fi
log_error "--- Build system auto-detected: uv (uv.lock) but no pyproject.toml found: please read template doc"
......@@ -632,6 +658,13 @@ variables:
poetry.core.masonry.api)
log_info "--- Build system auto-detected: PEP 517 with Poetry backend"
export PYTHON_BUILD_SYSTEM="poetry"
export PYTHON_BUILD_SYSTEM_CMD="poetry"
return
;;
hatchling.build)
log_info "--- Build system auto-detected: PEP 517 with Hatch backend"
export PYTHON_BUILD_SYSTEM="hatch"
export PYTHON_BUILD_SYSTEM_CMD="hatch"
return
;;
setuptools.build_meta)
......@@ -658,17 +691,12 @@ variables:
log_error "--- Build system auto-detect failed: please read template doc"
exit 1
fi
}
function maybe_install_poetry() {
if [[ "$PYTHON_BUILD_SYSTEM" =~ ^poetry ]] && ! command -v poetry > /dev/null
then
# shellcheck disable=SC2086
pip install ${PIP_OPTS} "$PYTHON_BUILD_SYSTEM"
fi
log_elapsed_time "guess_build_system" "$_start_time"
}
function maybe_install_uv() {
if [[ "$PYTHON_BUILD_SYSTEM" =~ ^uv ]] && ! command -v uv > /dev/null
function maybe_install_build_system() {
if ! command -v "$PYTHON_BUILD_SYSTEM_CMD" > /dev/null
then
# shellcheck disable=SC2086
pip install ${PIP_OPTS} "$PYTHON_BUILD_SYSTEM"
......@@ -677,23 +705,23 @@ variables:
# install requirements
function install_requirements() {
_start_time=$(get_current_ts_ms)
case "$PYTHON_BUILD_SYSTEM" in
poetry*)
if [[ ! -f "poetry.lock" ]]; then
log_warn "Using Poetry but \\e[33;1mpoetry.lock\\e[0m file not found: you shall commit it with your project files"
fi
maybe_install_poetry
maybe_install_build_system
poetry install ${PYTHON_EXTRA_DEPS:+--extras "$PYTHON_EXTRA_DEPS"}
;;
setuptools*)
# shellcheck disable=SC2086
pip install ${PIP_OPTS} "$PYTHON_BUILD_SYSTEM"
maybe_install_build_system
# shellcheck disable=SC2086
pip install ${PIP_OPTS} ".${PYTHON_EXTRA_DEPS:+[$PYTHON_EXTRA_DEPS]}"
;;
pipenv*)
# shellcheck disable=SC2086
pip install ${PIP_OPTS} "$PYTHON_BUILD_SYSTEM"
maybe_install_build_system
if [[ ! -f "Pipfile.lock" ]]; then
log_warn "Using Pipenv but \\e[33;1mPipfile.lock\\e[0m file not found: you shall commit it with your project files"
pipenv install --dev --system
......@@ -722,21 +750,31 @@ variables:
if [[ ! -f "uv.lock" ]]; then
log_warn "Using uv but \\e[33;1muv.lock\\e[0m file not found: you shall commit it with your project files"
fi
maybe_install_uv
maybe_install_build_system
uv sync --frozen ${PYTHON_EXTRA_DEPS:+--extra "$PYTHON_EXTRA_DEPS"}
;;
hatch*)
maybe_install_build_system
hatch env create
;;
esac
log_elapsed_time "install_requirements" "$_start_time"
}
function _run() {
if [[ "$PYTHON_BUILD_SYSTEM" =~ ^poetry ]]
then
maybe_install_poetry
maybe_install_build_system
poetry run "$@"
elif [[ "$PYTHON_BUILD_SYSTEM" =~ ^uv ]]
then
maybe_install_uv
maybe_install_build_system
uv run "$@"
elif [[ "$PYTHON_BUILD_SYSTEM" =~ ^hatch ]]
then
maybe_install_build_system
$PYTHON_BUILD_SYSTEM_CMD run "$@"
else
"$@"
fi
......@@ -752,7 +790,7 @@ variables:
if [[ "$PYTHON_BUILD_SYSTEM" =~ ^uv ]]
then
maybe_install_uv
maybe_install_build_system
# shellcheck disable=SC2086
uv pip "$cmd" ${PIP_OPTS} "$@"
else
......@@ -762,19 +800,31 @@ variables:
}
function py_package() {
_start_time=$(get_current_ts_ms)
if [[ "$PYTHON_BUILD_SYSTEM" =~ ^poetry ]]
then
maybe_install_poetry
poetry build
maybe_install_build_system
log_info "--- build packages (poetry)..."
poetry build ${TRACE+--verbose}
elif [[ "$PYTHON_BUILD_SYSTEM" =~ ^uv ]]
then
maybe_install_uv
uv build
maybe_install_build_system
log_info "--- build packages (uv)..."
uv build ${TRACE+--verbose}
elif [[ "$PYTHON_BUILD_SYSTEM" =~ ^hatch ]]
then
log_info "--- build packages (hatch)..."
maybe_install_build_system
$PYTHON_BUILD_SYSTEM_CMD build
else
log_info "--- build packages ..."
# shellcheck disable=SC2086
pip install ${PIP_OPTS} build
python -m build
fi
log_elapsed_time "py_package" "$_start_time"
}
function configure_scm_auth() {
......@@ -800,7 +850,31 @@ variables:
fi
}
function py_bump_my_version() {
mkdir -p -m 777 tbc_tmp >&2
py_cur_version="$1"
py_release_part="$2"
echo "$py_cur_version" > tbc_tmp/version.txt
_pip install bump-my-version >&2
log_info "[bump-my-version] increase \\e[1;94m${py_release_part}\\e[0m (from current \\e[1;94m${py_cur_version}\\e[0m)" >&2
_run bump-my-version bump ${TRACE+--verbose} --current-version "$py_cur_version" "$py_release_part" tbc_tmp/version.txt >&2
cat tbc_tmp/version.txt
rm -fr tbc_tmp/version.txt >&2
}
function py_commit_pyproject() {
py_cur_version="$1"
py_next_version="$2"
# Git commit and tag
git add pyproject.toml
# emulate bump-my-version to generate commit message
py_commit_message=$(python -c "print('$PYTHON_RELEASE_COMMIT_MESSAGE'.format(current_version='$py_cur_version', new_version='$py_next_version'))")
git commit -m "$py_commit_message"
git tag "$py_next_version"
}
function py_release() {
log_info "PYTHON_RELEASE_ENABLED:$PYTHON_RELEASE_ENABLED PYTHON_PUBLISH_ENABLED:$PYTHON_PUBLISH_ENABLED"
# 1: retrieve next release info from semantic-release
if [ "$SEMREL_INFO_ON" ] && [ "$PYTHON_SEMREL_RELEASE_DISABLED" != "true" ]
then
......@@ -819,7 +893,7 @@ variables:
# 2: bump-my-version (+ Git commit & tag)
if [[ "$PYTHON_BUILD_SYSTEM" =~ ^poetry ]]
then
maybe_install_poetry
maybe_install_build_system
if [[ -z "$py_next_version" ]]
then
py_cur_version=$(poetry version --short)
......@@ -837,7 +911,7 @@ variables:
git tag "$py_next_version"
elif [[ "$PYTHON_BUILD_SYSTEM" =~ ^uv ]]
then
maybe_install_uv
maybe_install_build_system
if [[ -z "$py_next_version" ]]
then
# quick version waiting for uv to manage bump
......@@ -862,6 +936,19 @@ variables:
py_commit_message=$(python -c "print('$PYTHON_RELEASE_COMMIT_MESSAGE'.format(current_version='$py_cur_version', new_version='$py_next_version'))")
git commit -m "$py_commit_message"
git tag --force "$py_next_version"
elif [[ "$PYTHON_BUILD_SYSTEM" =~ ^hatch ]]
then
maybe_install_build_system
if [[ -z "$py_next_version" ]]
then
py_cur_version=$(hatch version)
py_next_version=$(py_bump_my_version "$py_cur_version" "$PYTHON_RELEASE_NEXT")
fi
log_info "[hatch] change version \\e[1;94m${py_cur_version}\\e[0m → \\e[1;94m${py_next_version}\\e[0m"
_pip install toml-cli
_run toml set --toml-path pyproject.toml project.version "$py_next_version"
py_commit_pyproject "$py_cur_version" "$py_next_version"
else
# Setuptools / bump-my-version
# shellcheck disable=SC2086
......@@ -909,9 +996,14 @@ variables:
}
function py_publish() {
if [[ "$PYTHON_BUILD_SYSTEM" =~ ^hatch ]] && [[ "$PYTHON_PACKAGE_ENABLED" != "true" ]]
then
py_package
fi
if [[ "$PYTHON_BUILD_SYSTEM" =~ ^poetry ]]
then
maybe_install_poetry
maybe_install_build_system
if [[ "$PYTHON_PACKAGE_ENABLED" != "true" ]]
then
......@@ -924,7 +1016,7 @@ variables:
poetry publish ${TRACE+--verbose} --username "$PYTHON_REPOSITORY_USERNAME" --password "$PYTHON_REPOSITORY_PASSWORD" --repository user_defined
elif [[ "$PYTHON_BUILD_SYSTEM" =~ ^uv ]]
then
maybe_install_uv
maybe_install_build_system
if [[ "$PYTHON_PACKAGE_ENABLED" != "true" ]]
then
......@@ -934,6 +1026,11 @@ variables:
log_info "--- publish packages (uv) to $PYTHON_REPOSITORY_URL with user $PYTHON_REPOSITORY_USERNAME..."
uv publish ${TRACE+--verbose} --username "$PYTHON_REPOSITORY_USERNAME" --password "$PYTHON_REPOSITORY_PASSWORD" --publish-url "$PYTHON_REPOSITORY_URL"
elif [[ "$PYTHON_BUILD_SYSTEM" =~ ^hatch ]]
then
maybe_install_build_system
log_info "--- publish packages (hatch) to $PYTHON_REPOSITORY_URL with user $PYTHON_REPOSITORY_USERNAME..."
hatch publish ${TRACE+--verbose} --no-prompt --yes --user "$PYTHON_REPOSITORY_USERNAME" --auth "$PYTHON_REPOSITORY_PASSWORD" --repo "$PYTHON_REPOSITORY_URL"
else
# shellcheck disable=SC2086
pip install ${PIP_OPTS} build twine
......@@ -997,6 +1094,8 @@ stages:
POETRY_CACHE_DIR: "$CI_PROJECT_DIR/.cache/poetry"
PIPENV_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pipenv"
UV_CACHE_DIR: "$CI_PROJECT_DIR/.cache/uv"
HATCH_CACHE_DIR: "$CI_PROJECT_DIR/.cache/hatch/.cache/"
HATCH_DATA_DIR: "$CI_PROJECT_DIR/.cache/hatch/.local"
POETRY_VIRTUALENVS_IN_PROJECT: "false"
cache:
key: "$CI_COMMIT_REF_SLUG-python"
......@@ -1297,9 +1396,14 @@ py-trivy:
;;
uv*)
log_info "$PYTHON_BUILD_SYSTEM build system used (\\e[32mmust generate pinned requirements.txt from uv.lock\\e[0m)"
maybe_install_uv
maybe_install_build_system
uv export > ./reports/requirements.txt
;;
hatch*)
log_info "$PYTHON_BUILD_SYSTEM build system used (\\e[32mmust generate pinned requirements.txt\\e[0m)"
maybe_install_build_system
hatch run python -m pip freeze > reports/requirements.txt # hatch dep show requirements not working and complete
;;
*)
log_info "$PYTHON_BUILD_SYSTEM build system used (\\e[32mmust generate pinned requirements.txt\\e[0m)"
install_requirements
......@@ -1356,9 +1460,14 @@ py-sbom:
;;
uv*)
log_info "$PYTHON_BUILD_SYSTEM build system used (\\e[32mmust generate pinned requirements.txt from uv.lock\\e[0m)"
maybe_install_uv
maybe_install_build_system
uv export > ./reports/requirements.txt
;;
hatch*)
log_info "$PYTHON_BUILD_SYSTEM build system used (\\e[32mmust generate pinned requirements.txt from uv.lock\\e[0m)"
maybe_install_build_system
hatch run python -m pip freeze > reports/requirements.txt # hatch dep show requirements not working and complete
;;
*)
log_info "$PYTHON_BUILD_SYSTEM build system used (\\e[32mmust generate pinned requirements.txt\\e[0m)"
install_requirements
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment