From 3201240e9437d7602779fb33252e8dce7e028257 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?C=C3=A9dric=20OLIVIER?= <cedric3.olivier@orange.com> Date: Tue, 20 Sep 2022 07:29:35 +0000 Subject: [PATCH] feat: add custom DOCKER_CONFIG_FILE var --- README.md | 81 ++++++++++++++++++++++++++++++++++ kicker.json | 6 +++ templates/gitlab-ci-docker.yml | 57 +++++++++++++++--------- 3 files changed, 122 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index b7f2d51..8065ef6 100644 --- a/README.md +++ b/README.md @@ -47,6 +47,7 @@ In addition to this, the template supports _standard_ Linux proxy variables: | `https_proxy` | Proxy used for https requests | _none_ | | `no_proxy` | List of comma-separated hosts/host suffixes | _none_ | + ### Images For each Dockerfile, the template builds an image that may be [pushed](https://docs.docker.com/engine/reference/commandline/push/) @@ -72,6 +73,7 @@ The **snapshot** and **release** images are defined by the following variables: | `DOCKER_SNAPSHOT_IMAGE` | Docker snapshot image | `$CI_REGISTRY_IMAGE/snapshot:$CI_COMMIT_REF_SLUG` | | `DOCKER_RELEASE_IMAGE` | Docker release image | `$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME` | + As you can see, the Docker template is configured by default to use the GitLab container registry. You may perfectly override this and use another Docker registry, but be aware of a few things: @@ -90,6 +92,7 @@ But when using other registry(ies), you'll have also to **configure appropriate #### Using the same registry for snapshot and release If you use the **same registry** for both snapshot and release images, you shall use the following configuration + variables: | Name | Description | @@ -97,6 +100,7 @@ variables: | :lock: `DOCKER_REGISTRY_USER` | Docker registry username for image registry | | :lock: `DOCKER_REGISTRY_PASSWORD`| Docker registry password for image registry | + #### Using different registries for snapshot and release If you use **different registries** for snapshot and release images, you shall use separate configuration variables: @@ -108,6 +112,83 @@ If you use **different registries** for snapshot and release images, you shall u | :lock: `DOCKER_REGISTRY_RELEASE_USER` | Docker registry username for release image registry | | :lock: `DOCKER_REGISTRY_RELEASE_PASSWORD`| Docker registry password for release image registry | + + +#### Setting your own Docker configuration file (advanced) + +There might be cases where you need to provide the complete [Docker configuration file](https://docs.docker.com/engine/reference/commandline/cli/#configuration-files): + +* need to declare authentication credentials for other registries than the 2 predefined ones (snapshot & release), +* need to declare a [credentials store](https://docs.docker.com/engine/reference/commandline/login/#credentials-store) (ex: in order to [publish to Amazon ECR](https://github.com/GoogleContainerTools/kaniko#pushing-to-amazon-ecr) with Kaniko for instance), +* need to declare [proxies](https://docs.docker.com/engine/reference/commandline/cli/#automatic-proxy-configuration-for-containers), +* ... + +If you are in one of those cases, you will need to use the `DOCKER_CONFIG_FILE` variable, expected to declare the path to your custom Docker configuration file (JSON). You may: + +* leave the default value (`.docker/config.json`) or override it to some alternate location in your project repository and create the file **without any secret in it** using our dynamic variables replacement (see below), +* or override it as a GitLab project variable of type [File](https://docs.gitlab.com/ee/ci/variables/#cicd-variable-types), possibly inlining your secret credentials in it. + +| Name | Description | Default value | +| ------------------------- | --------------------- | ------------------------------------------------- | +| `DOCKER_CONFIG_FILE` | Path to the Docker configuration file (JSON) | `.docker/config.json` | + +Moreover, this file supports **dynamic environment variables replacement**. +That means it may contain references to other environment variables (in the format `${variable_name}`) that will be dynamically replaced +by the template before evaluation. +In addition to you own defined variables, you may use the following variables (provided and managed by the template): + +* `${docker_snapshot_authent_token}`: the authentication token required by the snapshot registry (computed from configured `DOCKER_REGISTRY_SNAPSHOT_USER` / `DOCKER_REGISTRY_SNAPSHOT_PASSWORD` variables) +* `${docker_snapshot_registry_host}`: the snapshot registry host (based on the configured `DOCKER_SNAPSHOT_IMAGE ` variable) +* `${docker_release_authent_token}`: the authentication token required by the release registry (computed from configured `DOCKER_REGISTRY_RELEASE_USER` / `DOCKER_REGISTRY_RELEASE_PASSWORD` variables) +* `${docker_release_registry_host}`: the release registry host (based on the configured `DOCKER_RELEASE_IMAGE ` variable) + +Example 1: Docker configuration file inlined in the project repository (`.docker/config.json`) with **dynamic variables replacement**: + +```json +{ + "auths": { + "${docker_snapshot_registry_host}": { + "auth": "${docker_release_authent_token}" + }, + "${docker_release_registry_host}": { + "auth": "${docker_snapshot_authent_token}" + }, + "my-readonly-repo-to-pull": { + "auth": "${MY_OWN_REGISTRY_TOKEN}" + } + } +} +``` + +This file uses: + +* template-managed `${docker_snapshot_authent_token}`, `${docker_snapshot_registry_host}`, `${docker_release_authent_token}` and `${docker_release_registry_host}` variables, +* the user-defined `${MY_OWN_REGISTRY_TOKEN}` (:information_source: an authentication token can be obtained with command `echo "user:password" | base64` and then be stored as a masked GitLab CI/CD project variable). + +Example 2: Docker configuration file declared as a GitLab project variable of type [File](https://docs.gitlab.com/ee/ci/variables/#cicd-variable-types) with **dynamic variables replacement**: + + +```json +{ + "auths": { + "$${docker_snapshot_registry_host}": { + "auth": "$${docker_release_authent_token}" + }, + "$${docker_release_registry_host}": { + "auth": "$${docker_snapshot_authent_token}" + }, + "my-readonly-repo-to-pull": { + "auth": "ZG9ja2VyZHVkZTpnb3RjaGEh" + } + } +} +``` + +This file uses: + +* template-managed `${docker_snapshot_authent_token}`, `${docker_snapshot_registry_host}`, `${docker_release_authent_token}` and `${docker_release_registry_host}` variables (:warning: mind the double `$$` to prevent GitLab from [trying to evaluate the variable](https://docs.gitlab.com/ee/ci/variables/index.html#use-the--character-in-variables)), +* the user-defined authentication may be inlined as a GitLab project variable is a place safe enough to store secrets. + ## Multi Dockerfile support This template supports building multiple Docker images from a single Git repository. diff --git a/kicker.json b/kicker.json index 66b0e6a..c54ca13 100644 --- a/kicker.json +++ b/kicker.json @@ -34,6 +34,12 @@ "description": "The Docker [context path](https://docs.docker.com/engine/reference/commandline/build/#build-with-path) (working directory) - _only set if you want a context path different from the Dockerfile location_", "advanced": true }, + { + "name": "DOCKER_CONFIG_FILE", + "description": "Path to the [Docker configuration file](https://docs.docker.com/engine/reference/commandline/cli/#sample-configuration-file) (JSON)", + "default": ".docker/config.json", + "advanced": true + }, { "name": "DOCKER_SNAPSHOT_IMAGE", "description": "Docker snapshot image", diff --git a/templates/gitlab-ci-docker.yml b/templates/gitlab-ci-docker.yml index 074a9cf..3e3bdc3 100644 --- a/templates/gitlab-ci-docker.yml +++ b/templates/gitlab-ci-docker.yml @@ -1,16 +1,16 @@ # ========================================================================================= # Copyright (C) 2021 Orange & contributors # -# This program is free software; you can redistribute it and/or modify it under the terms -# of the GNU Lesser General Public License as published by the Free Software Foundation; +# This program is free software; you can redistribute it and/or modify it under the terms +# of the GNU Lesser General Public License as published by the Free Software Foundation; # either version 3 of the License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; # without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # See the GNU Lesser General Public License for more details. # -# You should have received a copy of the GNU Lesser General Public License along with this -# program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth +# You should have received a copy of the GNU Lesser General Public License along with this +# program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth # Floor, Boston, MA 02110-1301, USA. # ========================================================================================= # default workflow rules: Merge Request pipelines @@ -54,6 +54,7 @@ variables: # for retro-compatibility (deprecated & undocumented) DOCKER_DOCKERFILE_PATH: "." DOCKER_FILE: "$DOCKER_DOCKERFILE_PATH/Dockerfile" + DOCKER_CONFIG_FILE: ".docker/config.json" # When testing a Docker Health (test stage), how long (in seconds) wait for the HealthCheck status (https://docs.docker.com/engine/reference/builder/#healthcheck) DOCKER_HEALTHCHECK_TIMEOUT: "60" @@ -166,8 +167,8 @@ stages: _test_op=$(echo "$_fields" | cut -d: -f5) case "$_test_op" in defined) - if [[ -z "$_not" ]] && [[ -z "$_cond_val" ]]; then continue; - elif [[ "$_not" ]] && [[ "$_cond_val" ]]; then continue; + if [[ -z "$_not" ]] && [[ -z "$_cond_val" ]]; then continue; + elif [[ "$_not" ]] && [[ "$_cond_val" ]]; then continue; fi ;; equals|startswith|endswith|contains|in|equals_ic|startswith_ic|endswith_ic|contains_ic|in_ic) @@ -186,28 +187,28 @@ stages: fi case "$_test_op" in equals*) - if [[ -z "$_not" ]] && [[ "$_cond_val" != "$_cmp_val" ]]; then continue; - elif [[ "$_not" ]] && [[ "$_cond_val" == "$_cmp_val" ]]; then continue; + if [[ -z "$_not" ]] && [[ "$_cond_val" != "$_cmp_val" ]]; then continue; + elif [[ "$_not" ]] && [[ "$_cond_val" == "$_cmp_val" ]]; then continue; fi ;; startswith*) - if [[ -z "$_not" ]] && [[ "$_cond_val" != "$_cmp_val"* ]]; then continue; - elif [[ "$_not" ]] && [[ "$_cond_val" == "$_cmp_val"* ]]; then continue; + if [[ -z "$_not" ]] && [[ "$_cond_val" != "$_cmp_val"* ]]; then continue; + elif [[ "$_not" ]] && [[ "$_cond_val" == "$_cmp_val"* ]]; then continue; fi ;; endswith*) - if [[ -z "$_not" ]] && [[ "$_cond_val" != *"$_cmp_val" ]]; then continue; - elif [[ "$_not" ]] && [[ "$_cond_val" == *"$_cmp_val" ]]; then continue; + if [[ -z "$_not" ]] && [[ "$_cond_val" != *"$_cmp_val" ]]; then continue; + elif [[ "$_not" ]] && [[ "$_cond_val" == *"$_cmp_val" ]]; then continue; fi ;; contains*) - if [[ -z "$_not" ]] && [[ "$_cond_val" != *"$_cmp_val"* ]]; then continue; - elif [[ "$_not" ]] && [[ "$_cond_val" == *"$_cmp_val"* ]]; then continue; + if [[ -z "$_not" ]] && [[ "$_cond_val" != *"$_cmp_val"* ]]; then continue; + elif [[ "$_not" ]] && [[ "$_cond_val" == *"$_cmp_val"* ]]; then continue; fi ;; in*) - if [[ -z "$_not" ]] && [[ "__${_cmp_val}__" != *"__${_cond_val}__"* ]]; then continue; - elif [[ "$_not" ]] && [[ "__${_cmp_val}__" == *"__${_cond_val}__"* ]]; then continue; + if [[ -z "$_not" ]] && [[ "__${_cmp_val}__" != *"__${_cond_val}__"* ]]; then continue; + elif [[ "$_not" ]] && [[ "__${_cmp_val}__" == *"__${_cond_val}__"* ]]; then continue; fi ;; esac @@ -303,23 +304,35 @@ stages: docker info > /dev/null 2>&1 } + function awkenvsubst() { + awk '{while(match($0,"[$]{[^}]*}")) {var=substr($0,RSTART+2,RLENGTH -3);val=ENVIRON[var];gsub(/["\\]/,"\\\\&", val);gsub("\n", "\\n", val);gsub("\r", "\\r", val);gsub("[$]{"var"}",val)}}1' + } + function configure_registries_auth() { docker_snapshot_authent_token=$(echo -n "${DOCKER_REGISTRY_SNAPSHOT_USER:-${DOCKER_REGISTRY_USER:-$CI_REGISTRY_USER}}:${DOCKER_REGISTRY_SNAPSHOT_PASSWORD:-${DOCKER_REGISTRY_PASSWORD:-$CI_REGISTRY_PASSWORD}}" | base64 | tr -d '\n') docker_snapshot_registry_host=$(echo "$DOCKER_SNAPSHOT_IMAGE" | cut -d/ -f1) + export docker_snapshot_authent_token + export docker_snapshot_registry_host docker_release_authent_token=$(echo -n "${DOCKER_REGISTRY_RELEASE_USER:-${DOCKER_REGISTRY_USER:-$CI_REGISTRY_USER}}:${DOCKER_REGISTRY_RELEASE_PASSWORD:-${DOCKER_REGISTRY_PASSWORD:-$CI_REGISTRY_PASSWORD}}" | base64 | tr -d '\n') docker_release_registry_host=$(echo "$DOCKER_RELEASE_IMAGE" | cut -d/ -f1) + export docker_release_authent_token + export docker_release_registry_host docker_snapshot_config_json=$(echo -n "{\"auths\":{\"$docker_snapshot_registry_host\":{\"auth\":\"$docker_snapshot_authent_token\"},\"HttpHeaders\":{\"User-Agent\":\"$USER_AGENT\"}}}") docker_release_config_json=$(echo -n "{\"auths\":{\"$docker_release_registry_host\":{\"auth\":\"$docker_release_authent_token\"},\"HttpHeaders\":{\"User-Agent\":\"$USER_AGENT\"}}}") - # Create the configuration file for Docker + # Create the configuration file for Docker and Kaniko mkdir -p /root/.docker - echo "${docker_snapshot_config_json}" > /root/.docker/config.json - - # Create the configuration file for Kaniko mkdir -p /kaniko/.docker - echo "${docker_snapshot_config_json}" > /kaniko/.docker/config.json + if [ -f "${DOCKER_CONFIG_FILE}" ] + then + awkenvsubst < "${DOCKER_CONFIG_FILE}" > /root/.docker/config.json + awkenvsubst < "${DOCKER_CONFIG_FILE}" > /kaniko/.docker/config.json + else + echo "${docker_snapshot_config_json}" > /root/.docker/config.json + echo "${docker_snapshot_config_json}" > /kaniko/.docker/config.json + fi # Create the configuration file for Skopeo mkdir -p /skopeo/.docker @@ -706,4 +719,4 @@ docker-publish: # if $AUTODEPLOY_TO_PROD: auto - if: '$AUTODEPLOY_TO_PROD == "true"' # else: manual + blocking - - when: manual \ No newline at end of file + - when: manual -- GitLab