Merge branch '36-use_auth_for_pull' into 'master'

feat: add custom DOCKER_CONFIG_FILE var

Closes #36

See merge request to-be-continuous/docker!50

README.md

@@ -47,6 +47,7 @@ In addition to this, the template supports _standard_ Linux proxy variables:
 | `https_proxy`         | Proxy used for https requests               | _none_        |
 | `no_proxy`            | List of comma-separated hosts/host suffixes | _none_        |
 
+
 ### Images
 
 For each Dockerfile, the template builds an image that may be [pushed](https://docs.docker.com/engine/reference/commandline/push/)
@@ -72,6 +73,7 @@ The **snapshot** and **release** images are defined by the following variables:
 | `DOCKER_SNAPSHOT_IMAGE`   | Docker snapshot image | `$CI_REGISTRY_IMAGE/snapshot:$CI_COMMIT_REF_SLUG` |
 | `DOCKER_RELEASE_IMAGE`    | Docker release image  | `$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME`          |
 
+
 As you can see, the Docker template is configured by default to use the GitLab container registry.
 You may perfectly override this and use another Docker registry, but be aware of a few things:
 
@@ -90,6 +92,7 @@ But when using other registry(ies), you'll have also to **configure appropriate
 #### Using the same registry for snapshot and release
 
 If you use the **same registry** for both snapshot and release images, you shall use the following configuration
+
 variables:
 
 | Name                             | Description                            |
@@ -97,6 +100,7 @@ variables:
 | :lock: `DOCKER_REGISTRY_USER`    | Docker registry username for image registry |
 | :lock: `DOCKER_REGISTRY_PASSWORD`| Docker registry password for image registry  |
 
+
 #### Using different registries for snapshot and release
 
 If you use **different registries** for snapshot and release images, you shall use separate configuration variables:
@@ -108,6 +112,83 @@ If you use **different registries** for snapshot and release images, you shall u
 | :lock: `DOCKER_REGISTRY_RELEASE_USER`    | Docker registry username for release image registry |
 | :lock: `DOCKER_REGISTRY_RELEASE_PASSWORD`| Docker registry password for release image registry |
 
+
+
+#### Setting your own Docker configuration file (advanced)
+
+There might be cases where you need to provide the complete [Docker configuration file](https://docs.docker.com/engine/reference/commandline/cli/#configuration-files):
+
+* need to declare authentication credentials for other registries than the 2 predefined ones (snapshot & release), 
+* need to declare a [credentials store](https://docs.docker.com/engine/reference/commandline/login/#credentials-store) (ex: in order to [publish to Amazon ECR](https://github.com/GoogleContainerTools/kaniko#pushing-to-amazon-ecr) with Kaniko for instance),
+* need to declare [proxies](https://docs.docker.com/engine/reference/commandline/cli/#automatic-proxy-configuration-for-containers),
+* ...
+
+If you are in one of those cases, you will need to use the `DOCKER_CONFIG_FILE` variable, expected to declare the path to your custom Docker configuration file (JSON). You may:
+
+* leave the default value (`.docker/config.json`) or override it to some alternate location in your project repository and create the file **without any secret in it** using our dynamic variables replacement (see below),
+* or override it as a GitLab project variable of type [File](https://docs.gitlab.com/ee/ci/variables/#cicd-variable-types), possibly inlining your secret credentials in it.
+
+| Name                      | Description           | Default value                                     |
+| ------------------------- | --------------------- | ------------------------------------------------- |
+| `DOCKER_CONFIG_FILE`      | Path to the Docker configuration file (JSON) | `.docker/config.json` |
+
+Moreover, this file supports **dynamic environment variables replacement**. 
+That means it may contain references to other environment variables (in the format `${variable_name}`) that will be dynamically replaced
+by the template before evaluation.
+In addition to you own defined variables, you may use the following variables (provided and managed by the template):
+
+* `${docker_snapshot_authent_token}`: the authentication token required by the snapshot registry (computed from configured `DOCKER_REGISTRY_SNAPSHOT_USER` / `DOCKER_REGISTRY_SNAPSHOT_PASSWORD` variables)
+* `${docker_snapshot_registry_host}`: the snapshot registry host (based on the configured `DOCKER_SNAPSHOT_IMAGE ` variable)
+* `${docker_release_authent_token}`: the authentication token required by the release registry (computed from configured `DOCKER_REGISTRY_RELEASE_USER` / `DOCKER_REGISTRY_RELEASE_PASSWORD` variables)
+* `${docker_release_registry_host}`: the release registry host (based on the configured `DOCKER_RELEASE_IMAGE ` variable)
+
+Example 1: Docker configuration file inlined in the project repository (`.docker/config.json`) with **dynamic variables replacement**:
+
+```json
+{
+    "auths": {
+        "${docker_snapshot_registry_host}": {
+            "auth": "${docker_release_authent_token}"
+        },
+        "${docker_release_registry_host}": {
+            "auth": "${docker_snapshot_authent_token}"
+        },
+        "my-readonly-repo-to-pull": {
+            "auth": "${MY_OWN_REGISTRY_TOKEN}"
+        }
+    }
+}
+```
+
+This file uses:
+
+* template-managed `${docker_snapshot_authent_token}`, `${docker_snapshot_registry_host}`, `${docker_release_authent_token}` and `${docker_release_registry_host}` variables,
+* the user-defined `${MY_OWN_REGISTRY_TOKEN}` (:information_source: an authentication token can be obtained with command `echo "user:password" | base64` and then be stored as a masked GitLab CI/CD project variable).
+
+Example 2: Docker configuration file declared as a GitLab project variable of type [File](https://docs.gitlab.com/ee/ci/variables/#cicd-variable-types) with **dynamic variables replacement**:
+
+
+```json
+{
+    "auths": {
+        "$${docker_snapshot_registry_host}": {
+            "auth": "$${docker_release_authent_token}"
+        },
+        "$${docker_release_registry_host}": {
+            "auth": "$${docker_snapshot_authent_token}"
+        },
+        "my-readonly-repo-to-pull": {
+            "auth": "ZG9ja2VyZHVkZTpnb3RjaGEh"
+        }
+    }
+}
+```
+
+This file uses:
+
+* template-managed `${docker_snapshot_authent_token}`, `${docker_snapshot_registry_host}`, `${docker_release_authent_token}` and `${docker_release_registry_host}` variables (:warning: mind the double `$$` to prevent GitLab from [trying to evaluate the variable](https://docs.gitlab.com/ee/ci/variables/index.html#use-the--character-in-variables)),
+* the user-defined authentication may be inlined as a GitLab project variable is a place safe enough to store secrets.
+
 ## Multi Dockerfile support
 
 This template supports building multiple Docker images from a single Git repository.

kicker.json

@@ -34,6 +34,12 @@
       "description": "The Docker [context path](https://docs.docker.com/engine/reference/commandline/build/#build-with-path) (working directory) - _only set if you want a context path different from the Dockerfile location_",
       "advanced": true
     },
+    {
+      "name": "DOCKER_CONFIG_FILE",
+      "description": "Path to the [Docker configuration file](https://docs.docker.com/engine/reference/commandline/cli/#sample-configuration-file) (JSON)",
+      "default": ".docker/config.json",
+      "advanced": true
+    },
     {
       "name": "DOCKER_SNAPSHOT_IMAGE",
       "description": "Docker snapshot image",

templates/gitlab-ci-docker.yml

@@ -54,6 +54,7 @@ variables:
   # for retro-compatibility (deprecated & undocumented)
   DOCKER_DOCKERFILE_PATH: "."
   DOCKER_FILE: "$DOCKER_DOCKERFILE_PATH/Dockerfile"
+  DOCKER_CONFIG_FILE: ".docker/config.json"
 
   # When testing a Docker Health (test stage), how long (in seconds) wait for the HealthCheck status (https://docs.docker.com/engine/reference/builder/#healthcheck)
   DOCKER_HEALTHCHECK_TIMEOUT: "60"
@@ -303,23 +304,35 @@ stages:
     docker info > /dev/null 2>&1
   }
 
+  function awkenvsubst() {
+    awk '{while(match($0,"[$]{[^}]*}")) {var=substr($0,RSTART+2,RLENGTH -3);val=ENVIRON[var];gsub(/["\\]/,"\\\\&", val);gsub("\n", "\\n", val);gsub("\r", "\\r", val);gsub("[$]{"var"}",val)}}1'
+  }
+
   function configure_registries_auth() {
     docker_snapshot_authent_token=$(echo -n "${DOCKER_REGISTRY_SNAPSHOT_USER:-${DOCKER_REGISTRY_USER:-$CI_REGISTRY_USER}}:${DOCKER_REGISTRY_SNAPSHOT_PASSWORD:-${DOCKER_REGISTRY_PASSWORD:-$CI_REGISTRY_PASSWORD}}" | base64 | tr -d '\n')
     docker_snapshot_registry_host=$(echo "$DOCKER_SNAPSHOT_IMAGE" | cut -d/ -f1)
+    export docker_snapshot_authent_token
+    export docker_snapshot_registry_host
 
     docker_release_authent_token=$(echo -n "${DOCKER_REGISTRY_RELEASE_USER:-${DOCKER_REGISTRY_USER:-$CI_REGISTRY_USER}}:${DOCKER_REGISTRY_RELEASE_PASSWORD:-${DOCKER_REGISTRY_PASSWORD:-$CI_REGISTRY_PASSWORD}}" | base64 | tr -d '\n')
     docker_release_registry_host=$(echo "$DOCKER_RELEASE_IMAGE" | cut -d/ -f1)
+    export docker_release_authent_token
+    export docker_release_registry_host
 
     docker_snapshot_config_json=$(echo -n "{\"auths\":{\"$docker_snapshot_registry_host\":{\"auth\":\"$docker_snapshot_authent_token\"},\"HttpHeaders\":{\"User-Agent\":\"$USER_AGENT\"}}}")
     docker_release_config_json=$(echo -n "{\"auths\":{\"$docker_release_registry_host\":{\"auth\":\"$docker_release_authent_token\"},\"HttpHeaders\":{\"User-Agent\":\"$USER_AGENT\"}}}")
 
-    # Create the configuration file for Docker
+    # Create the configuration file for Docker and Kaniko
     mkdir -p /root/.docker
-    echo "${docker_snapshot_config_json}" > /root/.docker/config.json
-
-    # Create the configuration file for Kaniko
     mkdir -p /kaniko/.docker
+    if [ -f "${DOCKER_CONFIG_FILE}" ]
+    then
+      awkenvsubst < "${DOCKER_CONFIG_FILE}" > /root/.docker/config.json
+      awkenvsubst < "${DOCKER_CONFIG_FILE}" > /kaniko/.docker/config.json
+    else
+      echo "${docker_snapshot_config_json}" > /root/.docker/config.json
       echo "${docker_snapshot_config_json}" > /kaniko/.docker/config.json
+    fi
 
     # Create the configuration file for Skopeo
     mkdir -p /skopeo/.docker