-
Road Surfer authored
Signed-off-by:
Jason "RoadSurfer" Irwin <jasonirwin73+gitlab@gmail.com>
Road Surfer authoredSigned-off-by:
Jason "RoadSurfer" Irwin <jasonirwin73+gitlab@gmail.com>
GitLab CI template for Docker
This project implements a GitLab CI/CD template to build, check and inspect your containers with Docker.
Usage
This template can be used both as a CI/CD component
or using the legacy include:project
syntax.
Use as a CI/CD component
Add the following to your .gitlab-ci.yml
:
include:
# 1: include the component
- component: $CI_SERVER_FQDN/to-be-continuous/docker/gitlab-ci-docker@5.11.0
# 2: set/override component inputs
inputs:
build-tool: buildah # ⚠ this is only an example
Use as a CI/CD template (legacy)
Add the following to your .gitlab-ci.yml
:
include:
# 1: include the template
- project: 'to-be-continuous/docker'
ref: '5.11.0'
file: '/templates/gitlab-ci-docker.yml'
variables:
# 2: set/override template variables
DOCKER_BUILD_TOOL: buildah # ⚠ this is only an example
Understanding the Docker template
The template supports following ways of building container images:
- The former Docker-in-Docker (DinD) technique, that was widely used for years because of no other alternative, but that is now commonly recognized to have significant security issues (read this post for more info),
- Or using kaniko, an open-source, daemonless tool from Google for building Docker images, and that solves Docker-in-Docker security issues (and also speeds-up build times).
- Or using buildah, an open-source, daemonless tool backed by RedHat for building Docker images, and that solves Docker-in-Docker security issues (and also speeds-up build times), and can also be configured to run rootless.
By default, the template uses the kaniko way, but you may
select an alternate build tool by using the DOCKER_BUILD_TOOL
variable (see below).
Global variables
The Docker template uses some global configuration used throughout all jobs.
Input / Variable | Description | Default value |
---|---|---|
build-tool / DOCKER_BUILD_TOOL
|
The build tool to use for building container image, possible values are kaniko , buildah or dind
|
kaniko |
kaniko-image / DOCKER_KANIKO_IMAGE
|
The image used to run kaniko - for kaniko build only
|
gcr.io/kaniko-project/executor:debug (use debug images for GitLab) |
buildah-image / DOCKER_BUILDAH_IMAGE
|
The image used to run buildah - for buildah build only
|
quay.io/buildah/stable |
image / DOCKER_IMAGE
|
The Docker image used to run the docker client (see full list) - for Docker-in-Docker build only | registry.hub.docker.com/library/docker:latest |
dind-image / DOCKER_DIND_IMAGE
|
The Docker image used to run the Docker daemon (see full list) - for Docker-in-Docker build only | registry.hub.docker.com/library/docker:dind |
file / DOCKER_FILE
|
The path to your Dockerfile
|
Dockerfile |
context-path / DOCKER_CONTEXT_PATH
|
The Docker context path (working directory) | none only set if you want a context path different from the Dockerfile location |
In addition to this, the template supports standard Linux proxy variables:
Input / Variable | Description | Default value |
---|---|---|
http_proxy |
Proxy used for http requests | none |
https_proxy |
Proxy used for https requests | none |
no_proxy |
List of comma-separated hosts/host suffixes | none |
Images
For each Dockerfile, the template builds an image that may be pushed as two distinct images, depending on a certain workflow:
- snapshot: the image is first built from the Dockerfile and then pushed to some Docker registry as the snapshot image. It can be seen as the raw result of the build, but still untested and unreliable.
-
release: once the snapshot image has been thoroughly tested (both by
package-test
stage jobs and/oracceptance
stage jobs after being deployed to some server), then the image is pushed one more time as the release image. This second push can be seen as the promotion of the snapshot image being now tested and reliable.
In practice:
- the snapshot image is always pushed by the template (pipeline triggered by a Git tag or commit on any branch),
- the release image is only pushed:
- on a pipeline triggered by a Git tag,
- on a pipeline triggered by a Git commit on
master
.
The snapshot and release images are defined by the following variables:
Input / Variable | Description | Default value |
---|---|---|
snapshot-image / DOCKER_SNAPSHOT_IMAGE
|
Docker snapshot image | $CI_REGISTRY_IMAGE/snapshot:$CI_COMMIT_REF_SLUG |
release-image / DOCKER_RELEASE_IMAGE
|
Docker release image | $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME |
As you can see, the Docker template is configured by default to use the GitLab container registry. You may perfectly override this and use another Docker registry, but be aware of a few things:
- the
DOCKER_SNAPSHOT_IMAGE
requires a Docker registry that allows tag overwrite, - the
DOCKER_RELEASE_IMAGE
may use a Docker registry that doesn't allow tag overwrite, but:- you should avoid overwriting a Git tag (at it will obviously fail while trying to (re)push the Docker image),
- you have to deactivate publish on
main
(ormaster
) branch by setting the$DOCKER_PROD_PUBLISH_STRATEGY
variable tonone
(as it would lead to themain
tag being overwritten).
Registries and credentials
As seen in the previous chapter, the Docker template uses by default the GitLab registry to push snapshot and release images.
Thus it makes use of credentials provided by GitLab itself to login (CI_REGISTRY_USER
/ CI_REGISTRY_PASSWORD
).
But when using other registry(ies), you'll have also to configure appropriate Docker credentials.
Using the same registry for snapshot and release
If you use the same registry for both snapshot and release images, you shall use the following configuration
variables:
Input / Variable | Description |
---|---|
DOCKER_REGISTRY_USER
|
Docker registry username for image registry |
DOCKER_REGISTRY_PASSWORD
|
Docker registry password for image registry |
Using different registries for snapshot and release
If you use different registries for snapshot and release images, you shall use separate configuration variables:
Input / Variable | Description |
---|---|
DOCKER_REGISTRY_SNAPSHOT_USER
|
Docker registry username for snapshot image registry |
DOCKER_REGISTRY_SNAPSHOT_PASSWORD
|
Docker registry password for snapshot image registry |
DOCKER_REGISTRY_RELEASE_USER
|
Docker registry username for release image registry |
DOCKER_REGISTRY_RELEASE_PASSWORD
|
Docker registry password for release image registry |
Setting your own Docker configuration file (advanced)
There might be cases where you need to provide the complete Docker configuration file:
- need to declare authentication credentials for other registries than the 2 predefined ones (snapshot & release),
- need to declare a credentials store (ex: in order to publish to Amazon ECR with Kaniko for instance),
- need to declare proxies,
- ...
If you are in one of those cases, you will need to use the DOCKER_CONFIG_FILE
variable, expected to declare the path to your custom Docker configuration file (JSON). You may:
- leave the default value (
.docker/config.json
) or override it to some alternate location in your project repository and create the file without any secret in it using our dynamic variables replacement (see below), - or override it as a GitLab project variable of type File, possibly inlining your secret credentials in it.
Input / Variable | Description | Default value |
---|---|---|
config-file / DOCKER_CONFIG_FILE
|
Path to the Docker configuration file (JSON) | .docker/config.json |
Moreover, this file supports dynamic environment variables replacement.
That means it may contain references to other environment variables (in the format ${variable_name}
) that will be dynamically replaced
by the template before evaluation.
In addition to you own defined variables, you may use the following variables (provided and managed by the template):
-
${docker_snapshot_authent_token}
: the authentication token required by the snapshot registry (computed from configuredDOCKER_REGISTRY_SNAPSHOT_USER
/DOCKER_REGISTRY_SNAPSHOT_PASSWORD
variables) -
${docker_snapshot_registry_host}
: the snapshot registry host (based on the configuredDOCKER_SNAPSHOT_IMAGE
variable) -
${docker_release_authent_token}
: the authentication token required by the release registry (computed from configuredDOCKER_REGISTRY_RELEASE_USER
/DOCKER_REGISTRY_RELEASE_PASSWORD
variables) -
${docker_release_registry_host}
: the release registry host (based on the configuredDOCKER_RELEASE_IMAGE
variable)
Example 1: Docker configuration file inlined in the project repository (.docker/config.json
) with dynamic variables replacement:
{
"auths": {
"${docker_snapshot_registry_host}": {
"auth": "${docker_release_authent_token}"
},
"${docker_release_registry_host}": {
"auth": "${docker_snapshot_authent_token}"
},
"my-readonly-repo-to-pull": {
"auth": "${MY_OWN_REGISTRY_TOKEN}"
}
}
}
This file uses:
- template-managed
${docker_snapshot_authent_token}
,${docker_snapshot_registry_host}
,${docker_release_authent_token}
and${docker_release_registry_host}
variables, - the user-defined
${MY_OWN_REGISTRY_TOKEN}
(ℹ️ an authentication token can be obtained with commandecho "user:password" | base64
and then be stored as a masked GitLab CI/CD project variable).
Example 2: Docker configuration file declared as a GitLab project variable of type File with dynamic variables replacement:
{
"auths": {
"$${docker_snapshot_registry_host}": {
"auth": "$${docker_release_authent_token}"
},
"$${docker_release_registry_host}": {
"auth": "$${docker_snapshot_authent_token}"
},
"my-readonly-repo-to-pull": {
"auth": "ZG9ja2VyZHVkZTpnb3RjaGEh"
}
}
}
This file uses:
- template-managed
${docker_snapshot_authent_token}
,${docker_snapshot_registry_host}
,${docker_release_authent_token}
and${docker_release_registry_host}
variables (⚠️ mind the double$$
to prevent GitLab from trying to evaluate the variable), - the user-defined authentication may be inlined as a GitLab project variable is a place safe enough to store secrets.
Multi Dockerfile support
This template supports building multiple Docker images from a single Git repository.
You can define the images to build using the parallel matrix jobs
pattern inside the .docker-base
job (this is the top parent job of all Docker template jobs).
Since each job in the template extends this base job, the pipeline will produce one job instance per image to build. You can independently configure each instance of these jobs by redefining the variables described throughout this documentation.
For example, if you want to build two Docker images, you must specify where the Dockerfiles are located and where the
resulting images will be stored.
You can do so by adding a patch to the .docker-base
job in your .gitlab-ci.yml
file so that it looks like this:
.docker-base:
parallel:
matrix:
- DOCKER_FILE: "front/Dockerfile"
DOCKER_SNAPSHOT_IMAGE: "$CI_REGISTRY_IMAGE/front:$CI_COMMIT_REF_SLUG"
DOCKER_RELEASE_IMAGE: "$CI_REGISTRY_IMAGE/front:$CI_COMMIT_REF_NAME"
- DOCKER_FILE: "back/Dockerfile"
DOCKER_SNAPSHOT_IMAGE: "$CI_REGISTRY_IMAGE/back:$CI_COMMIT_REF_SLUG"
DOCKER_RELEASE_IMAGE: "$CI_REGISTRY_IMAGE/back:$CI_COMMIT_REF_NAME"
If you need to redefine a variable with the same value for all your Dockerfiles, you can just declare this variable as a global variable. For example, if you want to build all your images using buildah
, you can simply define the DOCKER_BUILD_TOOL
variable as a global variable with value buildah
:
variables:
DOCKER_BUILD_TOOL: "buildah"
Secrets management
Here are some advices about your secrets (variables marked with a
- Manage them as project or group CI/CD variables:
- In case a secret contains characters that prevent it from being masked,
simply define its value as the Base64 encoded value prefixed with
@b64@
: it will then be possible to mask it and the template will automatically decode it prior to using it. - Don't forget to escape special characters (ex:
$
->$$
).
Jobs
docker-hadolint
job
This job performs a Lint on your Dockerfile
.
It is bound to the build
stage, and uses the following variables:
Input / Variable | Description | Default value |
---|---|---|
hadolint-disabled / DOCKER_HADOLINT_DISABLED
|
Set to true to disable Hadolint |
(none: enabled by default) |
hadolint-image / DOCKER_HADOLINT_IMAGE
|
The Hadolint image | registry.hub.docker.com/hadolint/hadolint:latest-alpine |
hadolint-args / DOCKER_HADOLINT_ARGS
|
Additional hadolint arguments |
(none) |
In case you have to disable some rules, either add --ignore XXXX
to the DOCKER_HADOLINT_ARGS
variable or create a Hadolint configuration file named hadolint.yaml
at the root of your repository.
You can also use inline ignores in your Dockerfile:
# hadolint ignore=DL3006
FROM ubuntu
# hadolint ignore=DL3003,SC1035
RUN cd /tmp && echo "hello!"
In addition to a textual report in the console, this job produces the following reports, kept for one day:
Report | Format | Usage |
---|---|---|
reports/docker-hadolint-*.native.json |
native hadolint test report (json) |
DefectDojo integration This report is generated only if DefectDojo template is detected |
reports/docker-hadolint-*.codeclimate.json |
hadolint (GitLab) codeclimate format | GitLab integration |
docker-*-build
jobs
This job builds the image and publishes it to the snapshot repository.
It is bound to the package-build
stage, and uses the following variables:
Input / Variable | Description | Default value |
---|---|---|
build-args / DOCKER_BUILD_ARGS
|
Additional docker/kaniko/buildah build arguments |
(none) |
registry-mirror / DOCKER_REGISTRY_MIRROR
|
URL of a Docker registry mirror to use during the image build (instead of default https://index.docker.io ) kaniko and dind options only |
(none) |
container-registries-config-file / CONTAINER_REGISTRIES_CONFIG_FILE
|
The registries.conf configuration to be usedbuildah build only |
(none) |
metadata / DOCKER_METADATA
|
Additional docker build /kaniko arguments to set label |
OCI Image Format Specification |
kaniko-snapshot-image-cache / KANIKO_SNAPSHOT_IMAGE_CACHE
|
Snapshot image repository that will be used to store cached layers (leave empty to use default: snapshot image repository + /cache )kaniko build only |
none (default cache path) |
build-cache-disabled / DOCKER_BUILD_CACHE_DISABLED
|
Set to true to disable the build cache.Cache can typically be disabled when there is a network latency between the container registry and the runner. |
none (i.e cache enabled) |
This job produces output variables that are propagated to downstream jobs (using dotenv artifacts):
Input / Variable | Description | Example |
---|---|---|
docker_image |
snapshot image name with tag | registry.gitlab.com/acme/website/snapshot:main |
docker_image_digest |
snapshot image name with digest (no tag) | registry.gitlab.com/acme/website/snapshot@sha256:b7914a91... |
docker_repository |
snapshot image bare repository (no tag nor digest) | registry.gitlab.com/acme/website/snapshot |
docker_tag |
snapshot image tag | main |
docker_digest |
snapshot image digest | sha256:b7914a91... |
They may be freely used in downstream jobs (for instance to deploy the upstream built Docker image, whatever the branch or tag).
If you want to use GitLab CI variables or any other variable in your Dockerfile, you can add them to DOCKER_BUILD_ARGS
like so:
DOCKER_BUILD_ARGS: "--build-arg CI_PROJECT_URL --build-arg MY_VAR='MY_VALUE'"
These variables will then be available for use in your Dockerfile:
FROM scratch
ARG CI_PROJECT_URL
ARG MY_VAR
LABEL name="my-project" \
description="My Project: $MY_VAR" \
url=$CI_PROJECT_URL \
maintainer="my-project@acme.com"
Default value for DOCKER_METADATA
supports a subset of the OCI Image Format Specification for labels and use GitLab CI pre-defined variables to guess the value as follow :
Label | GitLab CI pre-defined variable |
---|---|
org.opencontainers.image.url |
$CI_PROJECT_URL |
org.opencontainers.image.source |
$CI_PROJECT_URL |
org.opencontainers.image.title |
$CI_PROJECT_PATH |
org.opencontainers.image.ref.name |
$CI_COMMIT_REF_NAME |
org.opencontainers.image.revision |
$CI_COMMIT_SHA |
org.opencontainers.image.created |
$CI_JOB_STARTED_AT |
Note that spaces are currently not supported by Kaniko. Therefore, title couldn't be CI_PROJECT_TITLE
.