GitLab CI template for Kubernetes (k8s)

This project implements a GitLab CI/CD template to deploy your application to a Kubernetes platform using declarative configuration or Kustomize.

Usage

This template can be used both as a CI/CD component or using the legacy include:project syntax.

Use as a CI/CD component

Add the following to your .gitlab-ci.yml:

include:
  # 1: include the component
  - component: $CI_SERVER_FQDN/to-be-continuous/kubernetes/gitlab-ci-k8s@6.3.0
    # 2: set/override component inputs
    inputs:
      # ⚠ this is only an example
      kubectl-image: registry.hub.docker.com/bitnami/kubectl:1.26
      base-app-name: wonderapp
      review-space: myapp-nonprod
      staging-space: myapp-nonprod
      prod-space: myapp-prod

Use as a CI/CD template (legacy)

Add the following to your .gitlab-ci.yml:

include:
  # 1: include the template
  - project: 'to-be-continuous/kubernetes'
    ref: '6.3.0'
    file: '/templates/gitlab-ci-k8s.yml'

variables:
  # 2: set/override template variables
  # ⚠ this is only an example
  K8S_KUBECTL_IMAGE: registry.hub.docker.com/bitnami/kubectl:1.26
  K8S_BASE_APP_NAME: wonderapp
  K8S_REVIEW_SPACE: myapp-nonprod
  K8S_STAGING_SPACE: myapp-nonprod
  K8S_PROD_SPACE: myapp-prod

Understand

This chapter introduces key notions and principle to understand how this template works.

Managed deployment environments

This template implements continuous delivery/continuous deployment for projects hosted on Kubernetes platforms.

It allows you to manage automatic deployment & cleanup of standard predefined environments. Each environment can be enabled/disabled by configuration. If you're not satisfied with predefined environments and/or their associated Git workflow, you may implement you own environments and workflow, by reusing/extending the base (hidden) jobs. This is advanced usage and will not be covered by this documentation.

The following chapters present the managed predefined environments and their associated Git workflow.

Review environments

The template supports review environments: those are dynamic and ephemeral environments to deploy your ongoing developments (a.k.a. feature or topic branches).

When enabled, it deploys the result from upstream build stages to a dedicated and temporary environment. It is only active for non-production, non-integration branches.

It is a strict equivalent of GitLab's Review Apps feature.

It also comes with a cleanup job (accessible either from the environments page, or from the pipeline view).

Integration environment

If you're using a Git Workflow with an integration branch (such as Gitflow), the template supports an integration environment.

When enabled, it deploys the result from upstream build stages to a dedicated environment. It is only active for your integration branch (develop by default).

Production environments

Lastly, the template supports 2 environments associated to your production branch (master or main by default):

• a staging environment (an iso-prod environment meant for testing and validation purpose),
• the production environment.

You're free to enable whichever or both, and you can also choose your deployment-to-production policy:

• continuous deployment: automatic deployment to production (when the upstream pipeline is successful),
• continuous delivery: deployment to production can be triggered manually (when the upstream pipeline is successful).

Supported authentication methods

The Kubernetes template supports 3 ways of login/accessing your Kubernetes cluster(s):

1. Using GitLab agents with the CI/CD workflow: when enabled, the template automatically retrieves and uses your Kubernetes cluster configuration (KUBECONFIG env), ⚠️ don't forget to set the KUBE_CONTEXT variable (to path/to/agent/project:agent-name) as stated in the documentation.
2. By defining an explicit kubeconfig from env (either file or yaml content),
3. By defining explicit kubeconfig exploded parameters from env: server url, server certificate authority and user token.

Deployment context variables

In order to manage the various deployment environments, this template provides a couple of dynamic variables that you might use in your hook scripts, deployment manifests and other deployment resources:

• ${environment_type}: the current deployment environment type (review, integration, staging or production)
• ${environment_name}: a generated application name to use for the current deployment environment (ex: myproject-review-fix-bug-12 or myproject-staging) - details below
• ${environment_name_ssc}: the above application name in SCREAMING_SNAKE_CASE format (ex: MYPROJECT_REVIEW_FIX_BUG_12 or MYPROJECT_STAGING)
• ${k8s_namespace}: the Kubernetes namespace currently used for deployment/cleanup

Generated environment name

The ${environment_name} variable is generated to designate each deployment environment with a unique and meaningful application name. By construction, it is suitable for inclusion in DNS, URLs, Kubernetes labels... It is built from:

• the application base name (defaults to $CI_PROJECT_NAME but can be overridden globally and/or per deployment environment - see configuration variables)
• GitLab predefined $CI_ENVIRONMENT_SLUG variable (sluggified name, truncated to 24 characters)

The ${environment_name} variable is then evaluated as:

• <app base name> for the production environment
• <app base name>-$CI_ENVIRONMENT_SLUG for all other deployment environments
• 💡 ${environment_name} can also be overriden per environment with the appropriate configuration variable

Examples (with an application's base name myapp):

Supported deployment methods

The Kubernetes template supports three techniques to deploy your code:

1. script-based deployment,
2. template-based deployment using raw Kubernetes manifests (with variables substitution),
3. template-based deployment using Kustomization files.

1: script-based deployment

In this mode, you only have to provide a shell script that fully implements the deployment using the kubectl CLI.

The deployment script is searched as follows:

1. look for a specific k8s-deploy-$environment_type.sh in the $K8S_SCRIPTS_DIR directory in your project (e.g. k8s-deploy-staging.sh for staging environment),
2. if not found: look for a default k8s-deploy.sh in the $K8S_SCRIPTS_DIR directory in your project,
3. if not found: the GitLab CI template assumes you're using the template-based deployment policy.

⚠️ k8s-deploy-$environment_type.sh or k8s-deploy.sh needs to be executable, you can add flag execution with: git update-index --chmod=+x k8s-deploy.sh

2: template-based deployment

In this mode, you have to provide a Kubernetes deployment file in your project structure, and let the GitLab CI template kubectl apply it.

The template processes the following steps:

1. optionally executes the k8s-pre-apply.sh script in your project to perform specific environment pre-initialization (for e.g. create required services),
2. looks for your Kubernetes deployment file, performs variables substitution and kubectl apply it,
   1. look for a specific deployment-$environment_type.yml in your project (e.g. deployment-staging.yml for staging environment),
   2. fallbacks to default deployment.yml.
3. optionally executes the k8s-post-apply.sh script in your project to perform specific environment post-initialization stuff,

⚠️ k8s-pre-apply.sh or k8s-post-apply.sh needs to be executable, you can add flag execution with: git update-index --chmod=+x k8s-pre-apply.sh

3: Kustomize-based deployment

In this mode, you have to provide a Kustomization file in your project structure, and let the template kubectl apply it.

The template processes the following steps:

1. optionally executes the k8s-pre-apply.sh script in your project to perform specific environment pre-initialization (for e.g. create required services),
2. looks for your Kustomization file, performs variables substitution and kubectl apply it,
   1. looks for an environment-specific overlay file ./$environment_type/kustomization.yaml (e.g. ./staging/kustomization.yaml for staging environment),
   2. fallbacks to default kustomization.yaml.
3. optionally executes the k8s-post-apply.sh script in your project to perform specific environment post-initialization stuff,

⚠️ k8s-pre-apply.sh or k8s-post-apply.sh needs to be executable, you can add flag execution with: git update-index --chmod=+x k8s-pre-apply.sh

Variables substitution is performed by the deprecated feature from Kustomize based on configMapGenerator, using a non-valuated variable from a config map.

Readiness script

After deployment (either script-based or template-based), the GitLab CI template optionally executes the k8s-readiness-check.sh hook script to wait & check for the application to be ready (if not found, the template assumes the application was successfully started).

⚠️ k8s-readiness-check.sh needs to be executable, you can add flag execution with: git update-index --chmod=+x k8s-pre-apply.sh

Supported cleanup methods

The Kubernetes template supports three techniques to destroy an environment (actually only review environments):

1. script-based deployment,
2. template-based deployment using raw Kubernetes manifests (with variables substitution),
3. template-based deployment using Kustomization files.

1: script-based cleanup

In this mode, you only have to provide a shell script that fully implements the environment cleanup using the kubectl CLI.

The a deployment script is searched as follows:

1. look for a specific k8s-cleanup-$environment_type.sh in the $K8S_SCRIPTS_DIR directory in your project (e.g. k8s-cleanup-staging.sh for staging environment),
2. if not found: look for a default k8s-cleanup.sh in the $K8S_SCRIPTS_DIR directory in your project,
3. if not found: the GitLab CI template assumes you're using the template-based cleanup policy.

⚠️ k8s-cleanup-$environment_type.sh or k8s-cleanup.sh needs to be executable, you can add flag execution with: git update-index --chmod=+x k8s-cleanup.sh

TIP: a nice way to implement environment cleanup is to declare the label app=${environment_name} on every Kubernetes object associated to your environment. Then environment cleanup can be implemented very easily with command kubectl delete all,pvc,secret,ingress -l "app=${environment_name}"

2: template-based cleanup

In this mode, you mainly let Kubernetes delete all objects from your Kubernetes deployment file.

The template processes the following steps:

1. optionally executes the k8s-pre-cleanup.sh script in your project to perform specific environment pre-cleanup stuff,
2. looks for your Kubernetes deployment file, performs variables substitution and kubectl delete it,
   1. look for a specific deployment-$environment_type.yml in your project (e.g. deployment-staging.yml for staging environment),
   2. fallbacks to default deployment.yml.
3. optionally executes the k8s-post-cleanup.sh script in your project to perform specific environment post-cleanup (for e.g. delete bound services).

⚠️ k8s-pre-cleanup.sh or k8s-post-cleanup.sh needs to be executable, you can add flag execution with: git update-index --chmod=+x k8s-pre-cleanup.sh

3: Kustomize-based cleanup

In this mode, you mainly let Kubernetes delete all objects from your Kustomization file(s).

The template processes the following steps:

1. optionally executes the k8s-pre-cleanup.sh script in your project to perform specific environment pre-cleanup stuff,
2. looks for your Kustomization file, performs variables substitution and kubectl delete it,
   1. looks for an environment-specific overlay file ./$environment_type/kustomization.yaml (e.g. ./staging/kustomization.yaml for staging environment),
   2. fallbacks to default kustomization.yaml.
3. optionally executes the k8s-post-cleanup.sh script in your project to perform specific environment post-cleanup (for e.g. delete bound services).

⚠️ k8s-pre-cleanup.sh or k8s-post-cleanup.sh needs to be executable, you can add flag execution with: git update-index --chmod=+x k8s-pre-cleanup.sh

Cleanup job limitations

When using this template, you have to be aware of one limitation (bug) with the cleanup job.

By default, the cleanup job triggered automatically on branch deletion will fail due to not being able to fetch the Git branch prior to executing the job (obviously because the branch was just deleted). This is pretty annoying, but as you may see above, deleting an env may require scripts from the project...

In a future version, the template will rely on Kustomize and will be able to delete an entire environment using the app label. In the meantime we're just sorry about this bug, but for now there is not much we can do:

• remind to delete your review env manually before deleting the branch
• otherwise you'll have to do it afterwards from your computer (using kubectl CLI) or from the some k8s web console.

Using variables

You have to be aware that your deployment (and cleanup) scripts have to be able to cope with various environments, each with different application names, exposed routes, settings, ... Part of this complexity can be handled by the lookup policies described above (ex: one script/manifest per env) and also by using available environment variables:

1. deployment context variables provided by the template:
   • ${environment_type}: the current environment type (review, integration, staging or production)
   • ${environment_name}: the application name to use for the current environment (ex: myproject-review-fix-bug-12 or myproject-staging)
   • ${environment_name_ssc}: the application name in SCREAMING_SNAKE_CASE format (ex: MYPROJECT_REVIEW_FIX_BUG_12 or MYPROJECT_STAGING)
   • ${k8s_namespace}: the Kubernetes namespace currently used for deployment/cleanup
   • ${hostname}: the environment hostname, extracted from the current environment url (after late variable expansion - see below)
2. any GitLab CI variable
3. any custom variable (ex: ${SECRET_TOKEN} that you have set in your project CI/CD variables)

While your scripts may simply use any of those variables, your Kubernetes and Kustomize resources can use variable substitution with the syntax ${VARIABLE_NAME}. Each of those patterns will be dynamically replaced in your resources by the template right before using it.

You can prevent any line from being processed by appending # nosubst at the end of the line. For instance in the following example, ${REMOTE_SERVICE_NAME} won't be replaced by its environment value during GitLab job execution:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: ${APPLICATION_NAME}
  name: ${APPLICATION_NAME}
data:
  application.yml: |
    remote:
      some-service:
          name: '${REMOTE_SERVICE_NAME}' # nosubst

⚠️ In order to be properly replaced, curly braces are mandatory (ex: ${MYVAR} and not $MYVAR). Moreover, multiline variables must be surrounded by double quotes (").

Example:

[...]
containers:
- name: restaurant-app
  env:
  # multiline variable
  - name: MENU
    value: "${APP_MENU}"

Environments URL management

The K8S template supports two ways of providing your environments url:

• a static way: when the environments url can be determined in advance, probably because you're exposing your routes through a DNS you manage,
• a dynamic way: when the url cannot be known before the deployment job is executed.

The static way can be implemented simply by setting the appropriate configuration variable(s) depending on the environment (see environments configuration chapters):

• $K8S_ENVIRONMENT_URL to define a default url pattern for all your envs,
• $K8S_REVIEW_ENVIRONMENT_URL, $K8S_INTEG_ENVIRONMENT_URL, $K8S_STAGING_ENVIRONMENT_URL and $K8S_PROD_ENVIRONMENT_URL to override the default.

ℹ️ Each of those variables support a late variable expansion mechanism with the %{somevar} syntax, allowing you to use any dynamically evaluated variables such as ${environment_name}.

Example:

variables:
  K8S_BASE_APP_NAME: "wonderapp"
  # global url for all environments
  K8S_ENVIRONMENT_URL: "https://%{environment_name}.nonprod.acme.domain"
  # override for prod (late expansion of $K8S_BASE_APP_NAME not needed here)
  K8S_PROD_ENVIRONMENT_URL: "https://$K8S_BASE_APP_NAME.acme.domain"
  # override for review (using separate resource paths)
  K8S_REVIEW_ENVIRONMENT_URL: "https://wonderapp-review.nonprod.acme.domain/%{environment_name}"

To implement the dynamic way, your deployment script shall simply generate a environment_url.txt file in the working directory, containing only the dynamically generated url. When detected by the template, it will use it as the newly deployed environment url.

Deployment output variables

Each deployment job produces output variables that are propagated to downstream jobs (using dotenv artifacts):

• $environment_type: set to the type of environment (review, integration, staging or production),
• $environment_name: the application name (see below),
• $environment_url: set to the environment URL (whether determined statically or dynamically).

Those variables may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).

You may also add and propagate your own custom variables, by pushing them to the kubernetes.env file in your deployment script or hook.

Configuration reference

Secrets management

Here are some advices about your secrets (variables marked with a 🔒):

1. Manage them as project or group CI/CD variables:
   • masked to prevent them from being inadvertently displayed in your job logs,
   • protected if you want to secure some secrets you don't want everyone in the project to have access to (for instance production secrets).
2. In case a secret contains characters that prevent it from being masked, simply define its value as the Base64 encoded value prefixed with @b64@: it will then be possible to mask it and the template will automatically decode it prior to using it.
3. Don't forget to escape special characters (ex: $ -> $$).

Global configuration

The Kubernetes template uses some global configuration used throughout all jobs.

Review environments configuration

Review environments are dynamic and ephemeral environments to deploy your ongoing developments (a.k.a. feature or topic branches).

They are disabled by default and can be enabled by setting the K8S_REVIEW_SPACE variable (see below).

Here are variables supported to configure review environments:

Integration environment configuration

The integration environment is the environment associated to your integration branch (develop by default).

It is disabled by default and can be enabled by setting the K8S_INTEG_SPACE variable (see below).

Here are variables supported to configure the integration environment:

Staging environment configuration

The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production branch (main or master by default).

It is disabled by default and can be enabled by setting the K8S_STAGING_SPACE variable (see below).

Here are variables supported to configure the staging environment:

Production environment configuration

The production environment is the final deployment environment associated with your production branch (main or master by default).

It is disabled by default and can be enabled by setting the K8S_PROD_SPACE variable (see below).

Here are variables supported to configure the production environment:

kube-score job

The Kubernetes template enables kube-score analysis of your Kubernetes object definitions. This job is mapped to the test stage and is active by default.

Here are its parameters:

Variants

Vault variant

This variant allows delegating your secrets management to a Vault server.

⚠️ Change default K8S_KUBECTL_IMAGE

The Vault variant requires curl or wget to retrieve secrets from the Vault server, which makes it incompatible with the default K8S_KUBECTL_IMAGE from Bitnami, as curl and wget are no longer part of it.

As a result, when using the Vault variant, you'll have to select a K8S_KUBECTL_IMAGE that - in addition to kubectl - contains curl or wget. For instance container-oc images (see example below).

Configuration

In order to be able to communicate with the Vault server, the variant requires the additional configuration parameters:

By default, the variant will authentifacte using a JWT ID token. To use AppRole instead the VAULT_ROLE_ID and VAULT_SECRET_ID should be defined as secret project variables.

Usage

Then you may retrieve any of your secret(s) from Vault using the following syntax:

@url@http://vault-secrets-provider/api/secrets/{secret_path}?field={field}

With:

Example

include:
  # main template
  - component: $CI_SERVER_FQDN/to-be-continuous/kubernetes/gitlab-ci-k8s@6.3.0
    inputs:
      # ⚠ oc-container image (includes required curl)
      kubectl-image: registry.hub.docker.com/docker.io/appuio/oc:v4.14
  # Vault variant
  - component: $CI_SERVER_FQDN/to-be-continuous/kubernetes/gitlab-ci-k8s-vault@6.3.0
    inputs:
      # audience claim for JWT
      vault-oidc-aud: "https://vault.acme.host"
      vault-base-url: "https://vault.acme.host/v1"

variables:
    # Secrets managed by Vault
    K8S_DEFAULT_KUBE_CONFIG: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/my-app/kubernetes/noprod?field=kube_config"
    K8S_PROD_KUBE_CONFIG: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/my-app/kubernetes/prod?field=kube_config"

Google Cloud variant

This variant uses Application Default Credentials through the GOOGLE_APPLICATION_CREDENTIALS variable using Workload Identity federation.

List of requirements before using this variant:

1. You must have a Workload Identity Federation Pool and Provider configured,
2. You must have a Service Account with the roles/iam.workloadIdentityUser IAM role granted to the Workload Identity principal matching your Gitlab project or group,
3. Optionally, you can set the GOOGLE_CLOUD_PROJECT template variable to define the default Google Cloud project.
4. You must have create a kubeconfig.yaml configuration which enable application default credentials for kubectl

The Gitlab documentation has some details about Workload Identity Federation integration.

This blog post about OIDC impersonation through Workload Identify Federation might also be of help.

Configuration

The variant requires the additional configuration parameters:

Example

With a common default GCP_OIDC_PROVIDER and GCP_OIDC_ACCOUNT configuration for non-prod environments, and a specific one for production:

include:
  # main template
  - component: $CI_SERVER_FQDN/to-be-continuous/kubernetes/gitlab-ci-k8s@6.3.0
  # Google Cloud variant
  - component: $CI_SERVER_FQDN/to-be-continuous/kubernetes/gitlab-ci-k8ss-gcp@6.3.0
    inputs:
      # common OIDC config for non-prod envs
      gcp-oidc-provider: "projects/<gcp_nonprod_proj_id>/locations/global/workloadIdentityPools/<pool_id>/providers/<provider_id>"
      gcp-oidc-account: "<name>@$<gcp_nonprod_proj_id>.iam.gserviceaccount.com"
      # specific OIDC config for prod
      gcp-prod-oidc-provider: "projects/<gcp_prod_proj_id>/locations/global/workloadIdentityPools/<pool_id>/providers/<provider_id>"
      gcp-prod-oidc-account: "<name>@$<gcp_prod_proj_id>.iam.gserviceaccount.com"