Skip to content
Snippets Groups Projects
Select Git revision
  • 2977b4091f6afa4f50ad19e2fd2575b4e55e9f68
  • master default protected
  • 1
  • 1.2
  • 1.2.2
  • 1.2.1
  • 1.2.0
  • 1.1
  • 1.1.0
  • 1.0
  • 1.0.2
  • 1.0.1
12 results

docker-compose

  • Clone with SSH
  • Clone with HTTPS
  • user avatar
    to be continuous bot authored
    2977b409
    History

    GitLab CI template for Docker Compose

    This project implements a GitLab CI/CD template to deploy your application with Docker Compose or Docker Swarm.

    Usage

    This template can be used both as a CI/CD component or using the legacy include:project syntax.

    Use as a CI/CD component

    Add the following to your .gitlab-ci.yml:

    include:
      # 1: include the component
      - component: $CI_SERVER_FQDN/to-be-continuous/docker-compose/gitlab-ci-docker-compose@1.2.0
        # 2: set/override component inputs
        inputs:
          # ⚠ this is only an example
          base-app-name: wonderapp
          review-docker-host: "ssh://user@192.168.64.5" # enable review env
          staging-docker-host: "ssh://user@192.168.64.6" # enable staging env
          prod-docker-host: "ssh://user@192.168.64.7" # enable production env

    Use as a CI/CD template (legacy)

    Add the following to your .gitlab-ci.yml:

    include:
      # 1: include the template
      - project: 'to-be-continuous/docker-compose'
        ref: '1.2.0'
        file: '/templates/gitlab-ci-docker-compose.yml'
    
    variables:
      # 2: set/override template variables
      # ⚠ this is only an example
      DCMP_BASE_APP_NAME: wonderapp
      DCMP_REVIEW_DOCKER_HOST: "ssh://user@192.168.64.5" # enable review env
      DCMP_STAGING_DOCKER_HOST: "ssh://user@192.168.64.6" # enable staging env
      DCMP_PROD_DOCKER_HOST: "ssh://user@192.168.64.7" # enable production env

    Understand

    This chapter introduces key notions and principle to understand how this template works.

    Managed deployment environments

    This template implements continuous delivery/continuous deployment for projects hosted on Docker Compose or Docker Swarm.

    It allows you to manage automatic deployment & cleanup of standard predefined environments. Each environment can be enabled/disabled by configuration. If you're not satisfied with predefined environments and/or their associated Git workflow, you may implement you own environments and workflow, by reusing/extending the base (hidden) jobs. This is advanced usage and will not be covered by this documentation.

    The following chapters present the managed predefined environments and their associated Git workflow.

    Review environments

    The template supports review environments: those are dynamic and ephemeral environments to deploy your ongoing developments (a.k.a. feature or topic branches).

    When enabled, it deploys the result from upstream build stages to a dedicated and temporary environment. It is only active for non-production, non-integration branches.

    It is a strict equivalent of GitLab's Review Apps feature.

    It also comes with a cleanup job (accessible either from the environments page, or from the pipeline view).

    Integration environment

    If you're using a Git Workflow with an integration branch (such as Gitflow), the template supports an integration environment.

    When enabled, it deploys the result from upstream build stages to a dedicated environment. It is only active for your integration branch (develop by default).

    Production environments

    Lastly, the template supports 2 environments associated to your production branch (master or main by default):

    • a staging environment (an iso-prod environment meant for testing and validation purpose),
    • the production environment.

    You're free to enable whichever or both, and you can also choose your deployment-to-production policy:

    • continuous deployment: automatic deployment to production (when the upstream pipeline is successful),
    • continuous delivery: deployment to production can be triggered manually (when the upstream pipeline is successful).

    Supported authentication methods

    The Docker Compose template supports deployment on remote Docker hosts, using dedicated variables:

    • DCMP_REVIEW_DOCKER_HOST, DCMP_INTEG_DOCKER_HOST, DCMP_STAGING_DOCKER_HOST and DCMP_PROD_DOCKER_HOST to both enable and configure the target Docker host for each environment (ex: ssh://user@192.168.64.5),
    • 🔒 DCMP_SSH_PRIVATE_KEY, 🔒 DCMP_REVIEW_SSH_PRIVATE_KEY, 🔒 DCMP_INTEG_SSH_PRIVATE_KEY, 🔒 DCMP_STAGING_SSH_PRIVATE_KEY and 🔒 DCMP_PROD_SSH_PRIVATE_KEY to provide the global or per env SSH private key (in case SSH authentication is required).

    Deployment context variables

    In order to manage the various deployment environments, this template provides a couple of dynamic variables that you might use in your hook scripts or Docker Compose files:

    • ${environment_type}: the current deployment environment type (review, integration, staging or production)
    • ${environment_name}: a generated application name to use for the current deployment environment (ex: myapp-review-fix-bug-12 or myapp-staging) - details below

    ℹ️ the ${environment_name} is used by the Docker Compose template as the Docker Compose project name or Docker Stack project name.

    Generated environment name

    The ${environment_name} variable is generated to designate each deployment environment with a unique and meaningful application name. By construction, it is suitable for inclusion in DNS, URLs, Kubernetes labels... It is built from:

    • the application base name (defaults to $CI_PROJECT_NAME but can be overridden globally and/or per deployment environment - see configuration variables)
    • GitLab predefined $CI_ENVIRONMENT_SLUG variable (slugified name, truncated to 24 characters)

    The ${environment_name} variable is then evaluated as:

    • <app base name> for the production environment
    • <app base name>-$CI_ENVIRONMENT_SLUG for all other deployment environments
    • 💡 ${environment_name} can also be overridden per environment with the appropriate configuration variable

    Examples (with an application's base name myapp):

    $environment_type Branch $CI_ENVIRONMENT_SLUG $environment_name
    review feat/blabla review-feat-bla-xmuzs6 myapp-review-feat-bla-xmuzs6
    integration develop integration myapp-integration
    staging main staging myapp-staging
    production main production myapp

    Deployment and cleanup

    The Docker Compose template requires you to provide:

    • a required Compose file that implements your application deployment and cleanup,
    • optional .override. Compose files defining common and/or per-env override,
    • optional dotenv files defining common and/or per-env configuration,
    • optional hook scripts (shell) to implement logic that can't be implemented with Docker Compose.

    Compose files lookup strategy

    Unless you have explicitly set the COMPOSE_FILES variable, the Docker Compose template will handle it and implement the following Compose file(s) lookup strategy:

    Lookup order Compose file Additional .override. files (optional)
    1. environment specific (compose-${environment_type}.yaml) environment specific override (compose-${environment_type}.override.yaml)
    2. default (compose.yaml) 1. default override (compose.override.yaml)
    2. environment specific override (compose-${environment_type}.override.yaml)

    ℹ️ Important:

    • Compose file base name docker-compose is also supported as an alternative to compose,
    • Compose file extension .yml is also supported as an alternative to .yaml,
    • .override. files must match the same base name (compose or docker-compose) and extension (yaml or yml) as the found Compose file.

    Examples with different combinations of files:

    Files in your project Compose files for Review Compose files for Integration Compose files for Staging Compose files for Production
    - compose.yaml compose.yaml
    - compose.yaml
    - compose-production.yaml
    compose.yaml compose-production.yaml
    - docker-compose.yml
    - docker-compose-production.override.yml
    docker-compose.yml docker-compose.yml + docker-compose-production.override.yml (merged)
    - compose.yml
    - compose.override.yml
    - compose-review.override.yml
    - compose-production.yml
    compose.yml + compose.override.yml + compose-review.override.yml (merged) compose.yml+ compose.override.yml (merged) compose-production.yml

    Dotenv files lookup strategy

    Unless you have explicitly set the COMPOSE_ENV_FILES variable, the Docker Compose template will handle it and implement the following dotenv files lookup strategy:

    1. a .env file defining defaults for all environments,
    2. a ${environment_type}.env file that might redefine or override defaults for a specific environment (e.g. staging.env).

    Docker Swarm

    Setting the DCMP_CMD variable to docker stack allows targetting a cluster hence multiple hosts to deploy your containers. It uses the same configuration and dotenv files lookup strategy as the classic Docker Compose workflow.

    Deployment

    The deployment is processed as follows by the template:

    1. optionally executes the pre-compose-up.sh script found in your project to perform pre-deployment stuff (for e.g. create required services),
    2. runs docker-compose up or docker stack deploy based on the DCMP_CMD variable,
    3. optionally executes the post-compose-up.sh script found in your project to perform post-deployment stuff,

    ℹ️ Important:

    • Compose files, dotenv files and hook scripts are searched in the $DCMP_SCRIPTS_DIR directory (configurable),
    • Hook scripts need to be executable; you can add the execution flag with git update-index --chmod=+x pre-compose-up.sh.

    Cleanup

    The cleanup is processed as follows by the template:

    1. optionally executes the pre-compose-down.sh script found in your project to perform pre-cleanup stuff,
    2. runs docker-compose down or docker stack rm based on the DCMP_CMD variable,
    3. optionally executes the post-compose-down.sh script found in your project to perform post-cleanup stuff,

    ℹ️ Important:

    • Compose files, dotenv files and hook scripts are searched in the $DCMP_SCRIPTS_DIR directory (configurable),
    • Hook scripts need to be executable; you can add the execution flag with git update-index --chmod=+x pre-compose-up.sh.

    Using Variables

    Your deployment (and cleanup) scripts have to be able to cope with various environments, each with different application names, exposed routes, settings, ... Part of this complexity can be handled by the lookup strategies described above (ex: one file per env) and also by using available environment variables:

    1. deployment context variables provided by the template:
      • ${environment_type}: the current environment type (review, integration, staging or production)
      • ${environment_name}: the application name to use for the current environment (ex: myproject-review-fix-bug-12 or myproject-staging)
      • ${hostname}: the environment hostname, extracted from the current environment url (after late variable expansion - see below)
    2. any GitLab CI variable
    3. any custom variable (ex: ${SECRET_TOKEN} that you have set in your project CI/CD variables)

    Be aware that environment variables may be freely used and substituted in dotenv files using the appropriate interpolation syntax.

    Environments URL management

    The Docker Compose template supports two ways of providing your environments url:

    • a static way: when the environments url can be determined in advance, probably because you're exposing your routes through a DNS you manage,
    • a dynamic way: when the url cannot be known before the deployment job is executed.

    The static way can be implemented simply by setting the appropriate configuration variable(s) depending on the environment (see environments configuration chapters):

    • $DCMP_ENVIRONMENT_URL to define a default url pattern for all your envs,
    • $DCMP_REVIEW_ENVIRONMENT_URL, $DCMP_INTEG_ENVIRONMENT_URL, $DCMP_STAGING_ENVIRONMENT_URL and $DCMP_PROD_ENVIRONMENT_URL to override the default.

    ℹ️ Each of those variables support a late variable expansion mechanism with the %{somevar} syntax, allowing you to use any dynamically evaluated variables such as ${environment_name}.

    Example:

    variables:
      DCMP_BASE_APP_NAME: "wonderapp"
      # global url for all environments
      DCMP_ENVIRONMENT_URL: "https://%{environment_name}.nonprod.acme.domain"
      # override for prod (late expansion of $DCMP_BASE_APP_NAME not needed here)
      DCMP_PROD_ENVIRONMENT_URL: "https://$DCMP_BASE_APP_NAME.acme.domain"
      # override for review (using separate resource paths)
      DCMP_REVIEW_ENVIRONMENT_URL: "https://wonderapp-review.nonprod.acme.domain/%{environment_name}"

    To implement the dynamic way, your post deployment hook script shall simply generate a environment_url.txt file in the working directory, containing only the dynamically generated url. When detected by the template, it will use it as the newly deployed environment url.

    Deployment output variables

    Each deployment job produces output variables that are propagated to downstream jobs (using dotenv artifacts):

    • $environment_type: set to the type of environment (review, integration, staging or production),
    • $environment_name: the application name (see below),
    • $environment_url: set to the environment URL (whether determined statically or dynamically).

    Those variables may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).

    You may also add and propagate your own custom variables, by pushing them to the docker-compose.out.env file in your deployment scripts or hooks.

    Configuration reference

    Secrets management

    Here are some advices about your secrets (variables marked with a 🔒):

    1. Manage them as project or group CI/CD variables:
      • masked to prevent them from being inadvertently displayed in your job logs,
      • protected if you want to secure some secrets you don't want everyone in the project to have access to (for instance production secrets).
    2. In case a secret contains characters that prevent it from being masked, simply define its value as the Base64 encoded value prefixed with @b64@: it will then be possible to mask it and the template will automatically decode it prior to using it.
    3. Don't forget to escape special characters (ex: $ -> $$).

    Global configuration

    The Docker Compose template uses some global configuration used throughout all jobs and environments.

    Input / Variable Description Default value
    image / DCMP_IMAGE The Docker image used to run Docker Compose CLI commands registry.hub.docker.com/library/docker:latest
    cmd / DCMP_CMD The docker compose or stack command (docker compose, docker-compose or docker stack) none (auto)
    base-app-name / DCMP_BASE_APP_NAME Base application name $CI_PROJECT_NAME (see GitLab doc)
    environment-url / DCMP_ENVIRONMENT_URL Default environments url (only define for static environment URLs declaration)
    supports late variable expansion (ex: https://%{environment_name}.docker-compose.acme.com)
    none
    scripts-dir / DCMP_SCRIPTS_DIR Directory where Compose files, dotenv files and hook scripts are located . (root project dir)
    up-opts / DCMP_UP_OPTS compose up options (only when using Docker Compose) --no-build --remove-orphans --wait --wait-timeout 180
    down-opts/ DCMP_DOWN_OPTS compose down options (only when using Docker Compose) --volumes --remove-orphans --rmi all
    stack-deploy-opts / DCMP_STACK_DEPLOY_OPTS stack deploy options (only when using Docker Stack) --prune
    🔒 DCMP_SSH_PRIVATE_KEY Default SSH key to use when connecting to Docker hosts over SSH (can be overridden per env) none
    ssh-known-hosts / DCMP_SSH_KNOWN_HOSTS SSH known_hosts (file or text variable: you can get its value with ssh-keyscan -H <DOCKER_HOST>, mandatory if you're using a remote docker through ssh) none

    Review environments configuration

    Review environments are dynamic and ephemeral environments to deploy your ongoing developments (a.k.a. feature or topic branches).

    They are disabled by default and can be enabled by setting the DCMP_REVIEW_DOCKER_HOST variable (see below).

    Here are variables supported to configure review environments:

    Input / Variable Description Default value
    review-docker-host / DCMP_REVIEW_DOCKER_HOST Docker Host for review env (ex: ssh://user@docker-host-for-review) none (disabled)
    🔒 DCMP_REVIEW_SSH_PRIVATE_KEY review env specific SSH key to use when connecting to Docker Host over SSH $DCMP_SSH_PRIVATE_KEY
    review-app-name / DCMP_REVIEW_APP_NAME Application name for review env "${DCMP_BASE_APP_NAME}-${CI_ENVIRONMENT_SLUG}" (ex: myproject-review-fix-bug-12)
    review-environment-url / DCMP_REVIEW_ENVIRONMENT_URL The review environments url (only define for static environment URLs declaration and if different from default) $DCMP_ENVIRONMENT_URL
    review-autostop-duration / DCMP_REVIEW_AUTOSTOP_DURATION The amount of time before GitLab will automatically stop review environments 4 hours

    Integration environment configuration

    The integration environment is the environment associated to your integration branch (develop by default).

    It is disabled by default and can be enabled by setting the DCMP_INTEG_DOCKER_HOST variable (see below).

    Here are variables supported to configure the integration environment:

    Input / Variable Description Default value
    integ-docker-host / DCMP_INTEG_DOCKER_HOST Docker Host for integration env (ex: ssh://user@docker-host-for-integ) none (disabled)
    🔒 DCMP_INTEG_SSH_PRIVATE_KEY integration env specific SSH key to use when connecting to Docker Host over SSH $DCMP_SSH_PRIVATE_KEY
    integ-app-name / DCMP_INTEG_APP_NAME Application name for integration env ${DCMP_BASE_APP_NAME}-integration
    integ-environment-url / DCMP_INTEG_ENVIRONMENT_URL The integration environment url (only define for static environment URLs declaration and if different from default) $DCMP_ENVIRONMENT_URL

    Staging environment configuration

    The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production branch (main or master by default).

    It is disabled by default and can be enabled by setting the DCMP_STAGING_DOCKER_HOST variable (see below).

    Here are variables supported to configure the staging environment:

    Input / Variable Description Default value
    staging-docker-host / DCMP_STAGING_DOCKER_HOST Docker Host for staging env (ex: ssh://user@docker-host-for-staging) none (disabled)
    🔒 DCMP_STAGING_SSH_PRIVATE_KEY staging env specific SSH key to use when connecting to Docker Host over SSH $DCMP_SSH_PRIVATE_KEY
    staging-app-name / DCMP_STAGING_APP_NAME Application name for staging env ${DCMP_BASE_APP_NAME}-staging
    staging-environment-url / DCMP_STAGING_ENVIRONMENT_URL The staging environment url (only define for static environment URLs declaration and if different from default) $DCMP_ENVIRONMENT_URL

    Production environment configuration

    The production environment is the final deployment environment associated with your production branch (main or master by default).

    It is disabled by default and can be enabled by setting the DCMP_PROD_DOCKER_HOST variable (see below).

    Here are variables supported to configure the production environment:

    Input / Variable Description Default value
    prod-docker-host / DCMP_PROD_DOCKER_HOST Docker Host for production env (ex: ssh://user@docker-host-for-prod) none (disabled)
    🔒 DCMP_PROD_SSH_PRIVATE_KEY production env specific SSH key to use when connecting to Docker Host over SSH $DCMP_SSH_PRIVATE_KEY
    prod-app-name / DCMP_PROD_APP_NAME Application name for production env $DCMP_BASE_APP_NAME
    prod-environment-url / DCMP_PROD_ENVIRONMENT_URL The production environment url (only define for static environment URLs declaration and if different from default) $DCMP_ENVIRONMENT_URL
    prod-deploy-strategy / DCMP_PROD_DEPLOY_STRATEGY Defines the deployment to production strategy. One of manual (i.e. one-click) or auto. manual

    Compose Config job

    The Docker Compose template enables running Compose Config or Stack Config based on $DCMP_CMD, thus enabling detection of syntax errors in your Compose, dotenv or Stack files.

    This job is mapped to the package-test stage and is active by default.

    Here are its parameters:

    Input / Variable Description Default value
    config-disabled / DCMP_CONFIG_DISABLED Set to true to disable compose config none (enabled)
    config-opts / DCMP_CONFIG_OPTS compose config options --quiet (to avoid displaying secrets inadvertently)
    stack-config-opts / DCMP_STACK_CONFIG_OPTS stack config options ""
    stack-config-silent / DCMP_STACK_CONFIG_SILENT Silences standard output of stack config command true (standard output silenced)