# GitLab CI template for Docker Compose

This project implements a GitLab CI/CD template to deploy your application with [Docker Compose](https://docs.docker.com/compose/).

## Usage

This template can be used both as a [CI/CD component](https://docs.gitlab.com/ee/ci/components/#use-a-component) 
or using the legacy [`include:project`](https://docs.gitlab.com/ee/ci/yaml/index.html#includeproject) syntax.

### Use as a CI/CD component

Add the following to your `.gitlab-ci.yml`:

```yaml
include:
  # 1: include the component
  - component: $CI_SERVER_FQDN/to-be-continuous/docker-compose/gitlab-ci-docker-compose@1.0.2
    # 2: set/override component inputs
    inputs:
      # ⚠ this is only an example
      base-app-name: wonderapp
      review-docker-host: "ssh://user@192.168.64.5" # enable review env
      staging-docker-host: "ssh://user@192.168.64.6" # enable staging env
      prod-docker-host: "ssh://user@192.168.64.7" # enable production env
```

### Use as a CI/CD template (legacy)

Add the following to your `.gitlab-ci.yml`:

```yaml
include:
  # 1: include the template
  - project: 'to-be-continuous/docker-compose'
    ref: '1.0.2'
    file: '/templates/gitlab-ci-docker-compose.yml'

variables:
  # 2: set/override template variables
  # ⚠ this is only an example
  DCMP_BASE_APP_NAME: wonderapp
  DCMP_REVIEW_DOCKER_HOST: "ssh://user@192.168.64.5" # enable review env
  DCMP_STAGING_DOCKER_HOST: "ssh://user@192.168.64.6" # enable staging env
  DCMP_PROD_DOCKER_HOST: "ssh://user@192.168.64.7" # enable production env
```

## Understand

This chapter introduces key notions and principle to understand how this template works.

### Managed deployment environments

This template implements continuous delivery/continuous deployment for projects hosted on Docker Compose.

It allows you to manage automatic deployment & cleanup of standard predefined environments.
Each environment can be enabled/disabled by configuration.
If you're not satisfied with predefined environments and/or their associated Git workflow, you may implement you own environments and
workflow, by reusing/extending the base (hidden) jobs. This is advanced usage and will not be covered by this documentation.

The following chapters present the managed predefined environments and their associated Git workflow.

#### Review environments

The template supports **review** environments: those are dynamic and ephemeral environments to deploy your
_ongoing developments_ (a.k.a. _feature_ or _topic_ branches).

When enabled, it deploys the result from upstream build stages to a dedicated and temporary environment.
It is only active for non-production, non-integration branches.

It is a strict equivalent of GitLab's [Review Apps](https://docs.gitlab.com/ee/ci/review_apps/) feature.

It also comes with a _cleanup_ job (accessible either from the _environments_ page, or from the pipeline view).

#### Integration environment

If you're using a Git Workflow with an integration branch (such as [Gitflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow)),
the template supports an **integration** environment.

When enabled, it deploys the result from upstream build stages to a dedicated environment.
It is only active for your integration branch (`develop` by default).

#### Production environments

Lastly, the template supports 2 environments associated to your production branch (`master` or `main` by default):

* a **staging** environment (an iso-prod environment meant for testing and validation purpose),
* the **production** environment.

You're free to enable whichever or both, and you can also choose your deployment-to-production policy:

* **continuous deployment**: automatic deployment to production (when the upstream pipeline is successful),
* **continuous delivery**: deployment to production can be triggered manually (when the upstream pipeline is successful).

### Supported authentication methods

The Docker Compose template supports [deployment on remote Docker hosts](https://www.docker.com/blog/how-to-deploy-on-remote-docker-hosts-with-docker-compose/), using 
dedicated variables:

* `DCMP_REVIEW_DOCKER_HOST`, `DCMP_INTEG_DOCKER_HOST`, `DCMP_STAGING_DOCKER_HOST` and `DCMP_PROD_DOCKER_HOST` to both enable and configure the target Docker host for each environment (ex: `ssh://user@192.168.64.5`),
* :lock: `DCMP_SSH_PRIVATE_KEY`, :lock: `DCMP_REVIEW_SSH_PRIVATE_KEY`, :lock: `DCMP_INTEG_SSH_PRIVATE_KEY`, :lock: `DCMP_STAGING_SSH_PRIVATE_KEY` and :lock: `DCMP_PROD_SSH_PRIVATE_KEY` to provide the global or per env SSH private key (in case SSH authentication is required).

### Deployment context variables

In order to manage the various deployment environments, this template provides a couple of **dynamic variables**
that you might use in your hook scripts or [Docker Compose files](https://docs.docker.com/compose/compose-application-model/#the-compose-file):

* `${environment_type}`: the current deployment environment type (`review`, `integration`, `staging` or `production`)
* `${environment_name}`: a generated application name to use for the current deployment environment (ex: `myapp-review-fix-bug-12` or `myapp-staging`) - _details below_

> :information_source: the `${environment_name}` is used by the Docker Compose template as the [Docker Compose project name](https://docs.docker.com/compose/project-name/).

#### Generated environment name

The `${environment_name}` variable is generated to designate each deployment environment with a unique and meaningful application name.
By construction, it is suitable for inclusion in DNS, URLs, Kubernetes labels...
It is built from:

* the application _base name_ (defaults to `$CI_PROJECT_NAME` but can be overridden globally and/or per deployment environment - _see configuration variables_)
* GitLab predefined `$CI_ENVIRONMENT_SLUG` variable ([slugified](https://en.wikipedia.org/wiki/Clean_URL#Slug) name, truncated to 24 characters)

The `${environment_name}` variable is then evaluated as:

* `<app base name>` for the production environment
* `<app base name>-$CI_ENVIRONMENT_SLUG` for all other deployment environments
* :bulb: `${environment_name}` can also be overridden per environment with the appropriate configuration variable

Examples (with an application's base name `myapp`):

| `$environment_type` | Branch        | `$CI_ENVIRONMENT_SLUG`  | `$environment_name` |
|---------------------|---------------|-------------------------|---------------------|
| `review`            | `feat/blabla` | `review-feat-bla-xmuzs6`| `myapp-review-feat-bla-xmuzs6` |
| `integration`       | `develop`     | `integration`           | `myapp-integration` |
| `staging`           | `main`        | `staging`               | `myapp-staging` |
| `production`        | `main`        | `production`            | `myapp` |

### Deployment and cleanup

The Docker Compose template requires you to provide:

* a _required_ [Compose file](https://docs.docker.com/compose/compose-application-model/#the-compose-file) that implements your application deployment and cleanup,
* _optional_ [`.override.` Compose files](https://docs.docker.com/compose/multiple-compose-files/merge/) defining common and/or per-env override,
* _optional_ [dotenv files](https://docs.docker.com/compose/environment-variables/env-file/) defining common and/or per-env configuration,
* _optional_ hook scripts (shell) to implement logic that can't be implemented with Docker Compose.

#### Compose files lookup strategy

Unless you have explicitly set the [`COMPOSE_FILES`](https://docs.docker.com/compose/environment-variables/envvars/#compose_file) variable, the Docker Compose template will handle it and 
implement the following [Compose file(s)](https://docs.docker.com/compose/compose-application-model/#the-compose-file) lookup strategy:

| Lookup order | Compose file | Additional `.override.` files (optional)                                                                                           |
|:------------:|--------------|------------------------------------------------------------------------------------------------------------------------------------------|
|      1.      | **environment specific** (`compose-${environment_type}.yaml`) | **environment specific** override (`compose-${environment_type}.override.yaml`)                                                          |
|      2.      | **default** (`compose.yaml`) | 1. **default** override (`compose.override.yaml`)<br/>2. **environment specific** override (`compose-${environment_type}.override.yaml`) |

> :information_source: Important:
>
> * Compose file base name `docker-compose` is also supported as an alternative to `compose`,
> * Compose file extension `.yml` is also supported as an alternative to `.yaml`,
> * `.override.` files must match the same base name (`compose` or `docker-compose`) and extension (`yaml` or `yml`) as the found Compose file.

Examples with different combinations of files:

<table>
  <thead>
    <tr>
      <th>Files in your project</th>
      <th>Compose files for Review</th>
      <th>Compose files for Integration</th>
      <th>Compose files for Staging</th>
      <th>Compose files for Production</th>
    </tr>
  </thead>
  <tfoot>
    <tr>
      <td>- <code>compose.yaml</code></td>
      <td colspan="4" align=center><code>compose.yaml</code></td>
    </tr>
    <tr>
      <td>- <code>compose.yaml</code><br/>- <code>compose-production.yaml</code></td>
      <td colspan="3" align=center><code>compose.yaml</code></td>
      <td align=center><code>compose-production.yaml</code></td>
    </tr>
    <tr>
      <td>- <code>docker-compose.yml</code><br/>- <code>docker-compose-production.override.yml</code></td>
      <td colspan="3" align=center><code>docker-compose.yml</code></td>
      <td align=center><code>docker-compose.yml</code> + <code>docker-compose-production.override.yml</code> (merged)</td>
    </tr>
    <tr>
      <td>- <code>compose.yml</code><br/>- <code>compose.override.yml</code><br/>- <code>compose-review.override.yml</code><br/>- <code>compose-production.yml</code></td>
      <td align=center><code>compose.yml</code> + <code>compose.override.yml</code> + <code>compose-review.override.yml</code> (merged)</td>
      <td colspan="2" align=center><code>compose.yml</code>+ <code>compose.override.yml</code> (merged)</td>
      <td align=center><code>compose-production.yml</code></td>
    </tr>
  </tfoot>
  <tbody>
  </tbody>
</table>

#### Dotenv files lookup strategy

Unless you have explicitly set the [`COMPOSE_ENV_FILES`](https://docs.docker.com/compose/environment-variables/envvars/#compose_env_files) variable, the Docker Compose template will handle it 
and implement the following [dotenv files](https://docs.docker.com/compose/environment-variables/env-file/) lookup strategy:

  1. a `.env` file defining defaults for all environments,
  2. a `${environment_type}.env` file that might redefine or override defaults for a specific environment (e.g. `staging.env`).

#### Deployment

The **deployment** is processed as follows by the template:

1. _optionally_ executes the `pre-compose-up.sh` script found in your project to perform pre-deployment stuff (for e.g. create required services),
2. runs [`docker-compose up`](https://docs.docker.com/reference/cli/docker/compose/up/),
3. _optionally_ executes the `post-compose-up.sh` script found in your project to perform post-deployment stuff,

> :information_source: Important:
>
> * Compose files, dotenv files and hook scripts are searched in the `$DCMP_SCRIPTS_DIR` directory (configurable),
> * Hook scripts need to be executable; you can add the execution flag with `git update-index --chmod=+x pre-compose-up.sh`.

#### Cleanup

The **cleanup** is processed as follows by the template:

1. _optionally_ executes the `pre-compose-down.sh` script found in your project to perform pre-cleanup stuff,
2. runs [`docker-compose down`](https://docs.docker.com/reference/cli/docker/compose/down/),
3. _optionally_ executes the `post-compose-down.sh` script found in your project to perform post-cleanup stuff,

> :information_source:  Important:
>
> * Compose files, dotenv files and hook scripts are searched in the `$DCMP_SCRIPTS_DIR` directory (configurable),
> * Hook scripts need to be executable; you can add the execution flag with `git update-index --chmod=+x pre-compose-up.sh`.

#### Using Variables

Your deployment (and cleanup) scripts have to be able to cope with various environments, each with different application names, exposed routes, settings, ...
Part of this complexity can be handled by the lookup strategies described above (ex: one file per env) and also by using available environment variables:

1. [deployment context variables](#deployment-context-variables) provided by the template:
   * `${environment_type}`: the current environment type (`review`, `integration`, `staging` or `production`)
   * `${environment_name}`: the application name to use for the current environment (ex: `myproject-review-fix-bug-12` or `myproject-staging`)
   * `${hostname}`: the environment hostname, extracted from the current environment url (after late variable expansion - see below)
2. any [GitLab CI variable](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)
3. any [custom variable](https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-docker-host)
   (ex: `${SECRET_TOKEN}` that you have set in your project CI/CD variables)

Be aware that environment variables may be freely used and substituted in [dotenv files](https://docs.docker.com/compose/environment-variables/env-file/) 
using the appropriate [interpolation syntax](https://docs.docker.com/compose/environment-variables/env-file/#interpolation).

### Environments URL management

The Docker Compose template supports two ways of providing your environments url:

* a **static way**: when the environments url can be determined in advance, probably because you're exposing your routes through a DNS you manage,
* a [**dynamic way**](https://docs.gitlab.com/ee/ci/environments/#set-a-dynamic-environment-url): when the url cannot be known before the
  deployment job is executed.

The **static way** can be implemented simply by setting the appropriate configuration variable(s) depending on the environment (see environments configuration chapters):

* `$DCMP_ENVIRONMENT_URL` to define a default url pattern for all your envs,
* `$DCMP_REVIEW_ENVIRONMENT_URL`, `$DCMP_INTEG_ENVIRONMENT_URL`, `$DCMP_STAGING_ENVIRONMENT_URL` and `$DCMP_PROD_ENVIRONMENT_URL` to override the default.

> :information_source: Each of those variables support a **late variable expansion mechanism** with the `%{somevar}` syntax, 
> allowing you to use any dynamically evaluated variables such as `${environment_name}`.
>
> Example:
>
> ```yaml
> variables:
>   DCMP_BASE_APP_NAME: "wonderapp"
>   # global url for all environments
>   DCMP_ENVIRONMENT_URL: "https://%{environment_name}.nonprod.acme.domain"
>   # override for prod (late expansion of $DCMP_BASE_APP_NAME not needed here)
>   DCMP_PROD_ENVIRONMENT_URL: "https://$DCMP_BASE_APP_NAME.acme.domain"
>   # override for review (using separate resource paths)
>   DCMP_REVIEW_ENVIRONMENT_URL: "https://wonderapp-review.nonprod.acme.domain/%{environment_name}"
> ```

To implement the **dynamic way**, your post deployment hook script shall simply generate a `environment_url.txt` file in the working directory, containing only
the dynamically generated url. When detected by the template, it will use it as the newly deployed environment url.

### Deployment output variables

Each deployment job produces _output variables_ that are propagated to downstream jobs (using [dotenv artifacts](https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportsdotenv)):

* `$environment_type`: set to the type of environment (`review`, `integration`, `staging` or `production`),
* `$environment_name`: the application name (see below),
* `$environment_url`: set to the environment URL (whether determined statically or dynamically).

Those variables may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).

You may also add and propagate your own custom variables, by pushing them to the `docker-compose.out.env` file in your [deployment scripts or hooks](#deployment-and-cleanup).

## Configuration reference

### Secrets management

Here are some advices about your **secrets** (variables marked with a :lock:): 

1. Manage them as [project or group CI/CD variables](https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-docker-host):
    * [**masked**](https://docs.gitlab.com/ee/ci/variables/#mask-a-cicd-variable) to prevent them from being inadvertently 
      displayed in your job logs,
    * [**protected**](https://docs.gitlab.com/ee/ci/variables/#protected-cicd-variables) if you want to secure some secrets 
      you don't want everyone in the project to have access to (for instance production secrets).
2. In case a secret contains [characters that prevent it from being masked](https://docs.gitlab.com/ee/ci/variables/#mask-a-cicd-variable), 
  simply define its value as the [Base64](https://en.wikipedia.org/wiki/Base64) encoded value prefixed with `@b64@`: 
  it will then be possible to mask it and the template will automatically decode it prior to using it.
3. Don't forget to escape special characters (ex: `$` -> `$$`).

### Global configuration

The Docker Compose template uses some global configuration used throughout all jobs and environments.

| Input / Variable         | Description                            | Default value     |
| ------------------------ | -------------------------------------- | ----------------- |
| `image` / `DCMP_IMAGE` | the Docker image used to run Docker Compose CLI commands | `registry.hub.docker.com/library/docker:latest` |
| `cmd` / `DCMP_CMD`     | The docker compose command (`docker compose` or `docker-compose`) | _none_ (auto) |
| `base-app-name` / `DCMP_BASE_APP_NAME`| Base application name                  | `$CI_PROJECT_NAME` ([see GitLab doc](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)) |
| `environment-url` / `DCMP_ENVIRONMENT_URL`| Default environments url _(only define for static environment URLs declaration)_<br/>_supports late variable expansion (ex: `https://%{environment_name}.docker-compose.acme.com`)_ | _none_ |
| `scripts-dir` / `DCMP_SCRIPTS_DIR`| Directory where Compose files, dotenv files and hook scripts are located | `.` _(root project dir)_ |
| `up-opts` / `DCMP_UP_OPTS` | [`compose up` options](https://docs.docker.com/reference/cli/docker/compose/up/#options) | `--no-build --remove-orphans --wait --wait-timeout 180` |
| `down-opts`/ `DCMP_DOWN_OPTS` | [`compose down` options](https://docs.docker.com/reference/cli/docker/compose/down/#options) | `--volumes --remove-orphans --rmi all` |
| :lock: `DCMP_SSH_PRIVATE_KEY` | Default SSH key to use when connecting to Docker hosts over SSH (can be overridden per env) | _none_ |
| `ssh-known-hosts` / `DCMP_SSH_KNOWN_HOSTS` | SSH `known_hosts` (file or text variable) | _none_ |
| `compose-cleanup-review-job-tags` / `COMPOSE_CLEANUP_REVIEW_JOB_TAGS` | Tags to be used for selecting runners for the job | [] |

### Review environments configuration

Review environments are dynamic and ephemeral environments to deploy your _ongoing developments_ (a.k.a. _feature_ or 
_topic_ branches).

They are **disabled by default** and can be enabled by setting the `DCMP_REVIEW_DOCKER_HOST` variable (see below).

Here are variables supported to configure review environments:

| Input / Variable         | Description                                                                                                       | Default value     |
| ------------------------ |-------------------------------------------------------------------------------------------------------------------| ----------------- |
| `review-docker-host` / `DCMP_REVIEW_DOCKER_HOST`     | Docker Host for `review` env (ex: `ssh://user@docker-host-for-review`)                                            | _none_ (disabled) |
| :lock: `DCMP_REVIEW_SSH_PRIVATE_KEY` | `review` env specific SSH key to use when connecting to Docker Host over SSH                                      | `$DCMP_SSH_PRIVATE_KEY` |
| `review-app-name` / `DCMP_REVIEW_APP_NAME`    | Application name for `review` env                                                                                 | `"${DCMP_BASE_APP_NAME}-${CI_ENVIRONMENT_SLUG}"` (ex: `myproject-review-fix-bug-12`) |
| `review-environment-url` / `DCMP_REVIEW_ENVIRONMENT_URL`| The review environments url _(only define for static environment URLs declaration and if different from default)_ | `$DCMP_ENVIRONMENT_URL` |
| `review-autostop-duration` / `DCMP_REVIEW_AUTOSTOP_DURATION`| The amount of time before GitLab will automatically stop `review` environments                                    | `4 hours` |
| `compose-review-job-tags` / `COMPOSE_REVIEW_JOB_TAGS` | Tags to be used for selecting runners for the job | [] |

### Integration environment configuration

The integration environment is the environment associated to your integration branch (`develop` by default).

It is **disabled by default** and can be enabled by setting the `DCMP_INTEG_DOCKER_HOST` variable (see below).

Here are variables supported to configure the integration environment:

| Input / Variable         | Description                                                                                                          | Default value     |
| ------------------------ |----------------------------------------------------------------------------------------------------------------------| ----------------- |
| `integ-docker-host` / `DCMP_INTEG_DOCKER_HOST`      | Docker Host for `integration` env (ex: `ssh://user@docker-host-for-integ`)                                       | _none_ (disabled) |
| :lock: `DCMP_INTEG_SSH_PRIVATE_KEY` | `integration` env specific SSH key to use when connecting to Docker Host over SSH                                    | `$DCMP_SSH_PRIVATE_KEY` |
| `integ-app-name` / `DCMP_INTEG_APP_NAME`     | Application name for `integration` env                                                                               | `${DCMP_BASE_APP_NAME}-integration` |
| `integ-environment-url` / `DCMP_INTEG_ENVIRONMENT_URL`| The integration environment url _(only define for static environment URLs declaration and if different from default)_ | `$DCMP_ENVIRONMENT_URL` |
| `compose-integration-job-tags` / `COMPOSE_INTEGRATION_JOB_TAGS` | Tags to be used for selecting runners for the job | [] |

### Staging environment configuration

The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production 
branch (`main` or `master` by default).

It is **disabled by default** and can be enabled by setting the `DCMP_STAGING_DOCKER_HOST` variable (see below).

Here are variables supported to configure the staging environment:

| Input / Variable         | Description                                                                                                       | Default value     |
| ------------------------ |-------------------------------------------------------------------------------------------------------------------| ----------------- |
| `staging-docker-host` / `DCMP_STAGING_DOCKER_HOST`    | Docker Host for `staging` env (ex: `ssh://user@docker-host-for-staging`)                                          | _none_ (disabled) |
| :lock: `DCMP_STAGING_SSH_PRIVATE_KEY` | `staging` env specific SSH key to use when connecting to Docker Host over SSH                                     | `$DCMP_SSH_PRIVATE_KEY` |
| `staging-app-name` / `DCMP_STAGING_APP_NAME`   | Application name for `staging` env                                                                                | `${DCMP_BASE_APP_NAME}-staging` |
| `staging-environment-url` / `DCMP_STAGING_ENVIRONMENT_URL`| The staging environment url _(only define for static environment URLs declaration and if different from default)_ | `$DCMP_ENVIRONMENT_URL` |
| `compose-staging-job-tags` / `COMPOSE_STAGING_JOB_TAGS` | Tags to be used for selecting runners for the job | [] |

### Production environment configuration

The production environment is the final deployment environment associated with your production branch (`main` or `master` by default).

It is **disabled by default** and can be enabled by setting the `DCMP_PROD_DOCKER_HOST` variable (see below).

Here are variables supported to configure the production environment:

| Input / Variable          | Description                                                                                                         | Default value     |
| ------------------------- |---------------------------------------------------------------------------------------------------------------------| ----------------- |
| `prod-docker-host` / `DCMP_PROD_DOCKER_HOST`        | Docker Host for `production` env (ex: `ssh://user@docker-host-for-prod`)                                        | _none_ (disabled) |
| :lock: `DCMP_PROD_SSH_PRIVATE_KEY` | `production` env specific SSH key to use when connecting to Docker Host over SSH                                    | `$DCMP_SSH_PRIVATE_KEY` |
| `prod-app-name` / `DCMP_PROD_APP_NAME`       | Application name for `production` env                                                                               | `$DCMP_BASE_APP_NAME` |
| `prod-environment-url` / `DCMP_PROD_ENVIRONMENT_URL`| The production environment url _(only define for static environment URLs declaration and if different from default)_ | `$DCMP_ENVIRONMENT_URL` |
| `prod-deploy-strategy` / `DCMP_PROD_DEPLOY_STRATEGY`| Defines the deployment to production strategy. One of `manual` (i.e. _one-click_) or `auto`.                        | `manual` |
| `compose-production-job-tags` / `COMPOSE_PRODUCTION_JOB_TAGS` | Tags to be used for selecting runners for the job | [] |

### Compose Config job

The Docker Compose template enables running [Compose Config](https://docs.docker.com/reference/cli/docker/compose/config/), thus enabling detection of syntax errors in your Compose or dotenv files.

This job is mapped to the `package-test` stage and is **active** by default.

Here are its parameters:

| Input / Variable | Description                               | Default value                 |
| ----------------------- | ----------------------------------------- | ----------------------------- |
| `config-disabled` / `DCMP_CONFIG_DISABLED` | Set to `true` to disable `compose config`  | _none_ (enabled) |
| `config-opts` / `DCMP_CONFIG_OPTS` | [`compose config` options](https://docs.docker.com/reference/cli/docker/compose/config/#options) | `--quiet` _(to avoid displaying secrets inadvertently)_ |
| `compose-config-job-tags` / `COMPOSE_CONFIG_JOB_TAGS` | Tags to be used for selecting runners for the job | [] |