Merge branch 'feat/env-expand-enhancements' into 'master'

feat: env expand enhancements

See merge request to-be-continuous/kubernetes!88

README.md

@@ -144,7 +144,7 @@ Examples (with an application's base name `myapp`):
 The Kubernetes template supports three techniques to deploy your code:
 
 1. script-based deployment,
-2. template-based deployment using raw Kubernetes manifests (with variables substitution),
+2. template-based deployment using raw Kubernetes manifests (with [variables substitution](#variables-substitution-mechanism)),
 3. template-based deployment using [Kustomization files](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/).
 
 #### 1: script-based deployment
@@ -167,7 +167,7 @@ in your project structure, and let the GitLab CI template [`kubectl apply`](http
 The template processes the following steps:
 
 1. _optionally_ executes the `k8s-pre-apply.sh` script in your project to perform specific environment pre-initialization (for e.g. create required services),
-2. looks for your Kubernetes deployment file, performs [variables substitution](#using-variables) and [`kubectl apply`](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#apply) it,
+2. looks for your Kubernetes deployment file, performs [variables substitution](#variables-substitution-mechanism) and [`kubectl apply`](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#apply) it,
     1. look for a specific `deployment-$environment_type.yml` in your project (e.g. `deployment-staging.yml` for staging environment),
     2. fallbacks to default `deployment.yml`.
 3. _optionally_ executes the `k8s-post-apply.sh` script in your project to perform specific environment post-initialization stuff,
@@ -202,7 +202,7 @@ After deployment (either script-based or template-based), the GitLab CI template
 The Kubernetes template supports three techniques to destroy an environment (actually only review environments):
 
 1. script-based deployment,
-2. template-based deployment using raw Kubernetes manifests (with variables substitution),
+2. template-based deployment using raw Kubernetes manifests (with [variables substitution](#variables-substitution-mechanism)),
 3. template-based deployment using [Kustomization files](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/).
 
 #### 1: script-based cleanup
@@ -282,20 +282,26 @@ by using available environment variables:
 3. any [custom variable](https://docs.gitlab.com/ee/ci/variables/#for-a-project)
     (ex: `${SECRET_TOKEN}` that you have set in your project CI/CD variables)
 
-While your scripts may simply use any of those variables, your Kubernetes and Kustomize resources can use **variable substitution**
-with the syntax `${VARIABLE_NAME}`.
-Each of those patterns will be dynamically replaced in your resources by the template right before using it.
+#### Variables substitution mechanism
 
-You can prevent any line from being processed by appending `# nosubst` at the end of the line.
-For instance in the following example, `${REMOTE_SERVICE_NAME}` won't be replaced by its environment value during GitLab job execution:
+While your scripts may freely use any of the available variables, your Kubernetes and Kustomize
+resources can use a **variables substitution** mechanism implemented by the template:
 
+- Using the syntax `${VARIABLE_NAME}` or `%{VARIABLE_NAME}`.\
+  :warning: Curly braces (`{}`) are mandatory in the expression (`$VARIABLE_NAME` won't be processed).
+- Each of those expressions will be **dynamically expanded** in your resource files with the variable value, right before being used.
+- Variable substitution expressions **must be contained in double-quoted strings**.
+  The substitution implementation takes care of escaping characters that need to be (double quote `"`, backslash `\`, tab `\t`, carriage return `\n` and line feed `\r`).
+- Variable substitution can be prevented by appending `# nosubst` at the end of any line.\
+  Ex:
   ```yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
+    # ${environment_name} will be expanded
     labels:
-    app: ${APPLICATION_NAME}
-  name: ${APPLICATION_NAME}
+      app: "${environment_name}"
+    name: "${environment_name}"
   data:
     application.yml: |
       remote:
@@ -303,21 +309,6 @@ data:
             name: '${REMOTE_SERVICE_NAME}' # nosubst
   ```
 
-> :warning: In order to be properly replaced, curly braces are mandatory (ex: `${MYVAR}` and not `$MYVAR`).
-> Moreover, multiline variables must be surrounded by **double quotes** (`"`).
->
-> Example:
->
-> ```yaml
-> [...]
-> containers:
-> - name: restaurant-app
->   env:
->   # multiline variable
->   - name: MENU
->     value: "${APP_MENU}"
-> ```
-
 ### Environments URL management
 
 The K8S template supports two ways of providing your environments url:

templates/gitlab-ci-k8s.yml

@@ -419,9 +419,78 @@ stages:
     echo "$1" | tr '[:lower:]' '[:upper:]' | tr '[:punct:]' '_'
   }
 
-  function awkenvsubst() {
-    # performs variables escaping: '&' for gsub + JSON chars ('\' and '"')
-    awk '!/# *nosubst/{while(match($0,"[$%]{[^}]*}")) {var=substr($0,RSTART+2,RLENGTH-3);val=ENVIRON[var];gsub(/["\\&]/,"\\\\&",val);gsub("[$%]{"var"}",val)}}1'
+  function tbc_envsubst() {
+    awk '
+      BEGIN {
+        count_replaced_lines = 0
+        # ASCII codes
+        for (i=0; i<=255; i++)
+          char2code[sprintf("%c", i)] = i
+      }
+      # determine encoding (from env or from file extension)
+      function encoding() {
+        enc = ENVIRON["TBC_ENVSUBST_ENCODING"]
+        if (enc != "")
+          return enc
+        if (match(FILENAME, /\.(json|yaml|yml)$/))
+          return "jsonstr"
+        return "raw"
+      }
+      # see: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent
+      function uriencode(str) {
+        len = length(str)
+        enc = ""
+        for (i=1; i<=len; i++) {
+          c = substr(str, i, 1);
+          if (index("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-_.!~*'\''()", c))
+            enc = enc c
+          else
+            enc = enc "%" sprintf("%02X", char2code[c])
+        }
+        return enc
+      }
+      !/# *nosubst/ {
+        orig_line = $0
+        line = $0
+        count_repl_in_line = 0
+        # /!\ 3rd arg (match) not supported in BusyBox awk
+        while (match(line, /[$%]\{([[:alnum:]_]+)\}/)) {
+          expr_start = RSTART
+          expr_len = RLENGTH
+          # get var name
+          var = substr(line, expr_start+2, expr_len-3)
+          # get var value (from env)
+          val = ENVIRON[var]
+          # check variable is set
+          if (val == "") {
+            printf("[\033[1;93mWARN\033[0m] Environment variable \033[33;1m%s\033[0m is not set or empty\n", var) > "/dev/stderr"
+          } else {
+            enc = encoding()
+            if (enc == "jsonstr") {
+              gsub(/["\\]/, "\\\\&", val)
+              gsub("\n", "\\n", val)
+              gsub("\r", "\\r", val)
+              gsub("\t", "\\t", val)
+            } else if (enc == "uricomp") {
+              val = uriencode(val)
+            } else if (enc == "raw") {
+            } else {
+              printf("[\033[1;93mWARN\033[0m] Unsupported encoding \033[33;1m%s\033[0m: ignored\n", enc) > "/dev/stderr"
+            }
+          }
+          # replace expression in line
+          line = substr(line, 1, expr_start - 1) val substr(line, expr_start + expr_len)
+          count_repl_in_line++
+        }
+        if (count_repl_in_line) {
+          if (count_replaced_lines == 0)
+            printf("[\033[1;94mINFO\033[0m] Variable expansion occurred in file \033[33;1m%s\033[0m:\n", FILENAME) > "/dev/stderr"
+          count_replaced_lines++
+          printf("> line %s: %s\n", NR, orig_line) > "/dev/stderr"
+        }
+        print line
+      }
+    ' "$@"
   }
 
   function exec_hook() {
@@ -539,8 +608,8 @@ stages:
         exit 1
       fi
 
-      # replace variables (alternative for envsubst which is not present in image)
-      awkenvsubst < "$deploymentfile" > generated-deployment.yml
+      # variables substitution
+      tbc_envsubst "$deploymentfile" > generated-deployment.yml
 
       log_info "--- \\e[32mkubectl $action\\e[0m"
       kubectl ${TRACE+-v=5} "$action" -f ./generated-deployment.yml
@@ -562,7 +631,7 @@ stages:
     export appname_ssc=$environment_name_ssc
 
     # variables expansion in $environment_url
-    environment_url=$(echo "$environment_url" | awkenvsubst)
+    environment_url=$(echo "$environment_url" | TBC_ENVSUBST_ENCODING=uricomp tbc_envsubst)
     export environment_url
     # extract hostname from $environment_url
     hostname=$(echo "$environment_url" | awk -F[/:] '{print $4}')
@@ -644,7 +713,7 @@ stages:
     export appname_ssc=$environment_name_ssc
 
     # variables expansion in $environment_url
-    environment_url=$(echo "$environment_url" | awkenvsubst)
+    environment_url=$(echo "$environment_url" | TBC_ENVSUBST_ENCODING=uricomp tbc_envsubst)
     export environment_url
     # extract hostname from $environment_url
     hostname=$(echo "$environment_url" | awk -F[/:] '{print $4}')
@@ -755,8 +824,8 @@ stages:
         exit 1
       fi
 
-      # replace variables (alternative for envsubst which is not present in image)
-      awkenvsubst < "$deploymentfile" > generated-deployment.yml
+      # variables substitution
+      tbc_envsubst "$deploymentfile" > generated-deployment.yml
 
       # shellcheck disable=SC2086
       /usr/bin/kube-score score $K8S_SCORE_EXTRA_OPTS generated-deployment.yml
@@ -764,7 +833,7 @@ stages:
   }
 
   # export tool functions (might be used in after_script)
-  export -f log_info log_warn log_error assert_defined rollback awkenvsubst
+  export -f log_info log_warn log_error assert_defined rollback tbc_envsubst
 
   unscope_variables
   eval_all_secrets