GitLab CI/CD - Dynamic Pipelines

In pipeline best practices, dynamic pipelines are generally discouraged. Readability is crucial for any codebase, and once you start using dynamic pipelines it becomes difficult — sometimes impossible — to guarantee readability or maintainability.

That said, there are certain scenarios where dynamic pipelines can actually help you maintain or even improve readability while also improving maintainability. In those cases, using a dynamic pipeline is the right call.

For example, suppose you need to run 100 tests for the same project in CI, where the only difference between each test is the argument passed in — arguments that will evolve as your product grows. Would you really want to repeat the following snippet 100 times?

test-with-arg-100:
  image: ubuntu
  stage: test
  script:
    - auto/test 100

Parent-Child Pipelines

In GitLab CI/CD, a parent-child pipeline means nesting the execution of another pipeline configuration file (the child pipeline) inside a parent pipeline.

Child pipeline types:

• Merge request child pipelines
• Dynamic child pipelines
• Nested child pipelines

For more details, refer to the official documentation.

Example: Deploying Multiple Applications with a Dynamic Child Pipeline

In my Homelab environment, I have an Infrastructure Kubernetes cluster that needs a set of foundational applications deployed, such as:

• external-dns for registering Ingress DNS records to a DNS server
• Experimental apps like HashiCorp Vault and Jenkins
• Monitoring apps like Prometheus

Each application has its own dedicated deployment script, as shown below.

├── auto
│   ├── deploy-elasticsearch
│   ├── deploy-exdns-homelab-local
│   ├── deploy-grafana
│   ├── deploy-hashicorp-vault
│   ├── deploy-influxdb
│   ├── deploy-ingress
│   ├── deploy-jenkins
│   ├── deploy-prometheus
│   ├── deploy-sonarqube
│   └── deploy-vsphere-prometheus

All deployment scripts follow the naming pattern auto/deploy-*. Based on this pattern, we can generate the child pipeline’s YAML configuration file automatically. The generation script is shown below.

#!/bin/bash -eu

cd "$(dirname "$0")/.."

CI_CONFIG_FILE="child-ci.yml"

cat <<EOF > "${CI_CONFIG_FILE}"
image: $CI_REGISTRY_IMAGE

default:
  retry: 2

variables:
  KUBERNETES_SERVICE_ACCOUNT_OVERWRITE: k8s-infra-admin

stages:
  - deploy

EOF

for script_name in auto/deploy-* ; do
      cat <<EOF >> "${CI_CONFIG_FILE}"
${script_name//\//-}:
  stage: deploy
  script:
    - ${script_name}
  only:
    - main

EOF
done

The relevant section of the .gitlab-ci.yml pipeline configuration file looks like this:

stages:
  - generate-ci
  - run-ci
default:
  retry: 2

generate-config:
  image: alpine:3.12
  before_script:
    - apk --no-cache add bash
  stage: generate-ci
  script: auto/generate-ci-config
  artifacts:
    paths:
      - child-ci.yml

child-pipeline:
  stage: run-ci
  trigger:
    strategy: depend
    include:
      - artifact: child-ci.yml
        job: generate-config

The pipeline execution graph is shown below: