Writing Gitlab CI templates, Part 2/3: template structure

Writing CI templates to run build, test, and deploy your project is challenging to do in a way that prioritizes pipeline speed, safety, and easy maintenance. In the first part of this series, I discussed the best practices for developing changes to CI templates. Here I discuss the best ways to setup your templates to reduce code duplication and allow for reduced maintenance.

Use a separate repo to host your project templates.

Within Gitlab, you can use the includes keyword to reference templates held within another git repo. This allows you to move all of your CI templates to a monorepo which allows you to easy track changes within templates and provide easy tracking of inheritance. When you use a monorepo, you should be careful with file organization to allow easy changes later.

I find what works best is to have one master pipeline configuration file which contains file references to each stage and the individual jobs within the pipelines organized like following.

globals/
  setup/
    setup.yaml
  build/
    docker-build.yaml
  deploy/
    docker-deploy.yaml
  promote/
    promote.yaml
java/
node/
golang/
  full.yaml
  setup/
    setup.yaml
  build/
    docker-build.yaml
  test/
    code-quality.yaml
    go-test.yaml
    container-scanning.yaml
  deploy/
    docker-deploy.yaml
  promote/
    promote.yaml

Within the full.yaml you would see include references to each of the files responsible for running each stage of the pipeline. Within each folder you hold specific jobs which run during each stage. If jobs share common code across languages, such as pipeline setup, building docker images, promoting with git tags, and deploying a docker images, you can have these files refer to a template within a global folder.

This way you have a full working pipeline that developers can add into the projects .gitlab-ci.yml.

include:   
  - project: 'my-group/ci-templates'
    ref: main
    file: '/golang/full.yaml'
variables:
  GO_VERSION: 1.17

Developers only need to add those five lines of code to implement a full working pipeline in the language of your choice for the version of their choice (Note to leave GO_VERSION empty in the CI templates since each project should define this individually to match the version in their Dockerfile and in their go.mod.)

Moreover, this configuration allows you to decouple pipeline changes from your projects. So long as you keep the pipeline yaml syntactically correct at all times (can be performed with a CI lint job that runs in your ci-templates repo), you can change or add jobs and have it take effect in projects that reference the master branch. You can also version your CI changes with tags or separate branches for stable and latest.

Keep your templates DRY.

We want to reduce the amount of code duplication within our templates, and we can do this in many ways. If you use gitlab, you can use the extends keyword which allows you to inherit a jobs template spec from a parent. If you are outside of gitlab, you can use yaml anchors to reference repeated points within. Additionally, within gitlab you can use the reference keyword and the includes keyword to reference configuration held within separate files, or within separate files held within another git repository as shown above.

Make leverage of theextends keyword with variables and hidden jobs(jobs starting with a period within gitlabci). Ideally you should be able to move all of your logic into a hidden job, with each non-hidden job describing the implementation via variables. An example is shown below where the logic in handled in the .go-migrate job but the implementation is handled in the go-migrate-dev-up job.

variables:
  MIGRATE_SEARCH_PATH: public
  MIGRATE_SOURCE: db/migrations
  MIGRATE_COUNT: ''

.go-migrate:
  image: migrate/migrate:v4.14.1
  stage: build
  script:
  - migrate -path $MIGRATE_SOURCE -database "postgres://$DB_USER:$DB_PASSWORD@$DB_HOST:$DB_PORT/$DB_NAME?sslmode=disable&search_path=$MIGRATE_SEARCH_PATH" $DIRECTION $MIGRATE_COUNT

.go-migrate-dev:
  extends: .go-migrate
  except: [master, tags]
  environment:
    name: dev
  variables:
    DB_HOST: mydatabase
    DB_NAME: dbname
    DB_PORT: 5432
    DB_USER: postgres
    DB_PASSWORD: $DEV_DB_PASSWORD

    
go-migrate-dev-up:
  extends: .go-migrate-dev
  variables:
    DIRECTION: up
go-migrate-dev-down:
  extends: .go-migrate-dev
  when: manual
  variables:
    DIRECTION: down

In the above we see some global variables, MIGRATE_SOURCE_PATH, MIGRATE_SOURCE, MIGRATE_COUNT, exposed that the end use may want to override in their projects .gitlab-ci.yml. It is best to expose variables that the end-user may want to change under globalvariables rather than variableswithin a non-hidden or hidden job spec.

We also see two layers of inheritance done to reduce code duplication within the end jobs which simply have 1 use of the extends keyword, and one variable specified. This way if we ever want to adjust the logic of the migration, we only need to implement it at one place.

Note that the DEV_DB_PASSWORD is not defined in the pipeline configuration — This is up to each project to have this key managed securely.

Another way to reuse code is with the reference keyword which allows you to insert yaml template from files outside of the current one. An example of this would be to have hidden utility function scripts such as below which can be held in a global utils.yml file

.scripts
  git-promote:
  - VERSION=`git describe --abbrev=0 --tags`
  - VERSION=${VERSION/v/}
  - VERSION_BITS=(${VERSION//./ })
  - VNUM1=${VERSION_BITS[0]}
  - VNUM2=${VERSION_BITS[1]}
  - VNUM3=${VERSION_BITS[2]}

Then reference this within your promote.yml file.

hotfix-release:
  extends: .promote-template
  script:
  - !reference [.scripts, git-promote]
  - VNUM3=$((VNUM3+1))
  - NEW_TAG="v$VNUM1.$VNUM2.$VNUM3"
  - git tag -a "${NEW_TAG}" -m"Hotfix ${NEW_TAG}"
  - git push --tags

Another example could be to hold your caching configuration with your setup.yml file and reference these cache configurations within later jobs for your pipelines.

.cache:
  pull-only:
    key: $CI_COMMIT_REF_SLUG
    paths:
        - node_modules/
    policy: pull
  push:
    key: $CI_COMMIT_REF_SLUG
    paths:
        - node_modules/
    policy: push

Then reference the above caching configuration within each jobs spec:

eslint:
  stage: test
  cache: !reference [.cache, pull-only]
  ...
type-check:
  stage: test
  cache: !reference [.cache, pull-only]
  ...

Conclusion

• Use a separate Monorepo to store your template configuration.
• Use one file to reference the templates for a fully working end-to-end pipeline, and reference this pipeline file in projects using include
• Use extends reference and hidden jobs to reduce code duplication

Continue on to part 3 of this story here.