Writing Gitlab CI templates, Part 1/3: local development

Developing continuous integration pipelines is often both time consuming and error-prone. Below are a collection of best practices and helpful tips for developing better pipelines that will improve how quickly and safely you can move code from a developers laptop into production.

I am most experienced with Gitlab so the recommendations are based on their engine, though most will apply to other CI tools also.

Developing CI.

How do we write and test and develop CI templates quickly? If your process looks like the following you are doing it wrong:

1. Committing a CI change on a branch up to the remote repository
2. Waiting for your pipeline runner to start
3. Waiting for the jobs to run which come before your changed stage.
4. Viewing the behavior of the changes and the logs.
5. Based on the results go back to step 1 and repeat.

This is incredibly slow as the feedback between your changes will be on the order of minutes or more. Expect to be frustrated when after waiting for your changes to be ran, you find it was a simple mistake, only to go back and make the fix to find out you make another small mistake.

This brings us to our first heuristic for developing CI:

Develop CI locally as much as possible.

This may seem counterintuitive since CI by definition will run on your CI server. But often you can do a majority of your testing locally. If your CI is just a series of shell commands, you should be easily able to run these locally against a test project.

Local Development

When you develop CI locally you will likely run into the below issues.

1. Runner configuration (environment variables, services, secrets)
2. Runner environment (OS, program binaries, networking)
3. Pipeline logic (branches, approvals, dependent stages, promotion)

These issues often just make it so people may simply develop CI by pushing new commits to the CI server and going through the five steps above. In some cases, this is inevitable, but we should do everything possible to develop locally. Lets discuss how to fix each of these in turn:

Runner configuration (environment variables, services, secrets)

In many cases you can export the project level variables into your local environment. At least for gitlab, you can find these variables in the .gitlab-ci.ymlfile or in the project settings or in the group settings. Additionally, gitlab will add in a whole list of predefined environment variables. Depending on the needs of your CI, you will have to export these additionally, or stub them out.

How you handle dealing with secrets depends on your secret-management solution and how much access you have. In some cases, these may just be additional environment variables you need. Consider if you need these or can stub them for testing.

Services can provide additional resources on the network such as sidecar containers or redis, postgres, kakfa. These have an analog of just running docker-compose locally to spin up these resources.

Runner environment (OS, program binaries, networking)

Docker is your best friend here. In general, your CI steps should be using a docker image that packages necessary dependencies anyway, so if you can just use the same docker image that your CI step uses, you should have most of these issues taken care of. Better yet, this means your code developed locally will closely match the environment that it runs within your CI.

In some cases, you cannot simply run a docker image to replicate your CI environment. If you run within gitlab, you can simply run the runner locally and register it to your project. This can further ensure your local environment matches your CI. But this solution is native to gitlab, and has its own issues. Another solution specific to gitlab is to use their pipeline editor which provides a mini-IDE to make CI changes and includes a linter. There may even be local CI development frameworks for your engine, either from the original provider, open source such as gitlab-ci-local, or integration with your IDE.

Pipeline logic (branches, approvals, dependent stages, promotion)

When testing logic with the CI itself, you will likely have to push your changes up to test anyway. We can make this easy on ourself in two main ways. These hold for any CI development where you cannot develop locally.

1. Collapse your pipeline configuration to only test the desired changes. Skip or stub jobs. Move your changes to the earliest stage as possible. Remove any dependencies among jobs that are not required to test your change
2. Create dedicated projects to test CI. Make these as simple as possible so pipelines run quickly, yet thorough enough that they meet CI requirements for real projects.

Both of these together will ensure that your CI changes are not unreasonably slow. You should never have to wait for three stages to run if they do not affect the job you are testing. Testing your CI changes on a non-dedicated CI project adds extraneous commits and pipelines runs that may confuse the developers of the project, but more importantly it will be slower and less flexible than a project dedicated to testing CI changes.

Use a custom, dedicated CI runner image.

One final tip I will leave you with is to consider using a dedicated custom build image. If your CI job has 20 lines of shell scripts, would it be better to package this into a binary, containerize it, then use that container to run your job which would then call your script binary? I see this pattern often in Gitlab’s Auto-Devops CI templates. They use a custom build image and each CI job will have only a few lines of shell scripts which each call a custom command that is contained within the custom build image.

This not only helps to keep your CI DRY, it allows you to fully leverage bash scripting without being concerned that each of your commands fits into a neat line in your CI file. This becomes especially important when you deal with loops and conditionals as often these might work locally, but will have unexpected behavior when you place them directly into .gitlab-ci.yml.

Conclusion

• Do as much local development as possible to test your CI changes instead of committing these changes to the CI server.
• Export CI environment variables locally, use the same docker containers as your jobs, and use docker-compose to replicate services.
• Investigate your CI engine’s tools for doing local development.
• Use a slim, dedicated CI project, and stub out unneeded jobs when testing.
• Consider using a custom dedicated CI runner image.

Continue on to part 2 of this series here.