Managing kubernetes environments with kustomize

Kustomize is a tool that allows you to customize base kubernetes yaml files and manage yamls between environments.

Comparison to Helm

Kustomize is similar to helm in that it templates yaml between multiple environments. However, Kustomize base yamls are usable as is. Kustomize doesn’t use golang templating for managing configuration so is far easier to build and maintain. Kustomize has no package management system unlike helm. They are really two separate tools. Helm work best for building external charts and Kustomize shines at managing in-house yamls across separate environments.

An example use case

The example here can be found on my github here for you to follow along:

https://github.com/lundbird/KustomizeTutorial

Our example project consists of a deployment across three environments: dev, staging, and production. Our deployment of an nginx webserver will require a deployment.yaml, a service.yaml, and an ingress.yaml to define access to our service from outside our cluster.

Right now our three environments live together under a kube/ folder as kube/dev kube/staging and kube/prod and our job is to deduplicate the code across these environments.

kube/
  -dev/
    -deployment.yaml
    -service.yaml
    -ingress.yaml
-staging/
    -deployment.yaml
    -service.yaml
    -ingress.yaml
-prod/
    -deployment.yaml
    -service.yaml
    -ingress.yaml

Deduping the environments:

We setup our new folder structure:

kube/
  -base/
  -overlays/
    -dev/
    -staging/
    -prod/

where the directories under overlays/ are the original folders under kube/. We will have each of the environment overlays refer to the yamls that are stored in the base folder, so they can overlay their configuration overlay on top of the base yamls. Each folder will need a kustomization.yaml to define how to perform the merging of the base and environment yamls.

Next we need to extract all shared configuration between each environment and place them into the base folder. To do this easily I recommend copying the dev environment into base, then opening each yaml in vi with the -d option to show you the differences between each environment.

vi -d ../base/deployment.yaml dev/deployment.yaml staging/deployment.yaml prod/deployment.yaml.

Using vim, we can see that each environment lives in a separate namespace, and has a unique ENV variable. Prod also uses a hashed image version and an additional replica. Remove all the environment specific information from the base overlay. The labels differ for each environment. Just remove these for now.

Next we look at the service using the same command as above but replacing service instead of deployment.

We see that each environment is the same except for the selectors. Remove the labels. We will add these as a commonLabels to our kustomization.yaml

When you do the same with the ingress, you will find that it has a different hostname. Just remove the ingress file from each environment. We will use a json patch to patch in the different hostname to the base yaml

Now create a kustomization.yaml file in each folder. For our base folder, we want to tell kustomize to include each yaml in this folder.

resources:
- deployment.yaml
- service.yaml
- ingress.yaml
- configmap.yaml

Next we want to modify our dev kustomization.yaml as follows:

resources:
- ../../base
patchesJson6902:
- target:
  group: extensions
  version: v1beta1
  kind: Ingress
  name: nginx
  path: ingress_patch.yaml
commonLabels:
  app: dev-nginx
namespace: dev
configMapGenerator:
- name: nginx
  behaviour: merge
  literals:
  - ENV=dev

There is a lot going on here. In order:

“resources” refers to the base directory which we will be overlaying on top of.

“patchesJson6902” defines the patch we we be adding to our base, in this case a patch to overwrite the original hostname. We will need to add our patch under ingress_patch.yaml:

- op: replace
  path: /spec/rules/0/host
  value: dev-nginx.mydomain.com

“commonLabels” will create labels to all of our deployments, pods, services, and deployment and service selectors in this app.

“namespace” defines which kubernetes namespace to deploy the app.

“configMapGenerator” is a way for us to manage all of our configuration variables outside of our deployments. We will create a configmap with all of the shared environment variables in base, then merge in our environment specific values in our configMapGenerator. You will need to update the following to your base deployment to import the configs:

containers:
- name: nginx
  envFrom:
    configMapRef:
      name: nginx

as well as create a configmap.yaml in base:

apiVersion: v1
data:
  DB_HOST: postgres
  DB_HOST: "8081"
kind: ConfigMap
metadata:
  name: nginx

To see the result of the overlay run kustomize build.

>>>kustomize build overlays/dev
apiVersion: v1
data:
  DB_HOST: "8081"
  ENV: dev
kind: ConfigMap
metadata:
  annotations: {}
  labels:
    app: dev-nginx
  name: nginx
  namespace: dev
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: dev-nginx
  name: nginx
  namespace: dev
spec:
  ports:
  - name: http-app
    port: 80
  selector:
    app: dev-nginx
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: dev-nginx
  name: nginx
  namespace: dev
spec:
  metadata:
    labels:
      app: nginx
  selector:
    matchLabels:
      app: dev-nginx
  template:
    metadata:
      labels:
        app: dev-nginx
    spec:
      containers:
      - image: nginx:latest
        imagePullPolicy: Always
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 10
        name: nginx
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 10
        resources:
          limits:
            memory: 500Mi
          requests:
            cpu: 100m
            memory: 100Mi
      imagePullSecrets:
      - name: gitlab
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0
    ingress.kubernetes.io/whitelist-x-forwarded-for: "true"
  labels:
    app: dev-nginx
  name: nginx
  namespace: dev
spec:
  rules:
  - host: dev-nginx.mydomain.com
    http:
      paths:
      - backend:
          serviceName: nginx
          servicePort: http
        path: /

Next we will do the same for prod. Our kustomization.yaml looks very similar except for two additional fields:

replicas:
- name: nginx
  count: 3
images:
- name: nginx
  newTag: 6b654da03c0e
namePrefix: prod-

These will update our image to be tagged to a hash of nginx, increase our replica count, and prepend our resource names with prod-

Again run kustomize build to determine if the results are as you expect.

Final Thoughts

Kustomize has a lot of features, only a small fraction of which I have gone through today. Kustomize has some good tutorials on its gettings started page here: https://github.com/kubernetes-sigs/kustomize/tree/master/examples. Additionally for more detailed kustomize documentation use the kubectl docs here: https://kubectl.docs.kubernetes.io/pages/examples/kustomize.html

The two tools I use the most in kustomize is the JsonPatch and StrategicMergePatch because they can work on any environment. However, oftentimes you don’t need them. You can often place your differences within the kustomization.yaml itself as shown above for images, commonLabels, and replicas.

Kustomize can really shine in creating and referring to a a single source of truth. In the above example, we used a whitelist of 0.0.0.0/0, and referred to it in each overlay, but this example can be extended where you keep JsonPatches of whitelists in a separate repository. Then you can refer to these JsonPatches when you build your own ingresses. The same can also be done for whatever centralized patches that you can plug and play.