As of Dec 2021, this is now outdated. I am no longer using Dendron, nor nomad to deploy the site. I’ll have a follow up shortly.

This website is rebuilt in CI/CD every morning, then redeployed on the workload orchestrator Nomad.

Nomad lacks the “redeploy” function that you’d find in the Rancher API. We need to add a few steps to our DroneCI pipeline to automatically redeploy the built website to our Nomad cluster:

  1. publish a tagged container image (even on periodic rebuilds)
  2. fetch the .hcl spec for the job that will be redeployed
  3. nomad run that job with the newest tag (not latest, unfortunately)
  4. prune old images that were built by DroneCI

Publish a tagged image

First, create a tag when publishing the container we just built:

  - name: publish
    image: plugins/docker
    settings:
      registry: registry.quinncasey.com
      repo: registry.quinncasey.com/garden-quinncasey
      tags:
        - latest
        - ${DRONE_BUILD_NUMBER}

This updates the latest tag, and also adds the unique build number ${DRONE_BUILD_NUMBER} (i.e. 72).

Fetch the job spec

I keep all my job deployments in a separate infra repo. This needs to be pulled into the pipeline:

  - name: clone deployment dotfiles
    image: alpine/git
    commands:
      - git clone -b abathur https://git.quinncasey.com/qcasey/dotfiles

Deploy new tag in Nomad

The job file has a variable awaiting us at image_id. This way we can specify the tag to pull at runtime.

This can also be a UUID or datetime, but those go against Nomad’s philosophy of knowing exactly what build is in production.

These are defined at the top of the job:

variable "image_id" {
  type = string
  default = "latest"
}

and then used in the job’s docker plugin config. Note the force_pull = true, it is required to ensure the latest tag is pulled. It is false by default:

config {
    image = "registry.quinncasey.com/garden-quinncasey:${var.image_id}"
    force_pull = true
    ...
}

Now we’re all set to run this job in our CI/CD pipeline. Another step can be added in the site’s .drone.yml:

  - name: redeploy nomad
    image: multani/nomad
    commands:
      - nomad run -address=http://192.168.1.196:4646 -var=image_id=${DRONE_BUILD_NUMBER} dotfiles/nomad/webserver/quinncasey.com.nomad

This uses a dockerized version of Nomad, pointed at our server, to run the common job spec with our newly created ${DRONE_BUILD_NUMBER} image tag.

The evaluation should succeed, and the allocations will start updating one by one…

Nomad Deployment

Clean old tags from registry (optional)

When creating tags from build numbers you’re bound to clutter your registry quickly. We can add a final step in the pipeline to clean out old image tags:

  - name: prune images
    image: anoxis/registry-cli:latest
    commands:
      - /registry.py -l casey:$PASSWORD -r https://registry.quinncasey.com -i garden-quinncasey --delete --num 10 --keep-tags "stable" "latest"
    environment:
      PASSWORD:
        from_secret: registry_password

This will delete all but the last 10 tags, while also keeping stable and latest.

This requires your registry to have the environment variable REGISTRY_STORAGE_DELETE_ENABLED=true.

It’s also recommended to run a garbage collection on the registry host, since this cannot remove the containers from disk.