Lean Terraform, Part 3: The Layer Around the Code

What the code cannot hold

The first two posts were about structure. Modules hold no facts, data holds all of them, and one state file per environment keeps a mistake from travelling. Do all of it correctly and you get a repo that is small, readable and honest. You also get an infrastructure that can hand you a production outage because of an HTTP header.

Terraform is declarative, so it describes a destination. The cloud is not declarative. It has ordering, propagation delays, eventual consistency and failure modes that raise no error. None of that is a resource, so none of it fits in your code. It has to live somewhere deliberately, or it lives in one person's head and leaves when they do.

Drift

Someone will fix production by hand. At 2am, during an incident, with a customer waiting, they will run something like this:

az containerapp update -g $RG -n backend --set-env-vars SOME_FLAG=true

That was probably the right call at the time. Terraform cannot see it. State says one thing, the running container says another, and the repo now describes a system that does not exist. The next unrelated apply either reverts the fix or does something surprising, and nobody connects it to an edit made three weeks earlier.

Since drift is inevitable, treat it like anything else you monitor and make it cheap to look at:

env-diff-prod: ## Compare prod tfvars env vars vs running containers
	@bash scripts/env-diff.sh prod-containers.tfvars rg-prod-containers-weu

check-prod: ## Dump all prod container app status + missing env vars
	@python3 scripts/check-containers.py rg-prod-containers-weu

make env-diff-prod is a small script and one of the more useful things in the repo, because it answers the question the code cannot: does reality match the file. Writing it is an admission that the two can disagree, and that admission is what makes the answer available in ten seconds rather than during the next incident.

Putting reality back

Once you find drift, do not hand patch it. Recreate the resource from its declared config:

terraform taint 'module.apps["backend"].azurerm_container_app.this'
terraform apply -var-file=qa-containers.tfvars -target='module.apps["backend"]'

The file wins. If the 2am fix was correct it belongs in tfvars, in a PR, where the next person can see it. If it was not correct it should not survive. The running system should not keep a change nobody wrote down.

One trap here, and it is a good example of the theme. Tainting something with an identity attached gives it a new identity on recreation, and a targeted apply will not refresh the role assignments that pointed at the old one. The resource comes back looking healthy and unable to read its own secrets. Taint, then run a full apply, or include the role targets explicitly. None of this is discoverable from the code.

Failures that raise no error

An error tells you where you are. The failures that cost real days are the ones that look like success.

The vault firewall

Container Apps validate their secret references at creation time using their managed identity. If the Key Vault firewall is closed to them at that moment, provisioning does not error. The app is created in a Failed state and the apply finishes.

# Before an apply that creates apps with KV secret references:
az keyvault update --name kv-prod-weu --default-action Allow

terraform apply -var-file=prod-containers.tfvars

# After:
az keyvault update --name kv-prod-weu --default-action Deny

There is no place in a tfvars file for "open this firewall, then apply, then close it". It is a sequence, and Terraform does not model sequences you perform around it. So it goes at the top of the runbook, and the README's first line points at the runbook.

The missing secret that costs 84 minutes

A missing secret in a bootstrap script does not fail fast. It retries, politely, for six minutes. With fourteen missing secrets that is eighty four minutes of a script that appears to be working, followed by a failure whose cause is now an hour and a half in the past.

The fix is thirty seconds of shell, run before anything is created:

# List what the app needs, list what the vault has, diff them.
grep 'fetch_secret' cloud-init/backend.sh | grep -v '#' \
  | sed 's/.*fetch_secret "\(.*\)".*/\1/' | sort -u > /tmp/needed.txt

az keyvault secret list --vault-name kv-qa-weu --query "[].name" -o tsv \
  | sort > /tmp/have.txt

comm -23 /tmp/needed.txt /tmp/have.txt   # anything printed here will hang you later

Verify before you create. Nearly every long provisioning story I have been part of came down to something that was checkable in advance and got checked afterwards instead.

The header

An edge proxy sits in front of the container apps and routes on the Host header. It recognises internal FQDNs only. Someone sets the origin host header to the public domain, because that is obviously what it should be, and the proxy can no longer match the hostname to any app. Every request returns a 404 saying the app does not exist. It does exist, but the routing layer cannot find it under that name.

That was a full production outage, and the change looked correct in review. The knowledge that prevents it is one sentence about how a proxy resolves one header, and there is nowhere in a Terraform module for that sentence to live. It lives in a pitfalls list, which is why it happened once.

Where the knowledge goes

A README that explains terraform init is close to worthless, since the people reading it can read the Terraform docs. What they cannot get anywhere else is the accumulated list of ways this specific system will mislead them.

We keep the split like this:

  README.md         first-time setup only. how to init, how to plan, how changes ship.
                    one loud line at the top: operating the infra? read the runbook first.

  runbook.html      the source of truth for operations.
                    architecture, resource map, day to day commands,
                    secrets, troubleshooting, security model, CRITICAL PITFALLS.

  Makefile          the sharp edges, wrapped. make plan-qa, make env-diff-prod,
                    make health-prod, make logs.

  INVENTORY.md      what actually exists right now, with dates, and what is a
                    cleanup candidate. re-measured, not remembered.

The pitfalls section is the valuable one and it reads like a list of confessions. Every entry earned its place by costing somebody time. Seven hours lost because a proxy silently refused to forward the Redis protocol and the connections hung rather than erroring. An outage from a header. Ninety minutes from a missing secret. Written down, each of those is two sentences and does not happen again.

A runbook also has a shelf life, which is why our inventory carries dates and says which sections were re-verified and which are simply old. A confident document that is stale is worse than no document.

The series so far

Modules are logic and hold no facts. tfvars are data and hold all of them. One state file per environment, chosen at init, so a mistake reaches one environment. And a runbook for everything the code cannot express, because that category is not empty.

The goal was never elegant Terraform. It is that in six months, during an incident, someone tired can open one file, know what is deployed, change one line, and watch a plan that contains only that change.

Part 4 is the case study these rules came from: two real repositories, one with 1,400 Terraform files and four bastion modules, one with no infrastructure code at all.