Lean Terraform, Part 2: State and Environments

Why state gets its own post

Part 1 was about keeping modules free of facts. This post is about the one file that is nothing but facts.

Modules and tfvars are both safe to be wrong, because being wrong means a plan you do not merge. State is Terraform's belief about what exists in a cloud you are paying for, and every destructive thing Terraform will ever do to you comes from state and reality disagreeing.

So the design question for environments is not how to keep them tidy. It is how many environments a single mistake can reach.

No workspaces

Terraform ships workspaces and they look like they were built for this. We do not use them.

A workspace makes the environment a runtime value, so your code ends up containing terraform.workspace == "prod" ? ... : ..., which is the special case from part 1 arriving through a side door. The environment also becomes something you are in rather than something you chose, and the current workspace does not appear in the command you type. You can be one terraform apply away from production with a command line that looks identical to the QA one.

We use one state key per environment, passed at init:

# QA
terraform init -backend-config="key=qa-containers.tfstate" -reconfigure

# Production
terraform init -backend-config="key=prod-containers.tfstate" -reconfigure

The environment is now an init-time decision, it is written in the command, and nothing in the code branches on it. Same modules, same root, different data, different state:

  ENVIRONMENT       DATA                      STATE                    RESOURCE GROUP
  -----------       ----                      -----                    --------------
  QA                qa-containers.tfvars      qa-containers.tfstate    rg-qa-containers-weu
  Production        prod-containers.tfvars    prod-containers.tfstate  rg-prod-containers-weu

                    |____________________________________________________________________|
                                      zero overlap, on purpose

Different resource group, state, Key Vault, image tag, VNet CIDR and deploy workflow. Two environments that share nothing cannot take each other down, which is worth more than the small amount of duplication it costs.

The state key

All of that depends on one string being correct at init. Get it wrong and Terraform does exactly what you asked: read production's state, compare it to QA's tfvars, and write the result over production. Nothing in the tool stops you, because from where it stands this is a normal request.

Two habits handle it. Always pass -reconfigure when switching environments, since without it you can keep the previous backend config and aim at the wrong state. And wrap the commands so nobody types them:

init-prod: ## terraform init for prod-containers state
	terraform init -backend-config="key=prod-containers.tfstate" -reconfigure

init-qa: ## terraform init for qa-containers state
	terraform init -backend-config="key=qa-containers.tfstate" -reconfigure

plan-prod: ## terraform plan for prod-containers
	terraform plan -var-file=prod-containers.tfvars

plan-qa: ## terraform plan for qa-containers
	terraform plan -var-file=qa-containers.tfvars

make plan-qa cannot be pointed at production by a tired person on a Friday afternoon. Four lines of Makefile remove the failure mode.

Changes ship through a PR

A laptop apply has no audit trail, no review and no second pair of eyes on the diff. It also runs whatever is in your working tree, including the debug edit you forgot about.

  edit tfvars / module
        |
        v
  open a PR  ------>  CI runs `terraform plan`, posts the diff as a comment
        |
        v
  read the plan  -->  does it contain ONLY what I intended?
        |             (nothing extra created, replaced, or destroyed)
        v
  merge to main  -->  CI runs `terraform apply`

The review step is not about style. You are checking that the plan contains only your intended change. Terraform will happily bundle a resource replacement you did not ask for into an otherwise innocent diff, and the words to look for are destroy and must be replaced. Reading that diff is the last cheap moment before the change is real.

Production also gets a manual dispatch and an approval gate, while QA deploys on merge. The asymmetry is deliberate, since QA is where you are allowed to be fast.

Using -target

You will eventually reach for this:

terraform apply -var-file=$TFVARS -target='module.apps["backend"]'

It works, and sometimes it is the right call in the moment. But a targeted apply tells Terraform to ignore its own dependency graph because you believe you know better. Occasionally you do. The rest of the time it papers over a graph that is wrong, and each use makes the paper thinner.

Use it consciously, then follow with a full plan to see what you left behind. Targeted applies leave state and reality slightly out of step, which is the subject of part 3.

The lock

Sooner or later a CI job gets cancelled mid plan and every run after it dies with Error acquiring the state lock. This tends to cause hesitation, so it is worth knowing what the lock actually is.

On Azure the state lock is an infinite blob lease on the state file. A cancelled run never released its lease, so the blob stays leased and later runs are refused. The error prints a Lock Info block with an ID, a Who and a Created time.

Confirm it is stale before touching it. Nothing should be actually running, and the Created time should be old or the Who a runner that is long gone:

gh run list --repo YOUR-ORG/infra-terraform --status in_progress

Never unlock while a real run is in progress. Two concurrent writers is how state gets corrupted, which is a worse day than the one you are already having.

# Init with the SAME state key first, then use the ID from the Lock Info block.
terraform init -backend-config="key=prod-containers.tfstate" -reconfigure
terraform force-unlock <LOCK_ID>

KEY=$(az storage account keys list --account-name YOURSTATEACCOUNT \
  --resource-group rg-production-weu --query "[0].value" -o tsv)

az storage blob lease break --account-name YOURSTATEACCOUNT \
  --container-name YOURSTATECONTAINER --blob-name prod-containers.tfstate \
  --account-key "$KEY" --lease-break-period 0

# Verify (want status=unlocked):
az storage blob show --account-name YOURSTATEACCOUNT --container-name YOURSTATECONTAINER \
  --name prod-containers.tfstate --account-key "$KEY" --query "properties.lease" -o json

Both options remove only the lock marker. The state contents and every deployed resource are untouched, so nothing goes down while you do this. After unlocking, run a plan, which is read only, and read the diff before applying anything.

I spell this out because the instinct under pressure is to assume the lock protects something fragile and that breaking it will hurt. It is a lease on a file, and knowing that is the difference between a two minute fix and an hour of hesitation.

How the two rules work together

Modules with no environment knowledge are what make one state file per environment cheap. One state file per environment is what makes a mistake in your modules survivable. Neither is worth much alone.

Part 3 covers what lean Terraform cannot do for you, and where that knowledge has to live instead.