Lean Terraform, Part 4: What Sprawl Actually Looks Like

Field notes from two repositories. One had 1,400 Terraform files and four bastion modules. The other had no Terraform at all.

Part 4 of a series on writing Terraform that stays small as the company grows.
  1. Modules are logic, tfvars are input
  2. State and environments
  3. The layer around the code
  4. What sprawl actually looks like (you are here)

Where the rules came from

The first three posts read as though I derived the rules from first principles. In practice they came from two repositories I met within a few months of each other, and in this post we look at both, because the principles are cheap and the evidence is the part that changes minds.

One was a Series B startup's infrastructure repo. Well funded, real engineers, serious product. The other was the company I had just joined, which had no infrastructure code at all. They failed in opposite directions for the same underlying reason.

The first repository

Running tree printed 504 directories and 1,905 files. Around 1,400 of those were .tf, spread across 59 modules. This was one company's infrastructure.

The top level looked reasonable, which is part of why it is worth studying: 

terraform/
  analytics-infra/
  app-infra-setup/
  github/
  infra-setup/
  k8s-setup/
  landing-page-infra/
  modules/

Every folder there is a real thing the company has. The problem appears one level down, in the module list: 

modules/
  bastion/
  <product>-bastion/
  data-extraction-bastion/
  <vendor>/bastion/

  audit-logger/
  analytics-logger/
  control-plane-audit-logger/
  data-plane-audit-logger/

  ci-role/
  ci-role-experiment/
  iam-ci-role/

  application-vpc/
  <product>-vpc/
  <product>-cp-vpc/
  network/

Four bastions, four loggers, three CI roles, one of which is called experiment and lives in the infrastructure repo of a Series B company. Four things that are a VPC.

Nobody sat down and decided this. Each of those modules was a Tuesday where someone needed a bastion slightly different from the existing bastion, looked at the module, saw no way to express the difference, and copied it. Every individual step was locally correct, which is what makes sprawl worth taking seriously. It is built by careful people who have nowhere to put a variation.

The mechanism

When a module holds facts instead of taking them as input, the only way to express a new fact is a new module. That single sentence explains every number above.

A team that wants a bastion in a different account, with a different ingress rule, in a different region, has two options. Either the bastion module takes those as typed variables, giving you one module and four entries in a tfvars file, or it does not, giving you four modules. The repo records which choice was made, four times, for bastions alone.

The counting test from part 1 follows directly. If a concept appears in your module list more than once, it is not a module, it is a variable nobody extracted yet. The word bastion appearing four times is the repo telling you that bastion is data.

What grows on top

Wrappers around wrappers

Modules named -src, and then modules named -wrapper that wrap the -src ones. When a module cannot be configured, the instinct is to wrap it in another module that configures it. This does not terminate, because the wrapper inherits the original problem and now there are two files to read.

473 provider aliases

The repo contains 473 provider aliases. One module's main.tf is 463 lines and opens with nineteen provider "aws" blocks, one per region, plus ten more commented out: 

provider "aws" {
  alias  = "us_east_1"
  region = "us-east-1"
}

provider "aws" {
  alias  = "us_east_2"
  region = "us-east-2"
}

# ... 17 more, then ten commented-out ones

Providers declared inside a module are close to permanent. You cannot cleanly remove that module later, provider inheritance gets strange, and the commented-out blocks record someone trying to stop the growth. A region is a fact. It belonged in a tfvars file and instead became nineteen blocks of code.

count in 123 files, for_each in 34

Index-based resources outnumber keyed ones roughly four to one. Each of those is a future afternoon where removing an item from the middle of a list shifts everything after it and Terraform proposes to destroy and recreate resources nobody touched.

Smaller signals

To be fair to the people who built this: the product worked, customers were paying, and the company raised a Series B on top of this repo. Sprawl does not stop you from succeeding. It taxes every future change, the tax compounds, and eventually the answer to "can we add an environment" becomes "not this quarter".

The second repository

The other one was the company I joined, and there was no infrastructure repo at all. Production was VMs that had been built by hand over time, by people who mostly no longer worked there. They were not badly built. They were built ad hoc, then adjusted, then adjusted again, with the reasoning living in nobody's head.

On the surface this is the opposite of 1,400 files. The underlying problem is the same one. In the first repo the facts were scattered across 59 modules, in the second across a handful of running VMs and some tribal memory. Neither had a place where you could read what was deployed.

Codify first, change second

The instinct when you inherit hand-built VMs is to design the good architecture and migrate to it. Containers, proper networking, the whole thing. Resisting that instinct was the most useful thing I did that year.

We codified the mess first, exactly as it was. VMs, cloud-init scripts, the load balancer, DNS, the vaults, all of it into Terraform with no improvements. The branch was called freeze_main_with_vms, which is an ugly name that says what it does. Fourteen modules, one tfvars per environment, and the ad-hoc machines finally had a written description. Only then did we move to containers. 

  ad-hoc VMs              codified VMs               containers
  (nothing written)  -->  (written, unchanged)  -->  (changed, still written)

                          |_______________|
                          this step ships no
                          user-visible value

Two moves rather than one. The first makes reality legible without changing it, the second changes it. If you attempt both at once and the migration breaks, you cannot tell whether the new design is wrong or whether you forgot a firewall rule the old system had, because the old system was never written down. You need one known-good description before you are entitled to opinions about the next one.

Freezing also does something political that is easy to miss. It turns "the infrastructure is a mess" from an opinion into a diff. Once the VMs were in code, the argument for containers stopped being a matter of taste and became a list of specific things the file said.

The principles that came out of it

Seeing both repos within a few months of each other is what produced the philosophy in this series. Written down while they were fresh, it came to this:

Automation over headcount. Systems that scale without adding people.
Rules over manual edits. Define policies once, apply everywhere.
Brain over toil. Engineers solve problems, machines execute repetitively.
Ownership over handoffs. Small team, no throw-it-over-the-wall culture.

Anti-goal: we will not hire people to manually maintain sprawling configs. If we need to add headcount to maintain infrastructure, the architecture is wrong.

Scaling test: can this infrastructure support 50 developers without changing Terraform? 100? If no, redesign.

The scaling test is the whole series compressed into a question, and it is why access rules live in a tfvars file as group-to-scope-to-role mappings. Adding a hundred developers has to be a group membership. If the answer to what happens at 100 people is "we edit Terraform a hundred times", you have built a job rather than infrastructure.

The anti-goal is the part people skip and it is the honest one. Sprawling infrastructure creates roles. Somebody has to maintain those 1,400 files, that somebody becomes necessary, and necessary people are hard to argue with about whether the thing that makes them necessary should exist. Writing the anti-goal down before anyone's job depends on the answer is the only time it is cheap to write.

The one thing to take

Count how many times a concept appears in your module list. Not lines of code, not module count, not folder depth. Look for the word that appears more than once.

Bastion, four times. Logger, four times. VPC, four times. Each repetition is a variable somebody could not find a home for, and each was created by a competent engineer having a reasonable week. That is why the rule in part 1 is worth being strict about long before it feels necessary. The day it stops being true is not the day something breaks, it is the day someone copies a folder and is right to.