A Plan Is Green When Two of Three Views Agree. The Third One Is the Truth.

terraform helm eks state troubleshooting devops

A terraform plan for a managed observability stack came back with one change: an incident fix, shown as if it were about to be introduced. The on-call had applied that fix by hand during an incident two weeks earlier. The fix was live. The fix was in the file. State alone was behind, so the apply was a harmless catch-up. The dangerous moment had already passed. In the days between the hand fix and the file update, state and file agreed with each other and not with live, and a plan in that window would have said “no changes,” indistinguishable from “no drift.” The next apply in that window would have reverted every incident-mitigating line in a single shot. This is a story about a green plan that would have hidden a real problem, and a non-empty plan that changed nothing running.

A plan is a diff. The system has three views

A typical plan is a diff between two things: state and file. “State” is the last successful apply; “file” is what you would write if you were starting now. The plan renders unchanged whenever state and file agree, which is the green every operator learns to want.

The system has three sources of truth, and only two of them are in the diff. Live is the running thing, the cluster or the workload or the storage bucket. State is what Terraform or Helm recorded last. File is what the repo currently says. Each is authored separately. Each can drift. A plan ignores the first one.

The three sources line up in four shapes, and the plan compares only two of them. All three agree because all three came from the same file: the canonical case, which a green plan correctly reports. State is behind while file and live have already caught up: the plan shows a change that is already true at the API, and the apply is a catch-up that changes nothing running. File is behind while live and state agree (the “I applied it from my branch and never merged it” case): the plan shows a change toward the file, which on apply is a revert. Live is the odd one out while state and file agree (a hand fix not yet in the repo, or the “the platform reverted my change because it crashed” case): the plan renders green, identical to no drift, and that is the case that hides the most dangerous drift.

Two views agreeing with each other while disagreeing with the third is the case a plan cannot classify for you. When the two that agree are state and file, it renders green and papers over live entirely. A plan is a diff over a comparison between two things, and the third thing does not get a vote.

The first exhibit: the managed observability stack

It was the middle of an incident. The on-call ran hand helm upgrades against the live release; the values landed in-cluster, but the file in the repo was not updated. The incident closed, the team caught up later that week and updated the file to match, and on the surface everything reconciled. Then the planning phase of the next change came around, and terraform plan was not empty. It showed the incident fix itself, the old values on the - side and the fix on the + side, as if the fix were about to be introduced.

Read against the three views, that diff was nothing to fear. The - side was state, written by the last apply before the incident. The + side was the file, which now carried the fix. Live is not part of that diff, and live already held the fix, because the hand upgrade had put it there. Live and file agreed; state alone was behind. The apply would write into the cluster what was already running and bring state up to date. None of that was the plan’s job to know. The plan’s job is to compute the diff between state and file, and it did: this file wants to apply this fix. Only the third read says the fix is already there.

The danger was earlier, in a window that had already closed. Between the hand upgrade and the file update, state and file agreed with each other: both held the pre-incident values. Live alone held the fix. A plan in that window renders “no changes,” which is exactly what no drift looks like. And the next change applied in that window, any change, re-renders the release from the file, so its plan shows only the new edit while its apply rolls back every line of the incident fix in one shot.

That window is the bug. It is easy to miss because the plan in it is the dream: green, or showing just the change you meant to make. The postmortem would have read “apply reverted the production fix during the next change,” and would have been right, and would have been the most expensive possible framing of a misunderstanding the plan itself could not have caught.

The second exhibit: the managed Kubernetes cluster, a day later

Same mechanism, different surface. The repo directory for a managed Kubernetes cluster carried one real edit: a node group resize, general_arm from 6 to 4, a real drain of two nodes, deliberately requested. That was the change everyone was talking about. It was not the only thing in the plan.

Three other node groups on the same cluster were state-behind. A code sync had landed a week earlier and no apply had run since. Live already matched the file; only state still carried the old numbers. So the plan showed three more update in place changes to the groups’ scaling config, each one already true at the API. Applied, they would change nothing on the cluster and bring state up to date.

The risk was not a revert; it was in the reading. The general_arm drain drew all the attention, which is exactly the point. Operators tend to read the diff the change is about, check that one for safety, and wave the rest through. Here the rest happened to be harmless, but nothing in the plan said so; that came from reading live. And the apply writes four node groups, not the one the change description names. Read the other way round, three reconciles that look just like the drain invite you to file the whole plan under catch-up, and the one line that removes two nodes hides among them. Had one of the three quiet groups not matched live, the postmortem would have read “the apply dragged along unintended changes,” and would have blamed the loud diff instead of the three quiet ones.

Both exhibits walk the same mechanism. At the moment each plan ran, live and file agreed and state was behind, and the plan rendered that lag as a change that was already true at the API. That shape is safe to apply, and from the plan alone it is indistinguishable from a real change, which is how a real drain can hide among catch-ups. The dangerous shape is its mirror, the one the first exhibit had passed through days before: state and file agreeing while live has moved, which renders green and lets the next apply revert live. The plan never tells you which shape you are in; only the third read does.

Why a plan can’t see it

terraform plan consults state and file. It does not consult live at apply time. helm template does not consult live at plan time. Neither tool was designed to verify the live system; both were designed to verify what would change given the last known state. The three-way merge the platform does not do is the one you have to do yourself, before the plan, not as part of it.

The shape of the limitation is consistent across tools. Whatever is recorded somewhere outside the platform’s model is the thing the plan cannot see. For Terraform, that is the live API. For Helm, that is the live release that helm template does not query. For kubectl diff against a server-side apply, that is whichever admission controller last modified what, which the local dry-run never sees. The plan tool and the platform are paying attention to two different sources; the third source is whichever one the user forgets to mention.

A plan that comes back clean is not a statement that live matches file. It is a statement that state matches file, which is information the operator needs to convert into a different question, the question of what live is doing that neither knows about.

The way to actually read it

Three cheap reads against the three sources, in order: state from the backend, file from disk, live from the API. Lay them out side by side. The case where any two agree and one differs is the case a plan cannot classify for you: if state is the odd one out, the plan shows a change that is already live; if live is, the plan renders green. The case where all three agree is the case a green plan correctly reports. Distinguishing them costs a state pull and a read, no apply, no review latency.

For Terraform:

# Read 1: state
terraform state pull
# Read 2: file
terraform plan -no-color
# Read 3: live
# Whatever the platform surfaces (e.g., aws eks describe-cluster ...)

The point is not the commands. The point is the three reads together answer “which view is behind,” which is the question a single green plan cannot.

For Helm:

# Read 1: state
helm get values <release> -n <ns>
# Read 2: file
helm template <release> <chart> -f values.yaml
# Read 3: live (after dry-run)
helm diff upgrade <release> <chart> -f values.yaml

Each pair of reads can be cross-checked by eye. Whenever the three reads do not all match, pause: the one that disagrees with the other two is the view that is behind, and if all three differ, someone made a change nobody wrote down. The reads take longer to set up the first time than to run; after the first time they are a script, and the script is a check that runs before every plan, and the check is what turns the green diff from a guess into a decision.

The most valuable moment to run this is the moment you are about to call a plan “nothing to do” after recent hand-fixes to the same system. That is the only moment a green diff is more likely to be wrong than right, and it is the only moment where reading the third view is cheap. Cost is fixed; timing is everything.

The rule and the twin

Read all three sources before you plan; the plan is a diff over the gap, not a statement of truth.

The first exhibit’s contribution is that the dangerous plan was the green one, in the window before the file caught up: “nothing to do” would have been the wrong read, and the cost of being wrong was reverting the fix that kept the stack standing. The second exhibit’s contribution is that one real diff among three already-true ones gets either all the attention or none, and operators tend to read the diff the change is about. They are the same shape because the mechanism is the same; what differs is the surface, and which pair agreed at the moment you looked.

The next time a plan comes back unremarkable after a hand-fix or an out-of-band edit on the same system, the question is not “is this right.” The question is which of the three views is the one that’s behind. The answer is whatever the third source says, and reading it takes less time than writing this post did.

$ cat KAFKA .md
· 8 min read

I Inferred Three Things About a Live System. Two of Them Weren't True.

Three times in one week I made a claim about a live migration by reading a config file, a name prefix, or a template, and twice the claim was wrong. The artefact and the live system are two views of the same thing, and they can agree for reasons the artefact can't tell you. Only one of the views is the truth.

kafka opentelemetry terraform troubleshooting devops
$ cat OPENTELEMETRY .md
· 6 min read

The Config Was Byte-Identical. That Proved the Wrong Thing.

We re-derived a collector config, confirmed it was byte-for-byte identical to the one running in production, and shipped it with a stack of green checks behind it. It crash-looped on startup. Byte-identical is a real property; it just answers a question nobody was asking.

opentelemetry kafka terraform devops troubleshooting
$ cat CDKTF .md
· 6 min read

The Bug I Blamed on the Provider

For weeks a terraform plan showed drift I couldn't fix on an S3 bucket that was already configured correctly. I filed it under 'AWS provider quirk' and worked around it. The provider was innocent. My own code was lying to me before Terraform ever saw it.

cdktf terraform iac typescript devops troubleshooting