The new cluster ran 110 pods per node. Production was stuck at 58.

Two EKS clusters, same instance type. The one built this year schedules 110 pods per node. The production cluster, built earlier, tops out at 58 and was hitting that ceiling: plenty of CPU and memory left, no room for pods. The goal sounded like a one-line change: give production the same density. It turned into a new node group, a CNI prerequisite, and a Terraform plan that wanted to do far more than I asked.

Where 58 comes from

By default, EKS derives max pods from networking, not from compute. The VPC CNI gives every pod a real VPC IP from the node’s ENIs, so the ceiling is an ENI arithmetic: an m6g.xlarge gets 4 ENIs with 15 addresses each, and after reserving one per ENI plus two for the system, that lands on 58. The node has 4 vCPUs and 16 GB for 58 pods, which is fine for chunky workloads and terrible for a platform full of small ones.

Prefix delegation changes the arithmetic. Instead of attaching individual IPs to each ENI slot, the CNI attaches /28 prefixes, 16 addresses per slot, and the practical limit stops being the ENI table. The kubelet still needs to be told its new ceiling, and AWS recommends capping it at 110 for anything under 30 vCPUs. That is where the greenfield cluster’s density came from: its node groups were born with prefix delegation on and max_pods = 110 in the launch template.

The podsPerCore detour

My first instinct on a Karpenter-managed cluster had been podsPerCore: 10, which sounds adaptive: small nodes get less, big nodes get more. It is a trap in this setup, twice over. The kubelet applies podsPerCore as a cap on top of max pods, so a 4-core xlarge gets 40, which is below the 58 it already had. And Karpenter computes node capacity from ENI limits and is not aware of prefix delegation, so the scheduler’s view and the kubelet’s view drift apart. A flat maxPods: 110 with prefix delegation enabled is boring and correct. The proof is satisfying to watch on a fresh node:

$ kubectl get node <old-node> -o jsonpath='{.status.capacity.pods}'
8
$ kubectl get node <new-node> -o jsonpath='{.status.capacity.pods}'
110

That was an m6g.medium going from 8 pods, an almost comical default for a 4 GB node, to 110.

You cannot add a launch template to a live node group

Here is the part that makes production different from greenfield. Setting max_pods on a managed node group means the module renders a launch template for it, because the kubelet flag has to travel through user data. But a managed node group that was created without a launch template can never gain one. The EKS API rejects the update outright:

ValidationException: Cannot update launchTemplate

Terraform knows this, and models the change as ForceNew: destroy the node group, create a replacement. On a sandbox that is a shrug. On a production cluster that is every node in the group cycling on Terraform’s schedule instead of yours.

So the move is to not fight it: create a new node group alongside the old one, born with the launch template it needs.

general_arm_v2 = {
  instance_types = ["m6g.xlarge"]
  desired_size   = 3
  min_size       = 2
  max_size       = 20
  max_pods       = 110
  max_unavailable = 1
}

The old group keeps serving traffic untouched. Workloads drift over as the new group scales and the old one is cordoned and drained on a schedule you pick, with a rollback that is just “stop draining.” Spreading the new group across availability zones costs nothing on the instance side, by the way; the only bill is inter-AZ data transfer, the same line item that dominates the VPC peering versus Transit Gateway math.

Enable prefix delegation before the nodes, not after

The launch template is only half the change. The vpc-cni addon has to have ENABLE_PREFIX_DELEGATION=true before the new nodes come up:

vpc-cni = {
  configuration_values = jsonencode({
    env = { ENABLE_PREFIX_DELEGATION = "true" }
  })
}

Get the order wrong and you build a liar: the kubelet advertises 110 slots, the scheduler happily fills them, and the CNI can only actually assign the old ENI-limit’s worth of IPs. Every pod past that line sits in ContainerCreating waiting for an address that will never come. Nothing in the deploy pipeline catches it; it is the same class of failure as a ConfigMap update that never reaches the running pod, where everything is green except the runtime.

One more constraint hides in the instance list: prefix delegation is Nitro-only. Production’s spot group still listed m4.large, a pre-Nitro type from another era, and any node launched on it would silently fall back to the old limits. Old instance types in a mixed-instances policy are exactly the kind of residue nobody audits until a feature quietly excludes them.

The plan carried changes I never wrote

terraform plan came back with 6 to add, 1 to change, 2 to destroy. My change accounted for three: the node group, its launch template, the CNI setting. The rest was a module upgrade from weeks earlier that had been sitting unapplied, and it bundled a migration from the aws-auth ConfigMap to EKS access entries. Nobody had asked for that today, but applying my change would apply it too.

Reading it through was worth the hour. The two destroys looked scary and were safe: they were null_resource provisioners whose config blocks had been deleted from the module, and a destroy-time provisioner only runs if its block still exists, so Terraform would drop them from state without executing anything. The ConfigMap survives, and with the cluster in API_AND_CONFIG_MAP mode nobody loses access.

The creates were the actual landmine. The plan wanted to create an access entry for an SSO admin role, and a check against the live cluster showed that exact entry already existed, created by hand through the console months ago. EKS’s CreateAccessEntry is not idempotent:

ResourceInUseException: The specified access entry resource is already in use on this cluster

The apply would have failed halfway through, with the node group possibly created and the auth migration half-applied. The resolution was to split the blast radius: apply the density change with -target on exactly the three resources that are mine, and turn the auth migration into its own change, one that starts by importing the hand-made entries instead of colliding with them.

What to take from this

Pod density on EKS is a birth property. The launch template, the CNI mode, and the kubelet ceiling are all fixed when a node group is created, so the clean way to change them on a live cluster is to create the next generation next to the current one, not to mutate in place. Like the enableServiceLinks default, the ENI-based pod limit is something most people meet the day it blocks a rollout.

And treat every plan line you did not write as a question, not a formality. The unapplied module bump had been waiting in ambush for whoever ran plan next, and the live cluster held state, hand-made access entries, that no amount of reading HCL would reveal. The plan tells you what Terraform wants to do; only the cluster tells you whether it can.