# The Cluster Used 160 Millicores. Its Requests Needed Six Nodes.

> Before our first customer I asked whether three medium nodes could hold production. The cluster was using about 160 millicores of CPU, yet its resource requests kept six nodes busy. Here is why requests, not usage, decide node count, why the autoscalers read requests too, and why the capacity of a cluster is not the capacity of the product when the expensive work runs at a vendor.

- URL: https://ashkmb.com/blog/node-sizing-requests-not-usage/
- Author: Ashkan Kamyab (https://ashkmb.com)
- Published: 2026-10-27
- Language: en

The question was simple: would three medium nodes hold production? The product had no customers yet, the cluster had been rebuilt the day before, and the node bill looked high for something that was mostly idle. Production had no metrics-server, so before answering anything I installed it as an EKS add-on and looked.

The whole cluster was using about **160 millicores of CPU and 6 GiB of memory**. It was running on six nodes, roughly $245 a month of compute for a workload that would have fit comfortably on a laptop.

The rest of this post is about the gap between those two facts, because it is the single most common misunderstanding about Kubernetes capacity, and it decides what you pay.

## The scheduler never looks at usage

The Kubernetes scheduler places a pod on a node when the pod's **requests** fit into what is left of the node's allocatable resources. It does not read live CPU. It never asks how busy the pod is. A node with 5% real utilization is full, as far as the scheduler is concerned, if the requests on it add up to the node's size.

That is why `kubectl top` is the wrong tool for sizing, and why the output of `kubectl describe node` is the right one:

```sh
kubectl top nodes                                   # what is being used
kubectl describe node <node> | grep -A8 "Allocated resources"   # what is reserved
```

The first number was tiny. The second was what had packed three large nodes, one medium node and two small core nodes. The reservations came from a handful of components, none of them busy:

- the node provisioner (Karpenter) asked for a full CPU,
- a warm-pool placeholder reserved 900 millicores so new nodes would already be warm,
- the CPU embedding server ran two replicas at one CPU and 1.5 GiB each, and idled at about 12 MiB of memory.

Together those requests, not any traffic, decided how many machines existed. The cluster was being sized by what its workloads had asked for in a hurry, not by what they did.

The lesson is blunt: **requests decide node count.** If you want a smaller bill, the lever is the numbers in your manifests, and you can only trust those numbers once you have measured real use next to them.

## The reshape

With the measurement in hand, I picked the shape:

- **Three `m7g.large` nodes, one per availability zone.** Not `m6g`, which is slower for the same money, and not `m8g`, which costs more for CPU we did not need.
- **Cluster Autoscaler instead of Karpenter.** For a cluster this small and this predictable, one managed node group and a simple autoscaler is easier to reason about than a provisioner that picks instance types on the fly. This was a call made while there were no customers; I would not make it blindly on a large, spiky fleet.
- **Embedding server trimmed to 250 millicores and 1 GiB.** It idled at about 12 MiB, so even that is generous.
- **Zone spread on the workloads**, so losing a zone does not take out a whole service.

After the change, the requests fit on two nodes at about 92% CPU and 62% memory, and the third node exists for the zone spread. Scale-up to a fourth node took 75 seconds; scale-down back to three took about 12 minutes. The application answered 200 at every check during the migration.

## The autoscalers read requests too

Here is where it gets interesting, because the same rule that inflated the node count also quietly breaks both autoscalers.

**The Horizontal Pod Autoscaler's utilization target is a percentage of requests.** With a tiny request, 80% is a tiny number:

```yaml
resources:
  requests:
    cpu: 100m
metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80   # fires at 80m of CPU, not 80% of a core
```

A service doing almost nothing crosses 80 millicores immediately, so every service pins at its maximum replica count almost at once. The target looks reasonable in review and means nothing in production.

**Cluster Autoscaler grows the cluster on pending pods, not on busy CPU.** With tiny requests, three replicas of everything fit comfortably on three nodes, so no pod is ever pending, so the autoscaler never adds a node, even while the CPU is pegged. The failure mode is a cluster that is hot and refuses to grow.

Both problems have the same fix: set requests near real use before you turn autoscaling on. Otherwise HPA fires too early and Cluster Autoscaler fires too late, and both are working exactly as designed.

## The trap that costs about $60 a month, silently

One review finding is worth its own paragraph, because it is invisible until you read the bill. Cluster Autoscaler defaults to `skip-nodes-with-system-pods=true`. Any node that holds a non-DaemonSet `kube-system` pod without a PodDisruptionBudget is never scaled down. Two of our controllers, the load balancer controller and the EFS CSI controller, had no PDB.

So a node that scaled up under load and happened to receive one of them would stay forever, about $60 a month per stuck node, with no alert and nothing in the logs. A scale test with a throwaway pod on the new node does not catch it, because the throwaway pod is not a system pod. I changed the module default to `false` and applied it live. If you run Cluster Autoscaler, check this flag and check your system pods for PDBs.

## When the expensive work runs somewhere else

The follow-up question was the natural one: how many users can this serve? The answer has less to do with the nodes than I expected.

The cluster's nodes are nearly idle because the language model runs at a hosted provider. A chat message costs a few Spring hops, one query embedding on the CPU embedding server, and a vector query; the heavy lifting happens on someone else's hardware. So the first walls on capacity are not CPU:

1. the provider plan's rate limits, since one key serves everything,
2. document uploads queuing on the CPU embedding server,
3. fixed replica counts, and
4. the database's maximum capacity setting.

Only after those does compute matter. My rough model, which is an estimate and not a load test, says about 800 to 1,500 concurrent active users on three nodes once autoscaling and the vendor limits are sorted, and a realistic cost around $2,400 a month at 10,000 concurrent users, using a growth rule of three more nodes per 800 users. The per-request CPU in that model is a guess good to about a factor of two, and a load test against the demo environment is the thing that replaces it with numbers.

The point is not the figure. It is that **when the expensive work runs at a vendor, your cluster's capacity is not your product's capacity.** Read the vendor's limits before you scale your own nodes.

## What to keep

- **Requests decide node count, not usage.** Measure real use, then set requests next to it. Compare `kubectl top` with the allocated-resources block, not with your intuition.
- **Check autoscalers against your requests.** HPA's percentage is a share of requests, and Cluster Autoscaler only reacts to pending pods. Tiny requests make the first fire early and the second never fire.
- **Audit scale-down blockers.** `skip-nodes-with-system-pods` plus a missing PDB leaves paid nodes running forever, silently.
- **Find the real bottleneck before sizing.** If the expensive step runs at a vendor, its limits come before your CPU does.
