We Lost 50 Million Spans and Nothing Logged It.

observability opentelemetry kubernetes devops mechanism

The alert that fired was a warning about a saturated export queue. Its description said to expect consumer lag upstream. That was the only signal, and it was pointing at the wrong problem. By the time I understood what had happened, an observability pipeline had spent two hours silently dropping the very data it exists to collect: roughly 45 to 50 million spans, about 600,000 of them billing spans, gone with no counter incremented and no error logged anywhere in the fleet.

The trigger was mundane. A morning deploy repointed every service’s OTEL_EXPORTER_OTLP_ENDPOINT from the old trace path onto a new collector running as a StatefulSet. The new collector was correct in every way except one: it had been sized for a trickle, and the cutover handed it the firehose.

Why the loss left no trace

The new collector’s exporter had a persisted send queue of 100 records and used block_on_overflow. That one setting is the whole reason the incident was invisible.

When the queue fills, block_on_overflow makes the receiver stop accepting data until a slot frees up. The backpressure travels back down the gRPC stream to the service’s OpenTelemetry SDK, and the SDK does what SDKs do when they cannot hand a span off: it drops it. The collector’s own loss counters, the ones you would alert on, all stayed at exactly zero, because the collector never dropped anything. It refused to accept, cleanly, and the loss happened one hop upstream, inside hundreds of application processes, where nothing was counting.

I swept the logs of 230 pods for an export error. There were none. The SDKs drop silently by default; a dropped span is a metric you have to go looking for, not a line that appears in a log. This is the trap worth internalising: in an OTLP pipeline, the loss does not happen where you are watching. The collector is the obvious place to instrument, and the collector is exactly the component that stayed clean through the whole event.

Measuring what nothing had counted

With no counter to read, the only way to size the loss was conservation: compare the spans that arrived against the spans that should have arrived. The tracing backend still had the accepted data, so I pulled per-service span rates for the same wall-clock window on three days, the day of the incident, the day before, and a week before.

Two things fell out of that comparison. First, the shape. High-volume services had lost 45 to 85 percent of their spans in the window, while low-volume services were flat. A uniform outage does not look like that; backpressure does, because the services producing the most are the ones that hit the closed door first. Second, the size: about 25.5 million spans arrived in the two-hour-ten-minute window, against 73.8 million the day before and 77.6 million a week earlier. Scaled by the pre-cutover ratio, that put the loss at 45 to 50 million spans, as an upper bound. The billing path told the same story at smaller scale: 391,000 billing spans exported against roughly a million on both comparison days.

One detail about this method is a standing landmine. The comparison windows have to be explicit half-open ranges, [from, to), with the same boundaries on every day. A sloppy query that matches “hour 10” ends up including two different hours on two different days and quietly reports a difference that is really a window mismatch. I have been bitten by that exact thing before, and it bit again here before I pinned the ranges down.

The bottleneck was not the one I grabbed first

My first explanation was that the 100-record queue was the throughput cap. On a persisted queue the number of requests in flight is bounded by the queue depth, so raising it to 1000 and quadrupling the consumer count was the obvious live patch, and it did help: the per-replica rate went from about 1,000 spans a second to roughly 4,000. But raising the queue past the point where it stopped overflowing bought nothing more, which meant the queue depth was not the real ceiling.

The real ceiling was in how the exporter talks to the broker. The downstream broker is S3-backed and batches writes on a roughly one-second window, and the exporter keeps only one produce request in flight per broker with no knob to change it in the version we run. So each replica could complete about one produce per second, and a span arriving mid-produce simply waited for the next one. The lever was never the queue depth or the consumer count. It was how many spans ride in each produce.

Each record carried a batch of 20 spans, a number inherited from the old path, where records were megabytes each and 20 was already large. But live spans off the SDK are about 1.2 KB, not megabytes, so a 20-span record was tiny and the per-second produce limit translated into a laughably low span rate. Raising the batch to 100 spans per record took the per-replica ceiling from about 4,000 spans a second to roughly 50,000. I stopped at 100 rather than going higher because the sizing has to hold against the worst case, not the average: the largest one-minute-average record over the previous week was 2.8 MiB, and 100 spans keeps a 9x margin under the broker’s 128 MiB record limit, while 200 spans cut that margin to 4.5x and was not worth it.

The satisfying part was the cost check. Bigger batches mean fewer, larger records, and I expected to pay for that somewhere. At the same span rate before and after, the compressed byte throughput, the pod egress, the broker’s object storage request rates and the backend’s ingest were all flat. The broker writes one object per batch window regardless of how many records arrive inside it, so packing more spans into each record is close to free. The fix that mattered cost nothing.

The twist: dropping is not the worst outcome

There is a sharper edge to this than “we lost data,” and it showed up as an anomaly I almost dismissed. After the pipeline recovered, span volume did not just return to normal; it ran well above normal for a while, which made no sense for an event that was supposed to have lost spans.

The cause was one upstream integration whose reader did not drop under backpressure. It blocked. Its own throughput collapsed, its processing lag climbed, and when the collector finally freed up it replayed its entire backlog at its maximum rate, briefly about 2.3x normal volume. For that one data source, the span backpressure had been converted into real business-processing lag, data arriving late rather than not at all.

That is the uncomfortable design question the incident surfaced. “Never lose a production span” and “never delay production data” are not the same goal, and under a backed-up collector they pull in opposite directions. An SDK configured to drop protects the application’s latency and sacrifices telemetry. An SDK configured to block protects telemetry and injects latency into whatever is producing it. Which one is correct is an application-team decision about what that specific service can tolerate, and it is not a decision an observability default should be making silently on everyone’s behalf.

What actually prevents the repeat

The queue resize and the batch change fixed the throughput. Neither of them would stop this from happening again, because the failure was not that a queue was too small; it was that a queue filling up was not something anyone was told about until it had been full for two hours.

So the durable fix was a new critical alert that watches the live exporter queue directly and fires when it stays above 95 percent full for five minutes, per replica, before the SDKs upstream start refusing handoffs. The generic saturation warning that misfired at the start now explicitly excludes this exporter, so the two alerts do not talk over each other.

I also tried to add a day-over-day intake alert, “today’s spans are far below yesterday’s,” and could not. The production metrics store keeps only 48 hours, and traffic has strong weekly seasonality: a Saturday morning legitimately runs at a third of a Friday’s. There is no honest threshold for a rule that cannot see last week. That comparison lives on a dashboard backed by longer-term storage instead, with intake plotted against the same window a week earlier so a human can see a cutover-shaped hole at a glance.

What to keep

  • In an OTLP pipeline, instrument the senders, not just the collector. block_on_overflow moves the loss into the application SDKs, where the collector’s clean loss counters will never see it. Alert on the queue filling, upstream of the drop.
  • A queue depth is rarely the throughput ceiling. When raising a queue past the overflow point buys nothing, the limit is somewhere else; here it was one produce in flight per broker, and the real lever was spans per record.
  • Dropping and blocking are different failures. One loses telemetry, the other delays business data. Choose deliberately per service; don’t let a default choose for you.
  • Size batches against the worst case, not the average. The record limit is a hard ceiling, and the largest record in a week is what has to fit under it.
$ cat KUBERNETES .md
· 7 min read

The Cluster Used 160 Millicores. Its Requests Needed Six Nodes.

Before our first customer I asked whether three medium nodes could hold production. The cluster was using about 160 millicores of CPU, yet its resource requests kept six nodes busy. Here is why requests, not usage, decide node count, why the autoscalers read requests too, and why the capacity of a cluster is not the capacity of the product when the expensive work runs at a vendor.

kubernetes eks finops devops mechanism
$ cat KUBERNETES .md
· 7 min read

The Constraint Was Satisfied. The Zone Was Empty Anyway.

A topology spread constraint is only ever a statement about the population its selector matches, and an operator-set label can quietly pool sibling Deployments into one shared count. Three collectors each satisfied their hard zone spread while the union of them left a zone empty, and the standard repair converged on the same wrong answer every time.

kubernetes scheduling opentelemetry finops mechanism
$ cat KUBERNETES .md
· 10 min read

The Upstream Is Archived. The CVE Is Not.

Two unauthenticated NGINX HTTP/2 RCEs landed overnight, and our ingress controllers were exposed. The usual fix, bumping to the latest release, closed nothing: ingress-nginx was archived in March, and its last release ships an NGINX that will never be patched. This is how I proved what was reachable, turned HTTP/2 off as a stopgap, backported the fixes into our own build, and rolled it out without an outage.

kubernetes security devops helm nginx