Blog

$ cat KUBERNETES .md
· 10 min read

The Upstream Is Archived. The CVE Is Not.

Two unauthenticated NGINX HTTP/2 RCEs landed overnight, and our ingress controllers were exposed. The usual fix, bumping to the latest release, closed nothing: ingress-nginx was archived in March, and its last release ships an NGINX that will never be patched. This is how I proved what was reachable, turned HTTP/2 off as a stopgap, backported the fixes into our own build, and rolled it out without an outage.

kubernetes security devops helm nginx
$ cat AI .md
· 10 min read

My video studio is code. A new benchmark tested agents on the same job.

AgenticStudio is the code behind my YouTube channel: daily Shorts, a Sunday digest, a cartoon debate show and a kit for videos I present myself. No frame comes from a video model. VibeVideoBench just measured frontier agents on code-driven video. Here is how my setup compares, and the one thing no benchmark can tell you.

ai agents video automation benchmarks
$ cat AI .md
· 8 min read

A 30B coding model on a CPU mini-PC. 17 tokens a second, and the prompt is the limit.

I put Qwen3-Coder 30B on the always-on mini-PC that runs my agent harness, with no GPU. It writes at about 17 tokens a second, which sounds usable. The real limit is reading the prompt: a 10K-token context takes about two and a half minutes before the first word. Here are the numbers, the tuning that did not help, and what would.

ai self-hosting local-llm ollama performance
$ cat AGENTS .md
· 7 min read

Why I left Hermes Agent and built my own harness

In July I wrote that Hermes Agent ran the logistics of my digital life. On 18 October I turned it off and replaced it with a small harness of fixed pipelines, one gated door to the models and an allowlist per job. Here is why, what the rewrite cost me, and when I would still pick a general agent.

agents ai self-hosting architecture harness
$ cat KUBERNETES .md
· 7 min read

The Cluster Used 160 Millicores. Its Requests Needed Six Nodes.

Before our first customer I asked whether three medium nodes could hold production. The cluster was using about 160 millicores of CPU, yet its resource requests kept six nodes busy. Here is why requests, not usage, decide node count, why the autoscalers read requests too, and why the capacity of a cluster is not the capacity of the product when the expensive work runs at a vendor.

kubernetes eks finops devops mechanism
$ cat OBSERVABILITY .md
· 8 min read

We Lost 50 Million Spans and Nothing Logged It.

A deploy repointed every service's traces at a new collector that was sized for a trickle. The collector blocked, the services' SDKs dropped the overflow, and roughly 50 million spans vanished over two hours with no counter and no log line anywhere. The only warning was about the wrong thing. Here is how the loss stayed invisible, how I measured something nothing had counted, and why the real bottleneck was not the one I reached for first.

observability opentelemetry kubernetes devops mechanism
$ cat AWS .md
· 8 min read

The Snapshots Added Up to 21 TiB. Deleting Them Freed 687 GiB.

EBS snapshots bill on changed blocks, not on the size of the volume. A merge-heavy database volume can cost more to snapshot than everything else in the account, while a large idle disk costs nothing. Here is how I measured it, why a cleanup that looked like 21 TiB freed 687 GiB, and why a belief I had written down about deleted source volumes was wrong.

aws finops ebs devops mechanism
$ cat GITLAB .md
· 7 min read

The Registry Answered in 45 Milliseconds. The Build Took Two Minutes Longer.

After a server migration, builds against a self-hosted package registry got two minutes slower and every report pointed at the move. Every failing request was a 500 in 45 milliseconds; the entire regression was npm's retry backoff. The root cause was one row out of 63,000 still pointing at an object store that no longer existed, and the check written to catch exactly that iterated a hand-written list of tables and missed it.

gitlab debugging migration reliability observability
$ cat GIT .md
· 7 min read

84 Repositories Vanished. The Fix Was mkdir.

S3 has no empty directories, and in a fully packed git repository the refs directories are exactly that: empty. A file-by-file sync carried every object intact and dropped the two directories git requires to call something a repository, so 84 of 517 came back unreadable while the database still said they had commits. The migration's health checks stayed green the whole time, because none of them ever open a repository.

git aws s3 gitlab mechanism
$ cat KUBERNETES .md
· 7 min read

The Constraint Was Satisfied. The Zone Was Empty Anyway.

A topology spread constraint is only ever a statement about the population its selector matches, and an operator-set label can quietly pool sibling Deployments into one shared count. Three collectors each satisfied their hard zone spread while the union of them left a zone empty, and the standard repair converged on the same wrong answer every time.

kubernetes scheduling opentelemetry finops mechanism
$ cat CLICKHOUSE .md
· 8 min read

136 Million PUTs for 17 GiB of Data

Object storage bills per operation, and a ClickHouse part on an S3 disk is not one object but one per column. So the cost of a cold tier is a function of how many parts exist, not how many bytes they hold, and every setting that starves merges becomes a line on the bill. Two chart defaults did exactly that, and the fix that stopped it had never been committed.

clickhouse s3 finops observability mechanism
$ cat KAFKA .md
· 8 min read

The Error Was 25 Hours Old. The Client Was Healthy the Whole Time.

A Kafka client wrapper that recovers by counting errors and panicking past a threshold is recovery proportional to traffic: stream processors trip it in seconds, quiet request-driven producers never do, and so they latch the last error and serve it indefinitely while every health signal stays green. The tell is in the error's own digits: an elapsed-time value that is byte-identical across occurrences is one cached event, not a recurring failure.

kafka resilience mechanism reliability observability
$ cat DATABASES .md
· 7 min read

LakeBase Didn't Reinvent Postgres. It Moved the fsync.

Databricks LakeBase Postgres is real, wire-compatible Postgres: standard drivers, pgvector, PostGIS, unmodified query engine. The marketing headline is database branching, Git for your database. That is the side effect. The mechanism is that the commit path moved off the machine: a transaction acks when a quorum of safekeepers accepts the WAL record, not when a local SSD flushes. Branching, scale-to-zero, and point-in-time recovery all fall out of that one sentence, and so do the costs.

databases postgres architecture mechanism cloud
$ cat LINUX .md
· 7 min read

Opinionated Linux Stops Being a Contradiction When the Opinions Are Coherent.

Omarchy launched this summer and 37signals is moving the whole company to it on a three-year horizon. The interesting part is not the Linux part; it is that the distribution is the first one I have seen where the personality is the lead feature. The portable rule: the same opinionated default is a feature when its reasoning is visible and a bug when its reasoning is hidden.

linux design opinion product devops
$ cat AI .md
· 6 min read

Speculative Decoding Shipped Because It Doesn't Change the Output.

Liquid AI shipped LFM2.5-DSpark on August 20: three draft models under the lfm1.0 license, with a 2.67× mean speedup on H100 and a 57% function-calling latency reduction on Apple Silicon. The headline number is the least interesting thing about the release. The interesting thing is why this kind of release can ship at all. Speculative decoding is exact under greedy decoding, which means it does not change the output. That property is the whole game.

ai inference speculative-decoding mechanism performance
$ cat OBSERVABILITY .md
· 7 min read

The New Pipeline Seemed to Lose Spans. The Old One Had ~47k Duplicates.

During a parallel-run validation, a side-by-side per-service span count showed the new pipeline losing ~0.2% of spans per service across a 60-minute window. Read as regression in the new pipeline, it would have triggered a rollback. Read correctly, it was the old pipeline ingesting tens of thousands of duplicate rows in a few-second window. The storage layer has no unique constraint on the thing being counted; the unit it exposes as 'count' is not the unit the operator thinks it is.

observability clickhouse verification troubleshooting devops
$ cat DOCKER .md
· 5 min read

The Note Said the Image Was Wrong. The Image Was Right. Three Checks Agreed.

A note in the project's knowledge bundle said the image was amd64-only and might not run on Graviton. Three independent checks agreed. The image was not amd64-only, and a native arm64 build succeeded with zero source changes. A claim that has been verified three times is the most dangerous kind of wrong claim.

docker arm64 verification knowledge-management devops
$ cat KAFKA .md
· 8 min read

I Inferred Three Things About a Live System. Two of Them Weren't True.

Three times in one week I made a claim about a live migration by reading a config file, a name prefix, or a template, and twice the claim was wrong. The artefact and the live system are two views of the same thing, and they can agree for reasons the artefact can't tell you. Only one of the views is the truth.

kafka opentelemetry terraform troubleshooting devops
$ cat TERRAFORM .md
· 10 min read

A Plan Is Green When Two of Three Views Agree. The Third One Is the Truth.

A terraform plan for a managed observability stack showed an incident fix as a pending change, though the fix had been live since the incident and only state was behind. In the days before the file caught up, the same plan would have been green, and the next apply would have reverted the fix. A plan is a diff between two views, but the system has three. Reading all three before you plan is what turns the diff from a guess into a decision.

terraform helm eks state troubleshooting devops
$ cat OPENTELEMETRY .md
· 6 min read

The Config Was Byte-Identical. That Proved the Wrong Thing.

We re-derived a collector config, confirmed it was byte-for-byte identical to the one running in production, and shipped it with a stack of green checks behind it. It crash-looped on startup. Byte-identical is a real property; it just answers a question nobody was asking.

opentelemetry kafka terraform devops troubleshooting
$ cat AWS .md
· 6 min read

Two DNS Layers That Look Identical When They're Both Green

A VPC in one account couldn't resolve names in a private zone owned by another. Peering was healthy, routes were right, the DNS-resolution flag was checked. Every signal was green, and the answer was still NXDOMAIN, because two different DNS layers were wearing the same green light.

aws route53 dns vpc networking troubleshooting
$ cat CDKTF .md
· 6 min read

The Bug I Blamed on the Provider

For weeks a terraform plan showed drift I couldn't fix on an S3 bucket that was already configured correctly. I filed it under 'AWS provider quirk' and worked around it. The provider was innocent. My own code was lying to me before Terraform ever saw it.

cdktf terraform iac typescript devops troubleshooting
$ cat KUBERNETES .md
· 9 min read

The EKS 401 That Wasn't About Credentials

Our CI pipeline was minting a valid EKS token, presenting it cleanly, and getting rejected. The problem was not AWS creds. It was EKS access control, and the failure was hiding in plain sight: the cluster had no access entry for the role that was asking.

kubernetes eks aws devops platform-engineering troubleshooting
$ cat KUBERNETES .md
· 6 min read

The Agent Control Plane: Omnigent on Kubernetes

Databricks open-sourced Omnigent, a meta-harness that wraps any agent harness behind a unified API with policies, sessions, and shared state. If you have deployed stateful services to Kubernetes before, the path to self-hosting it is shorter than you think.

kubernetes devops platform-engineering ai-agents
$ cat KUBERNETES .md
· 10 min read

StackGres and Strimzi: Day-Two Lessons From a Real Platform Rebuild

Every operator promises a simpler setup. The real payoff shows up later, when a replica loses its control file or a certificate is three weeks from expiring. Here is what StackGres and Strimzi actually changed about running PostgreSQL and Kafka on Kubernetes.

kubernetes operators postgresql kafka devops platform-engineering
$ cat KUBERNETES .md
· 6 min read

Helm deploy succeeded. Config never applied.

A green CI pipeline while the live gateway serves 404s is not a test failure. It is a silent reload gap. Here is the cross-release checksum pattern that closes it in helmfile.

kubernetes helm helmfile devops platform-engineering