November 19, 2026 · 10 min read
The Upstream Is Archived. The CVE Is Not.
On 18 November, F5 released NGINX 1.30.7 and 1.31.9 with fixes for two unauthenticated remote code execution bugs. Both need HTTP/2. CVE-2026-123007 is a flow-control underflow that hits any NGINX serving HTTP/2. CVE-2026-123633 is a use-after-free that needs HTTP/2 plus proxying, which describes every ingress controller ever deployed.
For most software the next step is boring: read the release notes, bump the chart, roll it out. That step did not exist here. The community kubernetes/ingress-nginx project was archived in March 2026. Its last release, controller v1.15.1, bundles NGINX 1.27.1, and nobody is going to rebuild it. “Upgrade to the latest version” was already done. It closed nothing.
This post walks through what we did instead, in the order it happened: prove what is actually reachable, take the cheap mitigation where it is free, then build and ship a patched controller ourselves, and give that build an end date.
First, find out what is actually exposed
The advisory says “NGINX with HTTP/2”. That sentence covers a lot less of a cluster than the panic suggests, and a lot more than just the ingress controllers.
Find every NGINX, not just the obvious ones. Ingress controllers are the first suspects, but application frontends often ship their own NGINX to serve static files. I ran nginx -v in every running container, in parallel, and kept the hits. The controller image tag tells you nothing about the NGINX version inside it; only the binary does. Our controllers turned out to run NGINX 1.25.5 and 1.27.1, neither of which will ever get a fix.
Then ask whether HTTP/2 is even on. Inside a controller, grep -c "http2 on" /etc/nginx/nginx.conf answers it. In the frontends, nginx -T showed plain listen 80 with no http2, no ssl, sitting behind a load balancer that speaks HTTP/1.1 to its targets. Not exploitable, so rebuilding them on a fixed base image became routine upkeep instead of an emergency.
Where TLS ends decides whether clients can reach HTTP/2. Browsers negotiate HTTP/2 through ALPN during the TLS handshake. If the load balancer terminates TLS, NGINX only ever sees plain HTTP on port 80. If the load balancer passes TCP through, NGINX does the handshake and offers h2. The Service annotations tell you which: a TCP backend with no TLS ports configured on the load balancer means TLS ends at NGINX.
Probe from outside, and probe the bare hostname. An attacker does not need a valid Host header, so test the load balancer address itself, not only your public names:
# Does TLS negotiate h2?
curl -s -o /dev/null --http2 -w '%{http_version}\n' https://app.example.com/
# Does plain port 80 accept h2c? (NGINX 1.25.1+ with "http2 on" does)
curl -s -o /dev/null --http2-prior-knowledge -w '%{http_version}\n' http://<lb-hostname>/
The second probe is the trap. Since NGINX 1.25.1, http2 on also makes the plain listener accept HTTP/2 with prior knowledge (h2c). Our internal controller sat behind a load balancer that ended TLS, so I expected it to be safe. It was not: port 80 answered 2.
The result was three controllers in scope. One internal, with TLS ending at the load balancer and h2c open on port 80. Two internet-facing, with TLS ending at NGINX, one of them serving a partner integration that authenticates with client certificates. Mutual TLS did not help there either: the default server asks for no client certificate, so an attacker gets to the HTTP/2 layer without one.
The stopgap that costs nothing, and its limits
ingress-nginx has a ConfigMap switch that removes every http2 on from the generated config, including the h2c path on port 80:
controller:
config:
use-http2: "false"
It is a reload, not a restart, so no connections drop. Before flipping it, check that nothing uses backend-protocol: GRPC, because gRPC needs HTTP/2 end to end. After flipping it, verify three things: nginx.conf contains zero http2 on, curl --http2 negotiates 1.1, and a prior-knowledge request shows up in the access log as "PRI * HTTP/2.0" 400.
On the internal controller this was simply the right setting. TLS ends at the load balancer without ALPN, so no client ever had HTTP/2 there; h2c on port 80 was pure attack surface. It stays off for good.
On the internet-facing controllers it was a worse trade. Turning HTTP/2 off there is a real behaviour change for real clients. And the same NGINX release fixed a third bug, an infinite loop on TLS 1.3 request bodies (CVE-2026-118899), that has nothing to do with HTTP/2. A switch cannot close that one. Only a patched binary can.
Where NGINX actually lives in the controller image
The obvious idea is “rebuild the controller on NGINX 1.30.7”. It does not work, and the reason is in the repository layout.
The controller image is two builds stacked. The Go controller sits on top of a base image built from images/nginx/, and that base is NGINX 1.27.1 plus 34 OpenResty patches plus a set of Lua modules. Those patches and modules are pinned to 1.27.1. Bump NGINX underneath them and you are maintaining a port of OpenResty, not a security fix.
What does work is backporting. The base build script applies every file in patches/ in name order, so a fix is one more numbered file. nginx.org no longer publishes standalone patch files for advisories, but the fix commits are public in the nginx/nginx repository, and GitHub serves any commit as a patch:
gh api repos/nginx/nginx/commits/<fix-sha> \
-H 'Accept: application/vnd.github.patch' \
> images/nginx/rootfs/patches/<NN>_nginx-1.27.1-CVE-2026-123007.patch
The HTTP/2 fixes were about ten lines each and applied cleanly to 1.27.1. I replayed all patches, upstream’s and ours, on a clean 1.27.1 tree before every build and treated any fuzz or offset in ours as a failure. A patch that applies “close enough” to the wrong lines is worse than one that fails loudly.
One check made the whole approach trustworthy: at the v1.15.1 tag, images/nginx/ is byte for byte what produced upstream’s published base image. So our build reproduces upstream exactly, plus the patches we added, and nothing else.
Making the fork boring to run
A fork is only safe if releasing it is boring. The pipeline I set up does four things that each removed a way to get it wrong.
Native builds per architecture. Each architecture builds on its own runner and pushes <tag>-<arch>, and a final step joins them into one multi-arch manifest. No emulation, so no “works on amd64, crashes on arm64” surprises. It also made the cost visible: the x86 base build took 36 minutes, the arm one 13.
The controller builds FROM the base by digest, not by tag. A tag can move; a digest cannot. The controller you test is guaranteed to sit on the base you just built.
A smoke test that exercises the patched code paths. Not “the container starts”, but HTTP/2 and HTTP/1.1 over TLS 1.2 and 1.3, a 200 KB request body, parallel streams over one connection, a proxied request, and a check that no worker process crashed. If a backport broke HTTP/2 framing, this is where it shows up, not in production.
Tags are never overwritten. A release is a version bump; an existing tag in the registry is skipped unless someone forces a rebuild on purpose. The helper that checks whether a tag exists has to fail on registry errors instead of reading them as “absent”, otherwise one bad API call quietly overwrites a release.
While I was in there, rebuilding the Go controller on a current toolchain and current golang.org/x modules took govulncheck from 19 findings to 0. That is the other half of an archived project: it is not only NGINX that stops getting updates.
One lesson came from running upstream’s unit tests outside upstream’s own Docker wrapper. The CI job runs as a pod inside a Kubernetes cluster, so Kubernetes injects KUBERNETES_SERVICE_HOST into its environment. A test that builds an API server client from the environment picked that up and tried to talk to the real cluster the runner lived in. It failed, which was the lucky outcome:
unset KUBERNETES_SERVICE_HOST KUBERNETES_SERVICE_PORT
Rolling it out without surprises
Override the digest, not just the tag. The base image is only a build stage; what you deploy is the controller image, with the chart of the matching version (v1.15.1 goes with chart 4.15.1). The chart pins upstream’s image digest. Override only the registry and tag, and you get your repository with their digest, and the pull fails:
controller:
image:
registry: registry.example.com
image: ingress-nginx/controller
tag: v1.15.1-patched.2
digest: sha256:<your-digest>
Diff the live release against the values file before anything else. One controller had been installed by hand long ago and later adopted by Terraform. The live release carried settings the values file did not: snippet annotations allowed, the default IngressClass flag, a hand-scaled replica count. The Helm provider does not see that drift, so the first apply from the file would have silently flipped all three. I fixed the values file first, until helm diff showed only the image and labels changing.
Rollback is helm rollback, not a revert pipeline. Reverting through CI takes ten minutes or more. helm rollback <release> <previous-revision> takes one or two. Roll back first, then revert the change through the pipeline so the state matches again.
Baseline every host before, compare after. Before each swap I recorded the HTTP status for every host through a controller pod, and again after. 42 hosts on one controller and 12 on another came back identical, so “nothing changed” was a diff, not a feeling. One trap: probing from a controller pod to its own load balancer fails intermittently, because the load balancer drops hairpinned connections. Probe from outside the target nodes before you call it an outage.
The partner controller rolled out in 55 seconds.
Sixteen quiet minutes
Then the partner sent nothing. Not errors, nothing. For a client that authenticates with certificates, silence right after swapping the TLS terminator looks exactly like a handshake failure.
It was not. Their client had been sitting idle on connections the old pods closed during the rollout, and after 16 minutes it reconnected and carried on with 200s. What let me wait instead of rolling back was this, switched on for a couple of minutes (a reload, no restart):
controller:
config:
error-log-level: info
At info level, a failed handshake logs SSL_do_handshake() failed with the reason. There were none. The only handshake lines were client closed connection while SSL handshaking, with no SNI, from node IPs: the load balancer’s TCP health checks, not a client. Without that log level, both cases look the same, which is to say they both look like nothing.
A fork needs an end date
After that afternoon, all three controllers ran the patched build: HTTP/2 off on the internal one for good, on for the two internet-facing ones, no outage on any.
That does not make NGINX 1.27.1 current. 2026 already had other NGINX fixes that never made it into that version, and each one needs the same triage: does it apply to our config, and does the fix backport cleanly. Patching the headline CVEs closed the emergency, not the gap.
That is why the fork has a retirement plan written down next to it: move the internal traffic to a cloud load balancer’s native routing and the partner path to a load balancer that does mutual TLS itself, then delete the fork. Owning a security fork of an archived project is a reasonable bridge. It is a bad destination, because every future advisory is now yours to read, backport and test, for as long as the fork lives.