I deleted the same DNS record twice. The third time, I asked why.

A public endpoint went down. Not a 500, not a timeout; the name just stopped resolving. Deleting one orphaned DNS record fixed it in under five minutes, and I moved on. Three days later the same endpoint died the same way, and the same fix worked again. That recurrence is the only reason I didn’t close the ticket a second time and call it luck: a fix that needs repeating on a schedule isn’t a fix, it’s a symptom, and the actual bug was a DNS ownership record that can never write itself into existence.

The first outage looks like a leftover record

The symptom was a Go service failing every outbound call to the endpoint in about 4.5 milliseconds, too fast to be a real timeout, which is the tell for a DNS resolution failure rather than a backend error. dig on the name came back with a stray AAAA record pointing at the ingress load balancer, and nothing else. No A record at all.

The load balancer itself was fine. Before touching any DNS, I proved the target independently of what the zone claimed:

curl --resolve checkout.example.com:443:<load-balancer-ip> https://checkout.example.com/

A valid certificate and a real application response came back. So the backend was never in question; only the record type was wrong. The AAAA was dead weight, because the load balancer is IPv4-only and had never answered on that address.

external-dns runs with --policy=sync and --registry=txt, which means it owns a name by holding a matching TXT record next to it, and it refuses to touch anything it doesn’t own. The logs explained the rest:

Skipping endpoint checkout.example.com ... AAAA ... because owner id does not
match for one or more items to create, found: "", required: "eks-prod-cluster"

external-dns wanted to publish the A record from the Ingress, but the stray AAAA had no ownership TXT it recognized, so the whole changeset for that name was refused, not just the AAAA but the A too. Under sync, an unowned record doesn’t just sit there as an orphan; it blocks every other create at the same name until a human removes it. Deleting the AAAA let the next reconcile publish the A, and the endpoint came back.

Three days later, the same fix, and a decision to actually look

Same endpoint, same fast failures, same missing A record. Deleting the orphaned AAAA fixed it again. This time I pulled CloudTrail on the hosted zone before closing anything, because a fix that recurs on its own schedule is a sign the automation is regenerating the problem, not that someone keeps breaking the record by hand.

CloudTrail showed external-dns UPSERTing the A record every ten minutes, right up until a single DELETE on the AAAA, followed by silence. Not a one-time mistake, but a loop that ran cleanly for a while and then locked itself out. The question stopped being “what deleted the A record” and became “why does external-dns keep creating a record it can never keep.”

The record that can never own itself

The endpoint sits at the apex of its own delegated hosted zone: its own zone, not just a name inside a bigger one, with the parent zone living in a different AWS account entirely, the same kind of account boundary that makes peering cheaper than a shared transit hub once you’re pricing cross-account traffic instead of DNS writes. That distinction is what breaks the ownership model.

external-dns’s AWS provider expands an Ingress into a dual-stack desired state: an A and an AAAA, even when only the A will ever resolve to anything. The TXT registry disambiguates ownership per record type with a type-prefixed TXT name: for the AAAA on checkout.example.com, that’s a TXT literally named aaaa-checkout.example.com. At an ordinary subdomain that record lands in the same zone as everything else. At a zone apex, it’s a sibling of the apex name itself, which places it one level up, in the parent zone, example.com, which lives in an account external-dns has no access to. The logs say so plainly:

Skipping record aaaa-checkout.example.com because no hosted zone matching record DNS Name was detected

external-dns can create the AAAA record, but it can never write the ownership TXT that would let it claim the record as its own. On the very next reconcile, it reads its own AAAA back with no matching owner, exactly the “unowned record blocks the create” situation from the first outage, except now the unowned record is one external-dns wrote for itself. That trips the same guard, which refuses the whole changeset, including the A. The record that keeps the endpoint alive gets held hostage by a record nobody can ever own, in a zone nobody can ever write to.

This isn’t specific to one zone. It’s a general property of the type-prefixed TXT registry: it cannot own a record at the apex of a zone whose parent it doesn’t control. Only the older, bare-format TXT (the one written without an a-/aaaa-/cname- prefix) lands inside the zone itself, because it isn’t a sibling of the apex; it’s just the apex name with a different record type. A subdomain a few levels deep never hits this, because every ownership TXT it needs lives inside the same zone as the record.

The fix that actually holds

The durable version doesn’t touch the zone at all; it stops external-dns from wanting an AAAA in the first place:

# external-dns chart values
managedRecordTypes:
  - A
  - CNAME

This renders as --managed-record-types=A --managed-record-types=CNAME, replacing the default [A, AAAA, CNAME]. With no AAAA in the desired state, there’s no apex ownership TXT to fail to write, and nothing left to deadlock on. It’s safe here specifically because every backend behind this zone is an IPv4-only load balancer, so AAAA was never serving traffic, only breaking the thing that was.

Two questions were worth answering before trusting that fix, not just guessing at:

Does this break the TXT registry for everything else? No. managedRecordTypes filters which record types get published from Kubernetes sources; it says nothing about the registry’s own bookkeeping. TXT records were never in that list to begin with (the default doesn’t include TXT either), and the registry keeps working exactly as before for the A and CNAME records that remain.

Does external-dns even need TXT ownership? Only under --registry=txt. The alternative that avoids this class of problem entirely is a DynamoDB-backed registry, which stores ownership out of band instead of as sibling records in the zone. That’s a cluster-wide change with a bigger blast radius than one chart value, so it stayed as a noted option rather than something applied on the spot.

What to take from this

The first fix was correct and still wrong. Deleting the orphaned record was the right move both times, and it was never going to be the last time, because it treated the effect and left the cause generating a fresh one every reconcile. The tell wasn’t in the error message; it was in the fact that the same five-minute action was needed twice for the same reason. A bug that reappears on its own, without anyone touching the system in between, means something is producing it on a loop, and CloudTrail is what turns “it broke again” into “here’s the loop.” It’s the same shape of trap as a Helm release that reports success while the running config stays stale: the tooling isn’t lying, it’s just reporting on a layer above the one that actually broke.

It’s also a reminder that ownership systems built on convention, a TXT record that means something because of where it sits and how its name is prefixed, inherit every boundary of the naming hierarchy underneath them. Nothing about the delegated zone was misconfigured. The zone worked exactly as delegated zones are supposed to; it was the assumption that every record’s owner can always live next to it that didn’t survive contact with an apex. Kubernetes ships more of these convention-based defaults than most people notice, in the same way that enableServiceLinks quietly injects environment variables nobody asked for until the day it collides with something real.