The app read REDIS_PORT. Kubernetes had already overwritten it.

I was deploying a Django app to my homelab cluster. Nothing exotic: the app, a Postgres, a Redis, each behind its own Service. Pods came up green, fixtures loaded, /health returned 200. Then I tried to log in and got a bare HTTP 500 with an empty body.

Not just login. The API schema endpoint 500’d too. The health endpoint kept working, static files kept working. It took a few requests to see the pattern: every view that touched the cache failed, and everything that didn’t was fine.

Getting a traceback out of a production-mode pod

The image had DEBUG hardcoded to False, so the response body was empty and the logs showed the 500 without the exception behind it. Instead of rebuilding the image with debug on, I ran Django’s test client inside the pod:

# kubectl exec -it <pod> -- python manage.py shell
from django.test import Client

c = Client(raise_request_exception=True)
c.post("/backend/token/", {"username": "admin", "password": "..."})

raise_request_exception=True is the useful part. Instead of rendering the 500 page, the test client re-raises the original exception straight into your shell. No code change, no redeploy, no debug mode in a running cluster. The traceback came back immediately:

ValueError: Port could not be cast to integer value as 'tcp:'

A port that starts with tcp:. The Redis connection string the app had assembled looked like this:

redis://redis:tcp://10.233.61.42:6379/1

The app builds that URL the way most twelve-factor apps do:

REDIS_HOST = os.environ.get("REDIS_HOST", "redis")
REDIS_PORT = os.environ.get("REDIS_PORT", "6379")
CACHE_URL = f"redis://{REDIS_HOST}:{REDIS_PORT}/1"

REDIS_PORT was set in the pod’s environment. Not by my manifests, not by the chart, not by any ConfigMap. Its value was tcp://10.233.61.42:6379, which is not a port. It is a URL.

Who sets REDIS_PORT to a URL

The kubelet does. When a pod starts, Kubernetes injects a block of environment variables for every Service that exists in the namespace. For a Service named redis on port 6379, every pod in the namespace gets:

REDIS_SERVICE_HOST=10.233.61.42
REDIS_SERVICE_PORT=6379
REDIS_PORT=tcp://10.233.61.42:6379
REDIS_PORT_6379_TCP=tcp://10.233.61.42:6379
REDIS_PORT_6379_TCP_PROTO=tcp
REDIS_PORT_6379_TCP_PORT=6379
REDIS_PORT_6379_TCP_ADDR=10.233.61.42

This is a compatibility layer for Docker links, the pre-DNS mechanism Docker used to wire containers together. Kubernetes adopted the exact variable format in its first release so that containers written for Docker links would keep working. That was over a decade ago. DNS-based service discovery won, nobody writes apps against *_PORT_6379_TCP_ADDR anymore, and the injection is still on by default. The pod field that controls it is enableServiceLinks, and it defaults to true.

The collision is nasty for a specific reason: <SERVICE_NAME>_PORT is exactly the variable name a twelve-factor app would choose on its own. Name your Service redis and read REDIS_PORT in your app, and you have a guaranteed conflict where Kubernetes wins, because the injected environment is there before your code runs. The value isn’t even a number, since Docker links formatted it as a protocol-qualified URL.

My Postgres dodged the same bullet by accident. Kubernetes injected POSTGRES_PORT=tcp://... too, but the app reads DB_HOST and DB_PORT, so the injected variables sat there unused. That’s the frustrating part of this failure mode: whether it bites depends on an invisible naming coincidence between your Services and your app’s config convention.

The fix

Three options, in the order I’d reach for them.

Turn the injection off on the pod spec:

spec:
  enableServiceLinks: false

This is the clean fix. Service discovery through DNS is untouched; redis.namespace.svc still resolves. The only thing that disappears is the legacy variable block, and if your app runs on any cluster built in the last decade, nothing was reading it.

If you can’t touch the pod spec, set the colliding variable explicitly:

env:
  - name: REDIS_PORT
    value: "6379"

Variables declared in the container spec take precedence over the service links, so an explicit value shadows the injected one. This works, but it fixes one collision at a time and leaves the mechanism armed for the next Service someone adds.

The third option is naming discipline: never name a Service so that <NAME>_PORT matches something your app reads. I don’t trust this one. It depends on every future teammate knowing about a Docker-era injection rule while naming a Service, which is exactly the kind of knowledge that evaporates.

One more property worth knowing: the variables are injected at pod start, for Services that exist at that moment. Start the pod before creating the Service and the variable isn’t there; restart that same pod a week later and suddenly it is. A deploy that worked can break on the next reschedule with no change to any manifest. It’s the same class of failure as a ConfigMap update that never reaches the running pod: the deploy pipeline is green, and the breakage only exists at runtime.

What to take from this

Kubernetes still ships compatibility behavior older than most clusters running it, and it stays invisible until it lands in your environment variables. Like finalizers, enableServiceLinks is a mechanism most people learn about the day it blocks them.

Two habits worth adopting. When an app misbehaves in a pod but works everywhere else, print the pod’s actual environment (kubectl exec <pod> -- env | sort) before reading any more code; the answer to “who set this?” is sometimes the platform itself. And for charts you control, consider setting enableServiceLinks: false as the default for any workload that doesn’t explicitly need Docker-links variables, which today is close to all of them.