StackGres and Strimzi: Day-Two Lessons From a Real Platform Rebuild

We were rebuilding a pre-production environment from scratch. Two stateful systems had to move: a PostgreSQL cluster with TimescaleDB extensions handling time-series metrics, and a Kafka cluster processing event streams across multiple pipeline stages.

For both, the question was the same: vanilla StatefulSets managed with Helm charts, or purpose-built Kubernetes operators?

We went with operators. StackGres for PostgreSQL, Strimzi for Kafka. Neither decision was obvious at the start. Several weeks in, both earned their keep, though not in the ways I expected going in.

What operators actually promise

The theory is that operators encode operational knowledge into a controller. You describe what you want, a PostgreSQL cluster with three replicas or a Kafka cluster with SCRAM (Salted Challenge Response Authentication Mechanism) authentication, and the operator reconciles toward that state continuously. Unlike a Helm chart that renders static YAML and walks away, an operator watches and reacts.

The marketing case is Day-1 setup. The real case is Day-2: backups, failover, certificate rotation, replica recovery. That is where the difference between “someone wrote YAML for your StatefulSet” and “someone who understands this system deeply wrote a controller for your cluster” actually shows up.

StackGres: the backup moment

StackGres wraps a Patroni-managed PostgreSQL cluster behind a set of CRDs. You define an SGCluster, and the operator provisions and continuously manages the underlying StatefulSet, sidecars, replication configuration, and connection pooling. PgBouncer is built in. Patroni handles leader election. You declare intent; the operator handles the rest.

Backup configuration is declarative:

spec:
  configurations:
    backups:
      - sgObjectStorage: my-s3-storage
        cronSchedule: "0 2 * * *"
        retention: 7

The SGObjectStorage object defines where backups go. WAL-G handles the actual upload; the operator manages the lifecycle. You do not write backup scripts, cron Jobs, or separate backup Deployments. You describe the policy, and the operator enforces it continuously.

What took us a few hours to discover: the backup upload runs inside the primary database pod, not in a separate backup Job. This matters for cloud credential injection. If you are using pod-native IAM credentials (injecting a cloud identity via the pod’s service account rather than a static key), that annotation has to land on the database pod’s service account. The operator’s allResources field is the correct place to set this annotation, because any annotation applied manually gets stripped on operator reconcile:

spec:
  metadata:
    annotations:
      allResources:
        "eks.amazonaws.com/role-arn": "arn:aws:iam::123456789012:role/pg-backup-role"

After correcting this and rolling the pods, the backup that had been silently failing for days completed. That is the first Day-2 moment: not a setup task, but a failure that the operator’s monitoring surfaced and that the operator’s structure gave us a clean path to fix.

The replica reinit

The second moment was more dramatic.

A replica went into a crash loop. What happened: the replica had lost its pg_control file, could not stream from the primary, and fell through to a fallback behavior specific to StackGres, trying to restore itself from the latest WAL-G backup on S3. The restore timed out. The operator set a failed-backup flag, and the replica restarted every 60 seconds.

With a raw StatefulSet this would mean shelling into the pod, understanding the state of the Patroni member, deciding whether to wipe and resync or run a manual pg_basebackup, and hoping you do not make it worse. With StackGres, the operator’s fallback logic made the recovery path explicit:

# Remove the failed-restore marker; operator falls back to streaming from primary
kubectl exec -n data <cluster>-1 -c patroni -- \
  rm /var/lib/postgresql/replication/initialization-failed-backup
kubectl delete pod <cluster>-1 -n data

The replica came back, streamed from the primary, and rejoined. The operator’s structured fallback, “try S3 first, then try leader streaming,” gave us a deterministic recovery path instead of an open-ended debugging session.

Migrating TimescaleDB across operators

When we moved data from the old deployment to the new StackGres cluster, vanilla pg_restore is not enough for TimescaleDB. The extension has its own restore protocol:

-- Run in the target database before pg_restore
SELECT timescaledb_pre_restore();

pg_restore -Fc --no-owner < dump.pgfc | psql target_db

-- Run after pg_restore
SELECT timescaledb_post_restore();

The source and target extension versions must match exactly; that is the real gate before attempting the restore. We ran the migration as an in-cluster byte-pipe, piping output from a kubectl exec on the source cluster directly into kubectl exec -i on the target, to avoid routing database traffic over a VPN connection.

One detail that slowed us down: StackGres installs PostgreSQL extensions declaratively. A database that relies on uuid-ossp will not restore successfully until that extension appears in the cluster spec:

spec:
  postgres:
    extensions:
      - name: uuid-ossp
        version: "1.1"

Applying the updated cluster spec triggers a live install without a pod restart. The extension is available in minutes.

Strimzi: the version coupling

Strimzi manages Kafka clusters through a Kafka CRD. You declare the cluster, and the operator handles broker placement, configuration, certificate management, and rolling restarts.

The first constraint we ran into was version coupling. Strimzi 1.0, which supports Kubernetes 1.30 through 1.36, ships Kafka 4.x only. Kafka 4.x is KRaft-only, meaning Kafka’s built-in Raft-based consensus, with ZooKeeper gone entirely. If you are copying manifests from a Kafka 3.x deployment, remove the inter.broker.protocol.version and log.message.format.version settings; Kafka 4.x drops them entirely.

This matters less than it sounds for clients. The Kafka wire protocol negotiates versions bidirectionally. A 3.x client library can talk to a 4.x broker without modification.

TLS: why cert-manager, not ACM

For external access, we needed clients to trust the broker certificate without importing a custom CA. The options were a managed certificate service or Let’s Encrypt via cert-manager.

Managed certificates from cloud providers do not work here. They cannot be exported. You can terminate TLS at a load balancer, but you cannot mount a managed certificate into a broker pod. A Kafka broker needs the certificate locally for L4 TLS; a proxy terminating it upstream is not sufficient. cert-manager with Let’s Encrypt was the only path to a publicly trusted certificate that can be mounted in the pod.

One gotcha that cost a few minutes: cert-manager generates PKCS#1 private keys by default. Kafka’s SSL implementation rejects PKCS#1 with algid parse error, not a sequence. The fix is one field:

spec:
  privateKey:
    encoding: PKCS8

After fixing the key encoding and triggering an operator reconcile via the strimzi.io/force-reconcile annotation, the brokers started cleanly.

Certificate rotation that runs itself

The Let’s Encrypt certificate expires in 90 days. cert-manager renews it automatically at day 60 and overwrites the same Kubernetes Secret in place. The Secret name does not change, which means the Strimzi listener reference does not change.

Strimzi hashes the referenced Secret and rolls brokers when the hash changes, one at a time, preserving availability. Clients notice nothing: the host and SAN stay constant, and Let’s Encrypt’s root is already in the default JVM and OS trust stores.

The monitoring posture becomes: watch the cert-manager Certificate object for Ready=False near the renewal date, and watch for a missing CertificateRenewed event. If cert-manager cannot complete the DNS-01 challenge, the brokers keep serving the still-valid old certificate until it expires. You have a 30-day window to notice and fix the renewal before clients see anything.

Compare this to certificate rotation without an operator: generate a new cert, update a Secret, identify which pods need a restart, coordinate the restart to avoid a partition quorum loss, confirm clients reconnected. With Strimzi, that sequence is the default behavior on every renewal cycle.

Broker placement and the AZ deadlock

Kafka brokers need to be on separate nodes. Without required anti-affinity, all three brokers will schedule onto a single node, and that node going down takes the cluster with it:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: strimzi.io/name
              operator: In
              values: ["my-cluster-kafka"]
        topologyKey: kubernetes.io/hostname

Adding anti-affinity creates a subtler problem. If the brokers initially co-locate on one node during cluster bring-up, their EBS volumes provision in that node’s availability zone. When the anti-affinity rule displaces a broker to a node in a different zone, the volume cannot follow. EBS volumes are zone-local. The displaced broker enters a pending state with didn't match PersistentVolume's node affinity.

On a fresh cluster the fix is straightforward: delete the stuck broker’s empty PVC. Strimzi recreates it on reconcile, and this time the volume provisions in the zone where the broker actually scheduled.

For real fault tolerance, brokers should span three availability zones. Two zones is a cost compromise with a known failure mode: an AZ loss can take two of three brokers and drop quorum.

Declarative users and ACLs

KafkaUser CRDs let you manage authentication and authorization as YAML:

apiVersion: kafka.strimzi.io/v1
kind: KafkaUser
metadata:
  name: event-producer
spec:
  authentication:
    type: scram-sha-512
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: "*"
          patternType: literal
        operations: [Write, Create, Describe]

The operator creates SCRAM credentials, stores them in a Secret named after the user, and keeps ACLs synchronized with the live cluster. Revoking a user is a kubectl delete kafkauser. The operator removes the credentials and the ACLs. No broker config to edit, no password to rotate manually.

One behavior to keep in mind: renaming a KafkaUser requires deleting the old CR explicitly. kubectl apply with a new name creates a new user; it does not remove the old one. The old user keeps working with live credentials until you delete the old CR. This is correct behavior, but it can produce ghost users with live SCRAM credentials during a cleanup.

What operators do not fix

Operators reduce toil; they do not eliminate it.

SGInstanceProfile changes in StackGres do not fully propagate. The operator caches sidecar resource requests and can produce a requests > limits conflict when you change CPU or memory. The fix is to delete and recreate the profile so it recomputes cleanly. This is a known behavior, not a bug, but the “just update the YAML” assumption breaks on resource changes.

PVC resize in StackGres requires a specific sequence: update the cluster spec before deleting any pods. If you delete a pod before updating the spec, the in-pod cluster controller reads the old size and tries to shrink the PVC back. That gets a 422, and the controller crash-loops. The window for making this mistake is small but real.

Strimzi’s broker anti-affinity plus zone-local volumes means that a two-AZ node group provides no real fault tolerance. You get isolation from node failures, not zone failures.

The pattern that emerges

After running both operators through an environment rebuild, data migration, and the first weeks of production-like traffic, the pattern is clear. Operators earn their cost, which is real CRD complexity and a new layer of debugging, specifically on operations that are hard to get right once and nearly impossible to get right every time under pressure.

Backup restoration with StackGres. Certificate rotation with Strimzi. Replica reinit after a bad state transition. These are the tasks where “someone who understands this system deeply wrote a controller” is worth more than any amount of well-intentioned shell scripting.

The setup benefit is real but modest. The Day-2 benefit compounds every time something goes wrong and you know exactly what the operator will do next.