Making Postgres HA on Kubernetes with CloudNativePG (And the Operator SPOF Nobody Talks About)

The Itch

My wife runs a small property management business and I host the Rails apps that run it. Main app, background job workers, queue database, the usual. All of it sitting on a single PostgreSQL container on one Docker host in my homelab. One machine, one process, one very clear blast radius. When that box rebooted (and it did), her business app was offline until I noticed. Not great when your production system has exactly one customer and that customer lives in the next room.

I'd been putting off HA Postgres because every tutorial stopped at the same place. Spin up a 3 node cluster, delete the primary pod, watch the replica get promoted in five seconds, clap politely, post to Reddit. None of that looked like what would happen when a real machine actually fails. So I decided to build it properly and test it like I meant it.

What follows is the journey. The good parts (picking the right operator, sync replication, data migration), the weird parts (tests that showed no failover at all), and the plot twist I genuinely didn't see coming: the operator itself was a single point of failure sitting on the same node as my primary.

The Plan

1. Pick an operator (done, CloudNativePG)
2. Deploy a 3 instance cluster with synchronous replication, one per node (done)
3. Manage roles and databases declaratively (done)
4. Migrate existing app databases off the legacy single Postgres (done)
5. Run failover tests that actually reflect real failures (done, five scenarios)
6. Fix the "operator is also a SPOF" problem (done)
7. Switch the primary onto the most reliable node (done)
8. Configure backups to object storage (pending)
9. Enable Prometheus PodMonitor (pending, waiting for Prometheus itself)
10. Major version upgrade 17 to 18 as a live exercise (pending, its own blog post)

Picking the Operator

Three operators dominate the conversation. I went with CloudNativePG.

CNPG won on docs, momentum, and first class Kubernetes resources for everything. Cluster, Database, and Backup CRDs. You manage Postgres the same way you manage Deployments. No hidden state, no weird fallback to imperative kubectl for things you'd expect to be declarative.

Sync Replication: The Knobs That Matter

Three fields under .spec.postgresql.synchronous control the behavior.

• method: any is quorum based (any N standbys can ack). first is priority based. For a homelab where all nodes are equivalent, any.
• number: how many standbys must ack each commit. With 3 instances (1 primary, 2 replicas), number: 1 tolerates one replica being down.
• dataDurability: required stops writes if the sync quorum can't be met (RPO=0). preferred silently falls back to async and keeps writing, which means potential data loss if the primary dies next. The one that matches the "zero data loss" marketing is required.

The tradeoff matrix I drew before committing:

I picked any / 1 / required. Tolerates one replica down, blocks writes if both replicas are gone (correct, you can't accept commits you can't replicate), zero data loss on primary failure. Latency cost is about one extra LAN roundtrip per commit. Sub millisecond on my gigabit network.

Building the Cluster

Install was one kubectl command from the upstream release manifest. Operator into cnpg-system, CRDs provisioned, webhooks registered, done. (Spoiler: this is fine for a demo, not fine for production. More on that later.)

The Cluster resource itself was three instances of Postgres 17.9, pinned by SHA digest through a ClusterImageCatalog, synchronous replication configured as above, pod anti-affinity forcing exactly one instance per node. The part that matters:

spec:
  instances: 3
  postgresql:
    synchronous:
      method: any
      number: 1
      dataDurability: required
  affinity:
    enablePodAntiAffinity: true
    topologyKey: kubernetes.io/hostname
    podAntiAffinityType: required
  storage:
    storageClass: local-path
    size: 100Gi

podAntiAffinityType: required instead of the CNPG default preferred. In a 3 node 3 instance cluster there's no valid reason to colocate, and "we tried to spread" degrades the whole point. Full manifest lives in my homelab repo.

Three minutes later the cluster was up. CNPG auto creates three Services which is the nicest part of the whole operator.

pg-ha-rw   ClusterIP   192.168.10.10   5432/TCP   ← always points at the current primary
pg-ha-ro   ClusterIP   192.168.10.11   5432/TCP   ← load balanced across replicas
pg-ha-r    ClusterIP   192.168.10.12   5432/TCP   ← any instance

The app connects to pg-ha-rw.pg.svc.cluster.local:5432. The ClusterIP is stable forever. When the primary moves between pods, CNPG flips the Service's label selector and Kubernetes silently reroutes traffic. No DNS TTLs, no connection string changes, no external load balancer. This is the part of Kubernetes that earns its complexity tax.

Migrating the Data

Two databases to move, both belonging to a Rails app. Existing setup connected as the postgres superuser with a memorable password. I took the opportunity to create a dedicated role instead:

# managed roles live under .spec.managed.roles on the Cluster
roles:
  - name: myapp
    ensure: present
    login: true
    superuser: false
    passwordSecret:
      name: myapp-db-credentials

Plus a Database CRD per database, telling CNPG to create them and own them with the myapp role. Apply, wait three seconds, CNPG has created both databases with the right ownership.

pg_dump took eight seconds for 200 MB of data. Restoring into the CNPG cluster was the one place I hit friction: the container's /tmp is read only so kubectl cp fails. Streaming the dump through stdin works fine:

kubectl exec -i -n pg pg-ha-1 -c postgres -- \
  pg_restore -U postgres -d myapp_production \
  --role=myapp --no-owner --no-privileges < myapp_prod.dump

Table counts matched, row counts matched, ownership was right. Old Postgres stayed running as rollback insurance, because nothing good has ever come from decommissioning the only working copy of your data on the same day you migrate it.

Failover Testing, The Tutorial Path

I deployed a probe pod on a non test node, running a tight loop that hit pg-ha-rw every 200 ms, timestamped each success and failure, logged which backend IP answered. Don't trust kubectl get pods for failover timing. The API cache lies. A continuous probe is the only way to measure real client downtime.

Test A, kubectl delete pod on the primary. Readiness probe fails immediately, CNPG promotes a replica. 4.7 seconds of downtime, 12 failed probes. Textbook.

Test B, kubectl drain on the primary's node. CNPG's PodDisruptionBudget delayed the eviction about five seconds while it promoted a replica first. 6.4 seconds of downtime. Also textbook.

Two tests in, the tutorial is accurate. HA Postgres on Kubernetes works. Applause.

Failover Testing, Where It Gets Weird

Test C, systemctl stop kubelet on the primary's node. This test exposes how HA systems actually reason about health. Kubelet stops, but containerd keeps running, so Postgres stays very much alive.

Result: 50 seconds of downtime, no failover. Same backend IP before and after.

Breaking it down: CNPG decides to promote based on direct Postgres health checks. The process was alive, so CNPG saw no problem. But Kubernetes noticed kubelet stopped heartbeating after 40 seconds (the default node-monitor-grace-period), marked the node NotReady, and the endpoint controller removed the primary from the pg-ha-rw Service. No new endpoint appeared because CNPG hadn't promoted anyone. The probe got 50 seconds of nothing routable. When kubelet came back, the endpoint returned and the same pod resumed serving.

Lesson: kubelet flakiness on an otherwise healthy node causes client downtime without triggering failover. Monitor both layers independently.

Test D, actual reboot of the primary's node. sudo systemctl reboot. I expected fast failover, maybe 10 seconds, because now the Postgres process genuinely dies.

Result: 65 seconds of downtime, still no failover. The old primary pod came back on the rebooted node as primary.

I couldn't explain this so I read CNPG GitHub issues for an hour. Failover on node failure is gated by Kubernetes node-monitor-grace-period, default 40 seconds. A systemctl reboot brings a node back in about 60 seconds, right around that threshold. Kubernetes doesn't mark the node NotReady long enough for anything to escalate, so the pod is never marked for deletion, and CNPG never gets an opportunity to decide. From maintainer Marco Nenciarini (issue #6154):

It is not possible to use CloudNativePG for use cases where a failover time smaller than 40-45 seconds in case of node failure is required.

Expected behavior, documented, tracked for future decoupling. Fine. But the next test is where it got interesting.

The Plot Twist

Before Test E (a real node loss by pulling a power cable), I did one more kubectl get. I noticed something I should have checked on day one. The CNPG operator, the single replica that comes with the default install, was running on the same node as the primary. The node I was about to unplug.

Here is what happens when the primary's node dies in the default install:

1. Primary unreachable
2. Operator (on the same node) also unreachable
3. Kubernetes waits ~40 s, marks node NotReady
4. The operator pod has a default toleration for node.kubernetes.io/unreachable:NoExecute of 300 seconds. Kubernetes won't evict it for five minutes.
5. During those five minutes, no failover decision happens because there's no operator running anywhere
6. Client downtime: 5+ minutes

I pulled the cable. 65.6 seconds of probe failures. No replica promoted. Same pod came back on the rebooted node as primary, not because CNPG decided to wait it out, but because the operator wasn't there to decide anything at all.

This is the moment the whole HA story changes from "Postgres is highly available" to "Postgres is highly available conditional on the operator being scheduled on a node that hasn't failed". That's not HA. That's a coin flip.

The Fix That Actually Works

Run the operator with enough replicas that at least one is always on a healthy node. Plus leader election (built in) and topology spread (has to be configured). The default release manifest installs with replicas: 1. The official Helm chart supports what we need. So I switched.

helm repo add cnpg https://cloudnative-pg.github.io/charts
helm upgrade --install cnpg cnpg/cloudnative-pg \
  --namespace cnpg-system \
  --version 0.28.0 \
  --take-ownership \
  -f k8s/cnpg-operator/values.yaml

The --take-ownership flag in Helm 4 adopts resources previously applied with kubectl. A few resources (webhook configs, monitoring ConfigMap, webhook Service) had server side apply field manager conflicts that Helm couldn't resolve, so I deleted those four manually and let Helm recreate them clean. Existing CRDs and running pods stayed untouched. Zero Postgres downtime through the entire operator migration.

The values file that matters:

replicaCount: 3

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: cloudnative-pg
    matchLabelKeys:
      - pod-template-hash

crds:
  create: true

Two details that matter more than they look.

whenUnsatisfiable: DoNotSchedule is strict spread. Kubernetes refuses to schedule a pod that would violate the spread, rather than "trying" and degrading to colocation. With three nodes and three replicas, this forces exactly one pod per node.

matchLabelKeys: [pod-template-hash] is the detail I almost missed. Without it, during a rolling update the scheduler counts both old and new ReplicaSet pods when evaluating spread, which breaks rolling updates. My first rollout produced a 2-1-0 split because the old pods skewed the math. Adding matchLabelKeys tells the scheduler to only count same-revision pods, so each revision gets its own spread evaluation. Rolling updates now land balanced every time.

The Payoff

Same test as before: pull the cable on the node hosting both the primary and the leader operator. This time the leader was one of three pods and the other two were on healthy nodes.

What happened:

1. Node unplugged. Primary and leader operator both unreachable.
2. Lease on leader can't be renewed. After ~15 s (controller-runtime default), lease expires. Another operator pod acquires it.
3. New leader observes primary unreachable, promotes a replica.
4. Service pg-ha-rw endpoint flips to the new primary.
5. Probe starts getting OK from the new backend.

Total client downtime: 27.9 seconds. Real failover with an actual replica promotion.

Full results:

Without the operator HA fix, hard node failure recovery is 5+ minutes. With it, under 30 seconds. Nothing else I did moved the needle like this single change.

Moving the Primary Where It Belongs

After the tests the primary was bouncing between BeeLinks, which isn't where I wanted it. The BeeLinks are physically less reliable than the Proxmox VM hosting my third Kubernetes node (UPS backed host, longer uptime). Moved the primary over with one line:

kubectl cnpg promote pg-ha pg-ha-3 -n pg

About five seconds of write downtime, clean handoff, both BeeLink hosted replicas resumed streaming from the new primary on a new timeline. Putting the most important role on the most reliable hardware is one of those decisions that seems obvious in retrospect but easy to forget mid project.

The Bonus Finding

After pulling power from a BeeLink, I noticed something unpleasant. After power returns, the BeeLinks do not auto boot. You have to physically press the button. For an HA cluster this defeats the purpose of having three nodes when a site-wide power outage leaves you with two.

The fix is a BIOS setting. Most mini PCs default AC power recovery to "Power Off". Change to "Power On" or "Last State".

What I Learned

• The Kubernetes Service is the failover mechanism, not DNS. The app's connection string never changes. CNPG flips the Service selector, Kubernetes reroutes traffic. Don't overcomplicate with external load balancers.
• Kubernetes' own timeouts bound how fast you can fail over. node-monitor-grace-period is a global setting, not a CNPG knob. Hard node loss recovery is 40 seconds minimum unless you tune that flag.
• The operator is infrastructure too. Running one replica of an HA operator defeats the whole point. Three replicas with leader election and topology spread is not optional for production.
• matchLabelKeys: [pod-template-hash] is the detail that makes rolling updates respect topology spread. Without it, spread degrades on every update.
• Most HA tutorials test the happy path and call it done. "Delete the pod, watch it fail over." That's one scenario out of many and the one least representative of what breaks in production. The useful tests are the ones where things don't work the way you expected.

Postgres is now actually HA. Real failover, real measurements, real story for when someone asks how long an outage would take. The next person who sets this up gets to skip all the parts I didn't know to expect. That's the whole point of writing things down.

Featured Image Prompt

I used this prompt to generate the featured image.

A stylized isometric illustration of a three node Kubernetes cluster, three small server boxes arranged in a triangle on a dark tech grid background. Each box has a small PostgreSQL elephant logo on top. Glowing blue lines connect them showing bidirectional replication. Above the cluster floats a ghostly translucent operator pod with a crown icon, surrounded by two dimmer standby copies on either side. One of the server boxes has a red unplugged power cable, sparks flying off the end. Moody cyberpunk palette, deep blues and electric cyans, single warm orange accent on the sparks. Homelab vibes, minimal, technical, a hint of drama.