The Node I Thought Was Spare

What Started This

I wanted an offsite backup machine. That was the whole idea. Put a small box at a family member's house, hang a 12TB external drive off it, push backups to it over WireGuard, and finally stop pretending that "backed up to the NAS in the same room" counts as a backup strategy. Fire, theft, or a bad afternoon with a ransomware payload and the entire thing evaporates at once.

I did not want to buy hardware for it. I had two BeeLink mini PCs sitting in the rack, both of them part of the high availability Kubernetes cluster I built across mixed hardware, and one of them, I was fairly sure, was mostly idle. I remembered giving it a couple of jobs. Kubernetes something. Maybe a load balancer? I would go look, move whatever it was doing somewhere else, and reclaim the box.

Reader, it was not mostly idle.

The Plan

1. Work out what the box is actually doing (done, and it was more than I thought)
2. Work out whether you can even remove a node from a three node cluster (done, the answer is "not directly")
3. Build a replacement VM on the other hypervisor (done)
4. Join it, verify, then drain the old one (done)
5. Move the host level services across (done)
6. Fix the three things that broke (done, and two of them were quietly my fault)
7. Wipe the freed box and build the offsite machine (pending, next post)

The Machine That Wasn't Free

My mental model of that node was "Kubernetes control plane, a Postgres replica, maybe a load balancer node." Three jobs, two of which I could probably shuffle elsewhere in an afternoon. I checked it properly instead of trusting my memory, which is the first thing that went right.

kubectl get pods -A -o wide --field-selector spec.nodeName=n1

The real list:

The load balancer guess was wrong. My HA load balancer runs on three completely different machines. What I was thinking of was kube-vip, which serves the Kubernetes API VIP, a different VIP entirely from the one Caddy floats for services. Two VIPs, two purposes, one confused homelabber.

Nine jobs, not three. And note what is in that list: the only replica of sealed-secrets, one of exactly two CoreDNS replicas, a third of my tunnel connectors. This was not a spare box with a cluster membership card. It was a working node carrying a slice of nearly every service I run, and I had it filed in my head as "mostly idle" because the last time I thought about it, in April, it was.

Most of that list reschedules by itself, thankfully. Pods are cattle. The two that do not move on their own are the control plane role and the Postgres instance, and the control plane role is the one that turns this from "drain a node" into "plan a migration."

The Quorum Math That Decides Everything

Here is the thing I nearly got wrong, and it is the whole reason this post exists.

etcd needs a majority to accept writes. Three members means a quorum of two, so you can lose one member and keep running. That is the entire point of running three.

Two members means a quorum of two. You can lose zero.

A two member etcd cluster is not "slightly less redundant than three." It is worse than a single node in a specific and nasty way: you now have two machines that can each independently take the whole cluster down, instead of one. You have doubled your failure surface and kept zero tolerance. It is the worst square on the board.

So "just remove the node and run on two until I get around to it" was never an option. The replacement had to exist first.

The sequence that works is three, then four, then three:

Four members feels wasteful and is a bit odd (an even number buys you nothing over three), but it is a waypoint, not a destination. Crucially, tolerance never drops below where it started. At no point during the migration was the cluster less resilient than the day before.

Which Node Do You Actually Retire?

I had two BeeLinks and assumed they were interchangeable. They were not, and the thing that decided it was storage.

Both nodes use local-path provisioner volumes, which are node local disk with a hard node affinity baked into the PersistentVolume. A pod can reschedule anywhere. Its data cannot follow.

kubectl get pv -o json | jq -r '...'

pg-ha-1            -> node=n2
pg-ha-2            -> node=n1
pg-ha-3            -> node=ha-cp
vaultwarden-data   -> node=n2

There it is. My password vault lives on a 1GiB local-path volume pinned to n2. Retiring n2 would mean hand migrating that data or losing it. Retiring n1 means losing a Postgres replica, which the operator rebuilds from the primary automatically and without me being clever.

So n1 it was, decided by a single line of PersistentVolume node affinity rather than by my vague sense of which box was busier.

That discovery had a bonus prize attached, which is that I now know my password vault is sitting on one node's local disk with no backup anywhere. Small, irreplaceable, and the least protected thing I own. It went straight onto the TODO list, ahead of the photos.

Add Before You Remove

The replacement is a VM on my second hypervisor, which was running one Home Assistant VM and 6GB of its 62GB of RAM. Embarrassing amounts of headroom.

The VM itself is Terraform, cloned from the Ubuntu template:

resource "proxmox_virtual_environment_vm" "n3" {
  name      = "n3"
  node_name = "hypervisor2"
  vm_id     = 141

  clone { vm_id = 9000 }
  cpu    { cores = 4 }
  memory { dedicated = 16384 }
  disk   { interface = "scsi0"; size = 150; datastore_id = "local-lvm" }

  initialization {
    ip_config { ipv4 { address = "192.168.1.43/24"; gateway = "192.168.1.1" } }
  }
}

That disk is 150GB for a reason. A Postgres volume is going to land on this node, and I would rather size it once than resize it under pressure later.

The Version Pin I Almost Missed

My cluster runs Kubernetes 1.35.3. My Ansible playbook installed the packages like this:

- name: Install kubeadm, kubelet, kubectl
  apt:
    name: [kubeadm, kubelet, kubectl]
    state: present

No version. So it installs whatever the repository is offering today. I checked before running it:

Installed: 1.35.3-1.1
Candidate: 1.35.8-1.1

The repo had drifted five patch releases ahead of my cluster while I was not looking. Unpinned, my brand new control plane node would have joined at 1.35.8 alongside three peers on 1.35.3. It probably would have worked. Patch skew inside a minor version usually does. But "probably fine" is not a thing I want to discover about my control plane at 11pm, and more to the point it would have been an accidental upgrade of one node, which is the kind of drift that is invisible until it is not.

The fix is boring and permanent:

- name: Install kubeadm, kubelet, kubectl (pinned)
  apt:
    name:
      - "kubeadm={{ k8s_package_version }}"
      - "kubelet={{ k8s_package_version }}"
      - "kubectl={{ k8s_package_version }}"
    state: present
    allow_downgrade: true

The version lives in the inventory next to the cluster definition, where you bump it deliberately as an upgrade rather than absorbing it by accident.

Three Things That Went Wrong

None of these were in my plan. All three are now written into the runbook, which is the actual product of an exercise like this.

The tag that skipped a step

My playbook has a final play that removes the NoSchedule taint from control plane nodes, because in a three node homelab cluster every node has to run workloads. That play carries its own tag. I ran the join with --tags join. You can see where this is going.

The node joined perfectly. Then I moved the Postgres replica, and it sat in Pending for five minutes:

0/4 nodes are available: 1 node(s) had untolerated taint(s),
1 node(s) were unschedulable, 2 node(s) didn't match pod anti-affinity rules

That message is a complete story if you read it. One node cordoned (the one I was draining), two nodes already holding a Postgres instance so anti-affinity rules them out, and one node tainted. Nowhere to go. The new node was tainted because the untaint play never ran.

Running the tags separately felt tidy and surgical. It was actually just skipping steps.

The certificate the new node had never heard of

Draining n1 moved its pods onto the other nodes, and two of them landed on n3. Those two are an app I build myself, so its image comes from a registry running in my own house rather than from Docker Hub. They would not start:

Failed to pull image "registry.lan/myapp:latest":
  x509: certificate signed by unknown authority

Everything else in the cluster pulls from public registries that Ubuntu already trusts, so those pods came up without complaint. Only my own images come from my own registry, and that registry serves HTTPS with a certificate from my own internal certificate authority. Every other machine here has trusted that CA for as long as I can remember. The node I had built an hour earlier had never heard of it.

One playbook fixed that, and it fixes it for every internal service at once rather than just the registry:

ansible-playbook playbook-step-ca-trust.yml --limit n3

Then it kept failing. Same error, character for character. containerd reads the system trust store when it starts, and it had been running for an hour, so the certificate was sitting on disk where it could not see it. A restart and the pull went straight through.

An error message that does not change after you have fixed its cause is a genuinely nasty way to lose ten minutes.

The load balancer still pointing at a ghost

This is the one with real blast radius, and the one I nearly did not find.

My Caddy load balancer round robins Kubernetes traffic across the control plane nodes:

reverse_proxy 192.168.1.40:30443 192.168.1.41:30443 192.168.1.42:30443 {
  lb_policy round_robin
  health_interval 10s
  health_timeout 5s
  }

That first address is the node I had just removed from the cluster. A third of my upstream slots pointed at a machine that no longer served the port.

Every service still returned 200. Every single one. Because Caddy health checks the backends every ten seconds, noticed the dead one, and quietly routed around it. The system healed over the mistake so smoothly that nothing in my monitoring, and nothing in my browser, gave me any indication that my three way load balancing had silently become two way.

That is the failure mode I find genuinely unsettling. Not the outage that pages you. The degradation that works.

I only caught it because I went looking through the docs for stale references to the old node's address and found it sitting in an Ansible template. If I had trusted "everything returns 200," it would still be there, and I would have found out the next time one of the two surviving backends went down.

Config that references cluster membership is state, and it drifts. The health check is a safety net, not a substitute for the config being right.

What It Looks Like Now

The failure domain spread actually improved. Before, two of my three control plane nodes were BeeLinks sitting on the same shelf on the same power strip. Now the three members live on three separate physical machines. I did not set out to improve resilience, I set out to free up a box, but moving a node from "one of two identical mini PCs" to "a completely different hypervisor" is a genuine upgrade that fell out of the migration for free.

And the Postgres replica rebuild was the least dramatic part of the whole day. Delete the volume and the pod, and the operator notices an instance is missing, provisions a new volume on an available node, clones from the primary over streaming replication, and rejoins the quorum. My actual database is 922MB, so this took seconds. I watched a pod called pg-ha-4 appear on a machine that had existed for about an hour and start serving as a synchronous replica, and felt slightly redundant myself.

What I Actually Learned

Nothing in a three node cluster is spare. Three is the minimum for a quorum. Every node is load bearing by definition, and "I'll just take one back" is not a thing you get to do without building its replacement first. If you want a spare machine, the number you need to start from is four.

Audit, do not remember. My recollection of that node's job was wrong in one place and incomplete in about six others. Thirty seconds of kubectl get pods --field-selector was worth more than everything I thought I knew about my own cluster.

The storage decides which node you can retire, not the CPU. I was thinking about which box was busier. The actual constraint was one line of node affinity on a PersistentVolume, pinning my password vault to a specific machine. Node local storage turns pods back into pets.

A fresh node is not a peer. It is missing every piece of state that accreted onto the others one incident at a time. Certificate trust, package pins, taints. Those are not documented anywhere because they were never a decision, they were just Tuesday. Building a new node is the only reliable way to find them, and writing them down afterwards is the only way to not rediscover them next time.

Watch out for the systems that heal over your mistakes. Health checks, retries, and automatic failover are exactly what you want, right up until they hide a configuration error so completely that you would never know except by reading the config. Redundancy that quietly becomes less redundant is the thing that gets you, because it feels identical to redundancy that works.

The box is free now. The BIOS setting that makes it power itself back on after a blackout has to be sorted before it leaves the house, because at a family member's home there is nobody to press the button and a backup machine that silently stays off after a power blip is worse than no backup machine at all. That, and the offsite build itself, is the next post.

Featured Image Prompt

I used this prompt to generate the featured image.

A dimly lit home server rack photographed at a slight angle, three small mini PCs stacked on a shelf with soft blue and amber status LEDs glowing. One of the three machines is being lifted out by a pair of hands, and where it was there is a faint translucent blue wireframe outline of a replacement machine already materializing in the empty slot. Thin glowing cables connect all the machines, and the cables to the departing box are fading while the cables to the wireframe brighten. Shallow depth of field, moody dark background, cinematic teal and orange color grading, high detail, photorealistic, homelab aesthetic.