I Sat Down to Build an HA Kubernetes Cluster. I Did Not Build an HA Kubernetes Cluster.

I sat down tonight to build an HA Kubernetes cluster. I had two BeeLink mini PCs collecting dust, freshly installed with Ubuntu 24.04, both connected to my network over WiFi. Simple plan: switch them to wired, assign static IPs, and start building.

TLDR: I did not build an HA Kubernetes cluster tonight.

The Setup

I've been running a single control plane K8s cluster with two worker nodes on Proxmox VMs for a while now. It works, but it's all on one physical host. If Proxmox goes down, everything goes down. Not exactly "high availability." And I have real reasons to care about uptime:

• This blog — emir.fyi runs on a self-hosted Ghost instance. When it's down, you can't read posts like this one. Ironic.
• Vaultwarden - my password manager. When it's down, I can't log into anything. My wife can't log into anything. Nobody's happy.
• Paperless - I've been paying a lot of bills (just like everyone else), and this is where I keep all the docs. I often need to go back and confirm I actually paid something because the mail keeps on coming.
• Market Mind - a Rails app I built to help me trade options. It crunches market data, generates signals, and has helped me pull a 70% return over the last two years. When it dies mid-trading-day, I'm flying blind.
• PostgreSQL - currently running as a standalone container on Portainer with zero failover. If that host goes down, both Market Mind and every other service backed by it go with it. I want this on the cluster so K8s can reschedule it to a healthy node if one goes down.

What actually pushed me to do this now rather than "eventually" was hardware failure. One of the ECC RAM slots on my Proxmox server started acting up, the machine restarted on me a few times out of nowhere. Re-seating the DIMMs seems to have fixed it for now, and I've ordered a replacement stick, but "seems to have fixed it" isn't exactly confidence-inspiring. I want my services to survive the next time that machine decides to take an unscheduled nap.

The BeeLink mini PCs were perfect for this. Small, quiet, low power, and already sitting in a drawer. I installed Ubuntu 24.04 on both using WiFi during setup, gave them hostnames n1 and n2 (node 1, node 2. Very creative, I know), mounted them in my server rack, and ran ethernet from the patch panel. The plan was straightforward:

1. SSH in over WiFi
2. Enable the wired interface
3. Disable WiFi
4. Move on with life

I got through exactly one and a half of those steps before things went sideways.

The Quick Way (Don't Do This)

I started with n2. SSHed in over WiFi, saw the wired interface was DOWN with qdisc noop (kernel-speak for "I haven't even initialized this thing"), and in my infinite wisdom decided to do it all in one shot. Replace the entire WiFi netplan config with wired-only and netplan apply.

The SSH session hung. WiFi went down immediately. The wired interface? Also down. On these BeeLinks, the Realtek NIC exists and the driver loads, but it won't auto-negotiate a link until something explicitly brings it up. Netplan killed WiFi before the wired interface had a chance to come alive. No network, no SSH, no way back in.

The machine was sitting headless in a closet. No monitor, no keyboard. Just a blinking power LED and regret.

A normal person would hook up a monitor, keyboard, and mouse. A smart person would have a JetKVM or similar KVM over IP for exactly this scenario (JetKVM, if you're reading this, I'm still waiting on that sponsorship). I am neither of these people. I held the power button, waited, pressed it again, and hoped. It worked. pfSense showed a new DHCP lease on the wired MAC. Got lucky. If the config had been bad or the cable loose, I'd be dragging a monitor across the house like a normal person after all.

The Right Way

With n1, I did it properly. The key insight: never remove your working network connection before verifying the replacement works. Obvious in retrospect.

The approach: add wired alongside WiFi first, use netplan try (which auto-reverts if it breaks things), verify SSH works on the new wired IP, and only then remove WiFi in a second pass.

The transitional netplan config looks like this. Both interfaces active, wired preferred via lower metric:

network:
  version: 2
  ethernets:
    enp1s0:
      addresses:
        - 192.168.1.1.40/24
      routes:
        - to: default
          via: 192.168.1.1
          metric: 100
      nameservers:
        addresses:
          - 192.168.1.1
  wifis:
    wlo1:
      dhcp4: true
      access-points:
        "MyWiFiNetwork-real-network-totally":
          auth:
            key-management: "psk"
            password: "myrealpassword"

The magic command is netplan try --timeout 30 instead of netplan apply. It applies the config but automatically reverts after 30 seconds unless you confirm. If it breaks your connection, you don't have to go find a monitor. It rolls back on its own.

After verifying SSH worked on the wired IP, I removed the wifis block, ran netplan generate to validate, then netplan apply. WiFi went down, wired stayed up. No drama.

Final State

Both machines are on static IPs, wired only, reachable via .localdomain hostnames (set via my DNS server). The networking took the entire evening.

What I Learned

The actual lesson here isn't about netplan syntax or Realtek drivers. It's about making changes to remote systems you can't physically access.

Never remove your working connection before verifying the replacement. Add the new thing alongside the old thing. Test it. Confirm it. Then remove the old thing. This applies to network interfaces, DNS servers, firewall rules. Any change where a mistake means you're locked out. The best part? I already knew all of this. But it's a homelab, it's a local network, and I was trying to move fast. The best part? I already knew all of this. But it's a homelab, it's a local network, and I was trying to move fast. The mental math was simple. If it works, I save five minutes and start building the cluster tonight. If it doesn't, I'm configuring HA Kubernetes some other night. Seemed like a worthwhile bet at the time. I would not take it again.

netplan try exists specifically for this. Use it. It's the difference between "let me verify this works" and "let me go find an HDMI cable."

The HA Kubernetes cluster will have to wait for another night. But at least now when I sit down to build it, both machines will be reachable over a wire, with static IPs like civilized infrastructure.