Menu
Article

Building a High Availability Kubernetes Cluster Across Mixed Hardware (Part 2: Migration and the Graceful Goodbye)

Emir Ibrahimbegovic

Emir Ibrahimbegovic

• 15 min read
Building a High Availability Kubernetes Cluster Across Mixed Hardware (Part 2: Migration and the Graceful Goodbye)

Where We Left Off

Part 1 ended with a shiny new 3-node HA Kubernetes cluster: one Proxmox VM and two BeeLink mini PCs, glued together with kubeadm, kube-vip, and Flannel. The control plane was HA, the etcd quorum was solid, and the VIP survived any single-node failure.

The cluster was also empty.

I closed that post with "the hard part is done, the easy part is next, famous last words." Reader, I was not entirely wrong. Migrating the workloads was significantly easier than building the cluster. But "easy" is doing a lot of work in that sentence, and there were a few moments where I questioned my life choices.

This is the story of moving three stateful services from a single-master cluster that was one bad RAM stick from oblivion, to a new HA cluster running on physically separate hardware. Without losing data. Without extended downtime. While running both clusters in parallel and keeping the option to roll back at every step.

The Plan

  1. Install cluster foundations (Traefik, local-path-provisioner, Sealed Secrets, TLS) (done)
  2. Migrate Vaultwarden (stateful, highest-value) (done)
  3. Migrate ArgoCD (the GitOps controller that runs everything else) (done)
  4. Migrate MarketMind (the Rails app with Postgres) (done)
  5. Fix the CI/CD pipeline that SSH'd to the old master (done)
  6. Retarget the HA LB so vault.localdomain, argocd.localdomain, and market-mind.localdomain land on the new cluster (done)
  7. Shut down old cluster VMs, let them soak for a week (done)
  8. Terraform destroy the old VMs after the soak (pending)

The Core Pattern: Parallel Clusters, Flip DNS

The thing that made this migration bearable: both clusters run at the same time, and the HA load balancer (Caddy) decides which one handles any given hostname. Flipping vault.localdomain from old to new is a one-line change in the Caddyfile and a playbook re-run. Flipping back is the same change in reverse.

Every migration step followed the same shape:

  1. Stand up the new service on the new cluster with the migrated data
  2. Test it via kubectl port-forward (no DNS change, no blast radius)
  3. Flip the Caddyfile to point the real hostname at the new cluster
  4. Test from a real client
  5. If anything is off, flip back, debug, try again

This feels obvious in retrospect, but the previous me would have been tempted to do "one big cutover at midnight" and spend an hour panicking. Parallel clusters turn "cutover" into "routing decision" and routing decisions are cheap.

Laying the Foundation

Before any real workloads, I needed four things on the new cluster.

Traefik

Same version as the old cluster (traefik-39.0.0, Traefik v3.6.7), installed via Helm. Then a small NodePort service to expose 80:30080 and 443:30443 so the HA LB's Caddy can reach it:

apiVersion: v1
kind: Service
metadata:
  name: traefik-nodeport
  namespace: traefik
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: traefik
  ports:
    - name: web
      port: 80
      targetPort: web
      nodePort: 30080
    - name: websecure
      port: 443
      targetPort: websecure
      nodePort: 30443

Test: curl -sk https://192.168.1.40:30443/ on every cluster node. Got 404 on all three, which is exactly right. No IngressRoutes yet, so Traefik has nothing to route. 404 means "I'm here, I just don't know that hostname" and that's the healthy answer.

local-path-provisioner

Rancher's local-path-provisioner v0.0.30, same version as the old cluster. Creates a local-path StorageClass that turns node-local disk into PVCs. Fast, simple, and the honest truth that homelab storage is usually pinned to wherever the pod first scheduled.

Tested with a throwaway PVC and pod that wrote a file, then read it back. PVC bound, file present, done. Next.

Sealed Secrets (the part where key management matters)

Vaultwarden and MarketMind both use Bitnami Sealed Secrets so their secrets can live in git encrypted. The controller encrypts with a public key and decrypts with a private key that lives only in the cluster.

The catch: if you install a fresh Sealed Secrets controller on a new cluster, it generates a new private key. Existing sealed secrets in your repo, encrypted against the old key, will never decrypt. They just sit there looking broken.

The fix is to restore the old keys before installing the controller:

# 1. Export all sealed-secrets keys from the old cluster
kubectl --kubeconfig ~/.kube/config-old \
  get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > keys-backup.yaml

# 2. Apply them to the new cluster BEFORE installing the controller
kubectl --kubeconfig ~/.kube/config-ha apply -f keys-backup.yaml

# 3. Now install the controller
kubectl --kubeconfig ~/.kube/config-ha apply \
  -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.27.3/controller.yaml

# 4. Delete the backup file
rm keys-backup.yaml

The controller starts up, sees the restored keys, registers them, and happily decrypts existing sealed secrets from the repo. Log output:

INFO msg="Searching for existing private keys"
INFO msg="registered private key" secretname=sealed-secrets-key7m2wg
INFO msg="registered private key" secretname=sealed-secrets-keykzqw4
INFO msg="registered private key" secretname=sealed-secrets-keyt4fnz

Three keys because the controller rotates them periodically and keeps old ones around for backward compatibility. All three now live on both clusters. Lesson: back up the Sealed Secrets private key the day you install the controller. Put it in your password manager. Without it, every sealed-secret.yaml in your repos becomes encrypted garbage.

Wildcard TLS

The wildcard cert for *.localdomain was still valid for another year, so I just copied the wildcard-localdomain-tls Secret from the old cluster's vaultwarden namespace into the new cluster's vaultwarden, argocd, and later market-mind namespaces. Same cert works on both sides.

Migrating Vaultwarden: The Login That Almost Broke Me

Vaultwarden was first because it's the highest-value service (password manager) and also a clean test case: single pod, single SQLite database, single PVC. If the pattern works here, it works everywhere.

The copy

  1. Scale the old Vaultwarden deployment to 0 to stop writes
  2. Spin up a tiny busybox pod that mounts the same PVC
  3. tar czf /tmp/vw-data.tar.gz the /data directory (SQLite db, attachments, rsa_key.pem, icon cache)
  4. kubectl cp the tarball out to my Mac
  5. Apply the Vaultwarden manifest to the new cluster. New PVC is created empty, pod starts, I wait for it
  6. Scale the new deployment to 0
  7. Start another busybox helper on the new cluster, mount the new PVC, kubectl cp the tarball in, tar xzf, done
  8. Scale new deployment to 1
  9. Delete helper pods

Total data size: 3.6MB. Total downtime on the new side: a few minutes. Old cluster data was frozen the entire time.

The login test

Vaultwarden responded. HTTP 200. API config returned the right domain. Cert was valid. Everything on the technical side was green. So I flipped the Caddyfile:

 vault.localdomain {
   tls /etc/caddy/certs/wildcard.crt /etc/caddy/certs/wildcard.key
-  import k8s_backend
+  import k8s_backend_ha
 }

Reran the Ansible playbook, Caddy reloaded on all three LB nodes in a second, tested curl https://vault.localdomain/api/config and the right response came back. Felt great.

Then I tried to actually log in from my Bitwarden client.

Wrong password.

I tried it again. Wrong password. A cold feeling started forming in my stomach. Did the SQLite WAL files not flush? Did the copy corrupt something? Was I about to lose years of passwords?

Before changing anything, I asked my pair-assistant to flip the DNS back to the old cluster and scale the old Vaultwarden back up. Took about 30 seconds. Tried the same password on the old one.

Wrong password.

That's when I realized I'd been typing my email wrong. I had fat-fingered one character. The migration was fine. I was the bug.

We flipped back to the new cluster, I logged in successfully, and I learned a lesson that applies to every migration: when something doesn't work, verify it doesn't work on the known-good side first. Don't assume the thing you just changed is the thing that broke. Sometimes you're just typing your own email wrong.

Migrating ArgoCD: Server-Side Apply and Empty States

ArgoCD is the thing that manages the other things. It watches my MarketMind repo, notices when I push a manifest change, and syncs it into the cluster. Migrating ArgoCD itself is basically "install it again and re-register the same git repos."

Installed ArgoCD v3.3.0 with the manifest from the official repo:

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.3.0/manifests/install.yaml

First error, and it's the kind of error you only ever see when the tool is getting old:

The CustomResourceDefinition "applicationsets.argoproj.io" is invalid:
metadata.annotations: Too long: may not be more than 262144 bytes

The ArgoCD install manifest embeds its entire CRD history as annotations, and the ApplicationSet CRD has grown past Kubernetes' 256KB client-side apply limit. Fix is a single flag:

kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.3.0/manifests/install.yaml

Server-side apply bypasses the limit because it doesn't stuff the whole manifest into an annotation. --force-conflicts tells it "yes, I mean to overwrite fields some other controller used to own." Standard dance.

After install, I patched the argocd-cmd-params-cm ConfigMap with server.insecure: "true" so ArgoCD speaks HTTP internally and TLS terminates at Caddy, applied the IngressRoute, and restarted the server deployment to pick up the config. Five minutes of work.

The confusing part

I flipped argocd.localdomain to the new cluster and logged in. The new cluster's ArgoCD had no applications registered yet (by design, since MarketMind was still being held for a separate migration step). So when my browser followed a cached bookmark to argocd.localdomain/applications/argocd/market-mind?view=tree, the page showed "Failed to load data".

Briefly: was it broken? No. The URL was a deep link into an application that didn't exist yet on this ArgoCD instance. Clicking Applications in the sidebar showed the correct state: an empty list. The "Failed to load data" was the honest answer to "show me this nonexistent thing."

This is not a bug. It's a UX that assumed you wouldn't navigate directly to a nonexistent resource. Easy to miss when you're in migration mode and every error looks like a disaster.

Migrating MarketMind: The Actual Application

MarketMind is a Rails app that reads market data, runs background jobs, and writes to Postgres. It was the most complex migration because it had external dependencies that the new cluster had never seen before.

Pre-requisite: trusting step-ca

MarketMind's container image lives at image-registry.localdomain on the Portainer host. The registry presents a TLS cert signed by my internal step-ca. The old cluster nodes had the step-ca root CA installed in their system trust store. The new cluster nodes did not.

First pod creation on the new cluster failed with a line I've seen a hundred times in other contexts:

failed to resolve image: failed to do request: Head "https://image-registry.localdomain/v2/market-mind/manifests/latest":
tls: failed to verify certificate: x509: certificate signed by unknown authority

Fix is the standard Ubuntu trust-anchor dance on every node:

scp step-ca/root_ca.crt [email protected]:/tmp/emirs-lab-ca.crt
ssh [email protected] '
  sudo mv /tmp/emirs-lab-ca.crt /usr/local/share/ca-certificates/
  sudo update-ca-certificates
  sudo systemctl restart containerd
'

Then sudo crictl pull image-registry.localdomain/market-mind:latest on that node returned "Image is up to date" instead of a TLS error. Did this on all three nodes. (Also added to my list of things that should move into the Ansible playbook so re-provisioning a node doesn't hit this again.)

Pre-requisite: the step-ca root CA ConfigMap

Even after the nodes trust step-ca, the pods don't automatically. MarketMind makes outgoing HTTPS requests to errbit.localdomain (for error reporting) and needs to trust the homelab CA for those. This is done with a ConfigMap that's mounted as a CA bundle inside the pod:

# 1. Grab the system CA bundle from a cluster node and append the step-ca root
ssh [email protected] "cat /etc/ssl/certs/ca-certificates.crt" > /tmp/ca-bundle.crt
cat /tmp/emirs-lab-ca.crt >> /tmp/ca-bundle.crt

# 2. Create the ConfigMap in the market-mind namespace
kubectl create configmap step-ca-root-ca \
  --from-file=ca-certificates.crt=/tmp/ca-bundle.crt \
  -n market-mind

The deployment manifest already mounts this ConfigMap at /etc/ssl/custom-certs and sets SSL_CERT_FILE to point at it. This is one of those patterns that feels like overkill until you realize the pod has no idea your LAN hostnames are real.

The actual database swap

MarketMind previously connected to an external Postgres running on the Portainer host. The manifest used a clever ExternalName Service so the app could say "connect to postgres:5432" and Kubernetes would DNS-route that to postgres.localdomain.

On the new cluster I had already stood up a proper HA Postgres (CloudNativePG, a separate project worth its own post). So the whole external-name dance was unnecessary. Two changes in the MarketMind repo:

  1. Update the sealed secret with new connection strings pointing at the in-cluster Postgres service:

    DATABASE_URL=postgres://market_mind:[email protected]:5432/market_mind_production
QUEUE_DATABASE_URL=postgres://market_mind:[email protected]:5432/market_mind_production_queue

  2. Delete postgres-service.yaml (the ExternalName) and remove it from kustomization.yaml. The new DB URL uses the real in-cluster service name directly.

For the sealed secret update I used kubeseal --merge-into so only DATABASE_URL and QUEUE_DATABASE_URL got re-sealed. The other keys (API tokens, SMTP creds, Rails master key) stayed untouched:

kubeseal --fetch-cert \
  --controller-name=sealed-secrets-controller \
  --controller-namespace=kube-system \
  --kubeconfig ~/.kube/config-ha > /tmp/cert.pem

echo -n "postgres://market_mind:[email protected]:5432/market_mind_production" \
  | kubectl create secret generic market-mind-secret \
    --namespace market-mind \
    --dry-run=client \
    --from-file=DATABASE_URL=/dev/stdin \
    -o yaml \
  | kubeseal --format yaml --cert /tmp/cert.pem \
    --merge-into infra/k8s/market-mind/sealed-secret.yaml

Commit, push, register the ArgoCD Application on the new cluster:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: market-mind
  namespace: argocd
spec:
  destination:
    namespace: market-mind
    server: https://kubernetes.default.svc
  project: default
  source:
    path: infra/k8s/market-mind
    repoURL: [email protected]:example/market-mind.git
    targetRevision: main
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

The beautiful part

Within about 90 seconds of creating the Application, the worker pod logs showed:

INFO msg="Enqueued StateOfMarketJob to SolidQueue(default)"
INFO msg="Performing StateOfMarketJob"
Rate limit reset for tradier. Usage set to 0.
Rate limit reset for fmp. Usage set to 0.
Proceeding with request to get_market_clock.
INFO msg="Market status updated"
INFO msg="Performed StateOfMarketJob in 450.25ms"

SealedSecret unsealed, image pulled, init container ran db:prepare against the new HA Postgres, web started, worker started, worker picked up its first scheduled job, called the real Tradier and FMP APIs, wrote back to Postgres, and logged out "Market status updated" as if nothing had changed.

Nothing had changed from MarketMind's point of view. That's the point. Same image, same env vars, different infrastructure underneath. And now the underlying database is itself HA with replication and automatic failover.

The CI/CD Pipeline Surprise

After MarketMind was live, I pushed a code change to main expecting the CD workflow to build and deploy. The workflow file looked like this:

- name: Restart deployments
  run: |
    ssh -o StrictHostKeyChecking=no -i ~/.ssh/master_superfly_lan [email protected] \
      "kubectl rollout restart deployment/market-mind-web deployment/market-mind-worker -n market-mind"

Which would have been perfectly fine except 192.168.1.10 was the old K8s master and I had just shut it off. The build step succeeded, the image push succeeded, and then the rollout step hung waiting for SSH to a dead IP.

Two ways to fix this. Option A: change the SSH target to a node on the new cluster. Option B: give the GitHub Actions runner its own kubeconfig and run kubectl directly with no SSH hop.

Option B is strictly better. The SSH hop was only there because the runner didn't have kubectl or a kubeconfig. Once I put those in place, the workflow step collapses to one line:

- name: Restart deployments
  run: |
    kubectl rollout restart deployment/market-mind-web deployment/market-mind-worker -n market-mind

And because the kubeconfig points at the kube-vip API endpoint (192.168.1.222:6443), any single control-plane node can be down and the runner still works.

One-time setup on the runner:

# Install kubectl (pinned to match cluster version)
curl -sLO "https://dl.k8s.io/release/v1.35.3/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

# Drop the kubeconfig
scp ~/.kube/config-ha runner@gh-runner:~/.kube/config
ssh runner@gh-runner 'chmod 600 ~/.kube/config'

Next push, the pipeline ran green end-to-end. ArgoCD showed the new replica set (rev:2) alongside the old (rev:1) during the rolling update, both for web and worker. Two minutes later, only the new pods were running.

A note on rollout restart and :latest

If you've never thought about this: when your image tag is :latest, pushing a new image doesn't automatically redeploy anything. The Deployment manifest in git is byte-identical before and after. Kubernetes sees nothing to change. ArgoCD sees nothing to sync.

kubectl rollout restart fakes a change by adding a timestamp annotation to the pod template. Kubernetes notices the template is "different," creates a new ReplicaSet, and the rolling update begins.

This is a hack. The GitOps-correct thing is to tag images with the commit SHA and have your CD pipeline write the SHA into the Deployment manifest and commit it back. Then ArgoCD sees a real git change, syncs it, and you get honest rollbacks via git revert. I know this. I'll fix it eventually. It's in a TODO somewhere.

Retargeting the HA LB

Each service got the same one-line Caddyfile change as Vaultwarden:

-  import k8s_backend
+  import k8s_backend_ha

Where k8s_backend_ha was a new snippet I'd added alongside the existing k8s_backend:

# Old cluster — workers only, goes down with Proxmox
(k8s_backend) {
  reverse_proxy 192.168.1.11:30443 192.168.1.12:30443 {
    transport http { tls; tls_insecure_skip_verify }
    lb_policy round_robin
    health_interval 10s
  }
}

# HA cluster — all 3 nodes, survives any single-node failure
(k8s_backend_ha) {
  reverse_proxy 192.168.1.40:30443 192.168.1.41:30443 192.168.1.42:30443 {
    transport http { tls; tls_insecure_skip_verify }
    lb_policy round_robin
    health_interval 10s
  }
}

Keeping both snippets side by side let me migrate services one at a time and have trivial rollback. After all three (vault, argocd, market-mind) were successfully on k8s_backend_ha for a while, I deleted the old snippet, renamed the new one back to k8s_backend, and got the file back to a clean single-cluster config.

The Graceful Goodbye

Once every service was live on the new cluster and verified, the old cluster was dead weight: VMs burning CPU cycles in Proxmox, running workloads that no traffic reached.

I didn't destroy them immediately. The smart move is a soak period. For seven days, I'd leave them powered off. If anything quietly depended on the old cluster in a way I missed (a cron job, a forgotten hostname in a config file, a cloudflared route, a scheduled GitHub Action), it would surface within the week. If not, I'd terraform destroy with confidence.

Shutting them down was a one-shot on the Proxmox host:

for vmid in 110 111 112; do
  qm shutdown $vmid --timeout 60
  qm set $vmid --onboot 0
done

The onboot 0 flag is important. Without it, a Proxmox reboot would auto-start the VMs and the soak would silently restart the old cluster, and I wouldn't notice until I hit a ghost.

With the VMs off, I ran the smoke test one more time:

vault.localdomain:       200
argocd.localdomain:      200
market-mind.localdomain: 200

All three services still alive. Proof that nothing was secretly depending on the old cluster. The soak clock started.

If the week passes clean, the destroy script is already written:

cd terraform/proxmox
terraform destroy \
  -target=proxmox_virtual_environment_vm.k8s_master \
  -target=proxmox_virtual_environment_vm.k8s_workers

Then remove the resource definitions from the repo, clean up the old k8s_nodes group from the Ansible inventory, and delete the now-stale ~/.kube/config-old-cluster file.

What I Learned

Parallel clusters + DNS routing beats big-bang cutover every time. Every migration step was one line change in one file and a playbook re-run. Rollback was the same change in reverse. There was never a moment where I couldn't get back to a working state in under a minute. Compare that to the alternative "take everything down at 3am and hope" pattern and it's not even close.

Back up Sealed Secrets keys the day you install the controller. Without them, every sealed-secret.yaml in every repo becomes decorative. Put the backup in your password manager. Put it in two password managers. Treat it like the private key it actually is.

When something breaks during a migration, verify the known-good side first. I almost unwound a perfectly good Vaultwarden migration because I typed my email wrong. The instinct after any change is to suspect the change. Fight the instinct. Check the baseline first.

:latest + kubectl rollout restart is a hack, and hacks have a way of meeting you at 2am. Tag images with commit SHAs. Let your CD pipeline commit the new SHA into the Deployment manifest. Your future self will thank you.

Runner-as-cluster-client is simpler than runner-SSH-to-cluster-node. The old pattern had a hardcoded node IP, no redundancy, and one more hop to debug. Putting the kubeconfig on the runner and running kubectl directly collapsed three lines of SSH incantation into one line, and gave me automatic API server failover for free.

Soak before destroy. One week, VMs off, onboot disabled. Costs you nothing. Buys you a safety net against the thing you forgot. There's always a thing you forgot.

What's Next

After the soak expires, I'll destroy the old VMs, reclaim the compute, and remove the old cluster's scaffolding from the repo. That's the last mechanical step.

After that, the big remaining gap is backups. The new cluster uses local-path-provisioner, which is simple and fast and pins each PVC to a single node. If that node's disk dies, Vaultwarden's SQLite database dies with it. The HA-control-plane story is great; the data-durability story is nonexistent. Fix coming as a separate project in the disaster recovery series.

But that's a post for another day. Today, I've got a highly available Kubernetes cluster running on genuinely HA hardware, with three stateful services migrated cleanly, verified login paths, and an old cluster politely waiting for the graveyard. That's a good week.

I used this prompt to generate the featured image. Three glowing hardware nodes in a triangle (one rack server, two small mini PCs) with data streams flowing between them, while in the foreground a single old dusty server VM fades out and dissolves into pixels. The new triangle of nodes is vivid and bright; the dissolving server is desaturated and fragmentary. Dark homelab aesthetic with deep blue and neon green accent lighting, holographic Kubernetes wheel hovering above the triangle, photorealistic 3D render with shallow depth of field.

Share