The Problem
Here is a fun little paradox I let sit in my homelab for way too long. My monitoring hub, the piece of software whose entire job is to tell me when something in the house has fallen over, was running on a Mac Mini in the house. Same power. Same internet. Same failure domain as everything it was supposed to be watching.
So the day the power flickers or the ISP has a bad afternoon, here is what happens: the homelab goes dark, and the thing that is supposed to text me "hey, the homelab went dark" goes dark with it. The watchtower was inside the castle. If the castle burns, the tower does not get to send up a flare, because the tower is also on fire.
And sure enough this actually happened, while I was away for the world cup and I had no idea what's going on. Luckily I have great neighbors who came in and inspected the house. The issue was that my entire electrical circuit shut down, meaning my fancy ups power backup worked until it didn't. This isn't solution to that problem (which I've resolved in the meantime), but rather an additional enhancement towards remidiating a broken system.
I run Beszel for this, and I love it. It is tiny, it is fast, it does one job. But it was in the wrong place. What I wanted was a hub somewhere else entirely, on a box that stays up when my house does not, so it can be the one thing that survives to tell me the rest did not.
I already had a candidate. A little cloud VPS I stood up a while back to be a WireGuard exit node, sitting in a datacenter on someone else's power and someone else's uplink. It was doing one small job and had plenty of room for a second. Outside my blast radius, on infrastructure I trust more than my own breaker panel. Perfect.
This is the story of moving the hub there. It went well, right up until the moment I texted myself that fourteen services were down. All of them. At once. On purpose, sort of. Let me tell you about it over coffee.
The Plan
Work out which agents push to the hub and which get pulled, because it changes everything(done, and it was detective work)Convert the one stubborn pull agent to push(done, cleanly, no duplicate)Stand up the hub on the VPS as a plain binary, not Docker(done)Copy the database across and upgrade the version(done)Cut every agent over to the new hub and retire the old one(done)Accidentally page myself with a full fake apocalypse(extremely done)Add a firewall without dropping a live video stream(done, mid episode)
Push or Pull, and Why You Must Know Before You Touch Anything
Beszel agents can talk to the hub in two directions, and the whole migration hinges on which one each agent uses.
In push mode the agent dials out to the hub over a WebSocket. The agent is the one making the connection. Put the hub anywhere with a public address and the agent will find it, no inbound holes required.
In pull mode the hub reaches in to the agent over an SSH-style channel on port 45876. The hub is the one making the connection, so it needs a route to the agent.
That distinction is the difference between "trivial migration" and "please open a hole from a cloud box into your home network," which is a sentence that should make anyone uneasy. If every agent pushes, the cloud hub never needs to touch my LAN. If any agent gets pulled, I would have to expose it. So before I moved a single thing, I needed to know exactly what I was dealing with.
The hub keeps all of this in a little SQLite database. I pulled the list of systems out of it:
sqlite3 ~/beszel/data/data.db \
"SELECT name, host, status FROM systems ORDER BY name;"
Most rows looked normal, a hostname and a LAN address. But a cluster of them listed a host of 192.168.1.1, an address the hub could not possibly reach, and yet their status was cheerfully up. That stopped me. How is a system reporting healthy if the hub cannot even route to the address it has on file for it?
The answer, once it clicked, is airtight. The host field means two completely different things depending on direction. In pull mode it is a destination the hub dials, so it has to be reachable. In push mode it is just the return address stamped on the connection when the agent phoned in, whatever the network happened to rewrite it to on the way. So an unreachable address plus an up status can only mean one thing: the hub is not reaching out to that box at all. The data is arriving on its own. That is a push agent, full stop, and no amount of squinting at the address changes it.
I verified the handful of ambiguous ones by actually reading each agent's config, but that one piece of logic did most of the work. When I finished, the tally was fifteen push and one pull. The one holdout was my container host.
Converting the Last Pull Agent Without Making a Mess
The container host was the only box the old hub was reaching in to. To move the hub to the cloud, that agent had to start pushing instead.
Here is the part I was nervous about. Beszel identifies a pushing agent by a fingerprint tied to a registration token. If I just flipped the agent to push, would it bind to the existing system record and keep all its history, or would it show up as a brand new duplicate and leave a ghost behind?
I went looking in the database and found that every system already has a row in a fingerprints table pairing it with a token. The pull agent's row had a token but an empty fingerprint, which is exactly the shape of "a push agent has been provisioned here but has not connected yet." And the agent already held that same token in its config. So the moment it pushed, it would present the matching token, the hub would look it up, find the waiting row, and fill in the fingerprint. Same system, same history, no duplicate.
Which is precisely what happened. I added the hub URL to the agent, restarted it, watched the log say WebSocket connected, and checked the hub. Still one record. Fingerprint now populated. History intact. That is the good kind of boring.
Standing Up the Hub, and Why Not Docker
The VPS is an OpenVZ container, and OpenVZ containers are fussy about the things Docker wants from a kernel. Rather than fight overlay filesystems and cgroup quirks, I ran the hub the way it ships, as a single Go binary under systemd. Fewer moving parts, nothing to argue with.
# /etc/systemd/system/beszel-hub.service
[Unit]
Description=Beszel Hub
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=beszel
Group=beszel
WorkingDirectory=/opt/beszel
ExecStart=/opt/beszel/beszel serve --http 127.0.0.1:8090
Restart=on-failure
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/opt/beszel/beszel_data
[Install]
WantedBy=multi-user.target
Notice it binds to 127.0.0.1. The hub itself is never exposed. In front of it I put Caddy, which handles TLS and gets a real Let's Encrypt certificate automatically. The entire reverse proxy config is four lines:
beszel.example.com, metrics.example.com {
reverse_proxy 127.0.0.1:8090
}
That Let's Encrypt certificate turned out to matter more than I expected, and it quietly solved a problem I had been carrying. My old hub used a certificate from my private CA, which meant every agent had to trust my private root or it would refuse the connection and silently go offline. A public certificate is trusted by everything out of the box. Moving to the cloud with a real cert deleted an entire class of "why is this agent down" from my life.
Two hostnames, on purpose. The UI lives at metrics.example.com and goes through Cloudflare's proxy, which is nice for a login page facing the open internet. The agents connect to beszel.example.com, which is a plain DNS record pointing straight at the box, no proxy. That split is not decoration. Beszel's agent WebSocket does not survive Cloudflare's proxy, it comes back with a 401 and the agent sulks. So the agents get a direct path and the humans get the proxied one, and everyone is happy.
Copy the Database, Then Upgrade
I wanted the history, the alert rules, and my login to come across, so this was a database copy rather than a fresh start. Beszel uses SQLite, so a consistent snapshot is one command that does not even need the hub stopped:
sqlite3 ~/beszel/data/data.db ".backup '/tmp/data.db'"
I copied that to the VPS, dropped it into the new hub's data directory, and started it. Sixteen systems, all my history, my account. Then I upgraded the hub binary to the newest release and let it run its migrations against the copied database, which is the officially blessed upgrade path and went without a complaint. The new version had a feature I had been waiting on, container health alerts, so the upgrade was worth doing on its own.
Old hub still running at home the whole time, untouched, ready to flip back to. That detail becomes important in about two paragraphs.
The Part Where I Paged Myself an Apocalypse
Here is where I earned the coffee.
The database I copied did not just bring my systems and history. It brought my alert rules and my notification config, which is to say my Telegram webhook. And the new hub, freshly started, looked at its sixteen systems, saw that almost none of them were talking to it yet because they were all still pushing to the old hub at home, and did exactly what I built it to do. It decided they were down. All of them. And it reached for the phone.
My pocket started buzzing like an angry hornet. "System down. System down. System down." Fourteen of them, in a tight little cluster, each one a small lie. Nothing was actually down. Everything was fine and happily reporting to the other hub. I had simply stood up a second watchtower, handed it a stale map, and it panicked on cue.
The fix in the moment was to stop the new hub before it could send more. The lesson underneath it is the one worth keeping. When you run two monitoring hubs at once during a migration, and your agents are only reporting to one of them, the other hub sees a graveyard and starts screaming about it. And it is worse than that, because during a partial cutover the pain flips: as each agent moves to the new hub, the old hub loses sight of it and wants to alert too. Whichever hub an agent is not talking to will try to page you about it.
So the real move, the one I should have made from the start, is to silence both hubs before you touch anything. I blanked the webhook on each one, saving the original first, so the rules could keep evaluating but had nowhere to send. Then I cut everything over in a batch, confirmed all sixteen were green on the new hub, and only then restored the webhook on the winner. Quiet migration, no hornets.
If you take one thing from this post, that is it. A copied monitoring database is a loaded notification cannon. Point it at the wall before you plug it in.
Cutting Over, and the Fallback That Would Not Let Go
With both hubs muzzled, moving the agents was mechanical. For the Linux boxes the hub URL lives in the systemd unit, so it was a small edit and a restart, scripted across the fleet. The Mac agents keep it in an env file. The container host meant recreating a container. The TrueNAS box and the Home Assistant add on each needed a hand. One by one they lit up green on the new hub.
Then I noticed one box kept flickering. It would connect to the new hub, then a few minutes later drop and reappear on the old one. I would push it back, it would drift home again.
The culprit was a fallback I had forgotten I built in. Every pushing agent also keeps that pull-style listener open on 45876 as a backup path, and the old hub still had all these systems in its list, so it was actively reaching in and grabbing them the instant a WebSocket so much as hiccuped. The old hub was not a passive has-been waiting to be retired. It was standing there with a fishing rod, reeling my agents back the moment my attention wandered.
There is no clever fix for that. The old hub had to actually go. I stopped it and disabled it from ever starting again, and the flickering stopped instantly. The new hub was finally the only voice in the room.
A Firewall, While the VPN Was Carrying My Favorite TV Show
I tune to my favorite TV Show on Bosnian TV via VPN because they only allow local broadcast. So here we are, last thing we need to do.
The box now hosted a monitoring hub as well as a VPN, and it had no host firewall at all, just a wide open default policy leaning entirely on the fact that only a few things were listening. I wanted a proper default deny.
The complication: as I sat down to do this, that same box was actively streaming my favorite show to a phone over WireGuard. Get this wrong and I do not just lock myself out, I cut the episode off right before the cliffhanger. That is not a mistake you get forgiven for.
So the rules were surgical. Touch only the inbound chain, never the forwarding rules that carry the VPN traffic, put "keep every connection that already exists" as the very first rule so the live stream sails through untouched, and explicitly allow the VPN port on top of that for belt and suspenders. Allow SSH, the VPN, and the two web ports. Deny the rest.
# accepts first, while the policy is still open, so nothing drops mid-setup
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -p icmp -j ACCEPT
iptables -A INPUT -p tcp --dport 49222 -j ACCEPT # ssh, on a non-default port
iptables -A INPUT -p udp --dport 51820 -j ACCEPT # wireguard
iptables -A INPUT -p tcp --dport 80 -j ACCEPT # acme
iptables -A INPUT -p tcp --dport 443 -j ACCEPT # caddy + beszel
# and only now, the door
iptables -P INPUT DROP
The ordering is the whole trick. Every allow rule goes in while the policy is still open, so there is never a moment where a legitimate packet gets dropped because its rule had not been added yet. The drop policy goes on last, by which point the "keep existing connections" rule is already in place protecting both my SSH session and the stream.
And because I have locked myself out of exactly enough remote boxes to have learned, I did all of this with a second SSH session held open and a script already counting down to undo the whole thing in five minutes unless I confirmed it worked. I confirmed. The stream never so much as stuttered. I watched the VPN's byte counter climb straight through the change, which is the only proof I actually trust.
I reached for the modern firewall tool first, nft, and found it was not even installed on this little container, and its existing VPN rules were all in old-school iptables anyway. Mixing a fresh firewall framework into a box carrying a live stream is not a move you make on a whim, so I stayed on iptables to match what was already there. Right tool is sometimes just the one already in the room.
What I Actually Took Away
The hub works. It sits in a datacenter, watches my whole house from the outside, and if the house goes dark it is the one thing still standing to tell me. The paradox is resolved. The watchtower is finally outside the castle.
But the running service is not really the souvenir. Three things are.
The first is a piece of reasoning I will reuse forever: an unreachable address plus a healthy status can only mean the data is being pushed, not pulled. Systems tell you how they are wired if you read them carefully enough.
The second is a scar. A copied monitoring database carries live notification config, and it will fire the instant it disagrees with reality. Silence your alerting before a migration, on every hub involved, and turn it back on only when the new world is actually true. I learned that at the cost of one buzzing pocket and a brief, sincere belief that my entire homelab had died.
The third is the same reflex I keep relearning and keep being grateful for. Before you change anything that controls how you reach a box, or whether it can reach you, arm the undo first. The held session, the five minute timer, the saved-off original config. Every single time it has felt like overkill right up until the one time it was the only thing between me and a very long evening.
The homelab still cannot tell me it is on fire from inside a fire. But now something outside can. That is the whole point, and it only took one fake apocalypse to get there.
Featured Image Prompt
I used this prompt to generate the featured image.
A cinematic tech-noir scene. A tall stone watchtower stands outside a castle wall, perched on a distant hill lit by warm amber light, while the castle itself sits in cool blue shadow across a dark valley. From the tower a single glowing beam of data, rendered as circuit-board traces and light, arcs across the valley toward the castle, watching over it. In the foreground a phone screen glows with a cascade of red "system down" alert bubbles, slightly out of focus, hinting at a false alarm. The tower is clearly safe and separate from whatever might happen to the castle. Deep blues and warm ambers, subtle 3D render mixed with clean vector line work, a faint terminal prompt glowing in one corner. A mood of calm vigilance earned through one chaotic night.