I tested my own deploy platform on a real VPS | Levelrail

I had 15+ sites spread across Coolify on a few VPSs, Vercel, an AWS EC2 container, ECS, RDS and Neon. Every VPS renewal turned into a migration weekend. So I built a self-hosted deploy platform called Levelrail, and before I moved my own sites onto it I ran it on two real servers: an Oracle free-tier ARM VM and a cheap VPS that also runs Coolify.

I expected small bugs. I got four that would have hurt real users. Here they are, in the order I found them.

1. Every local backup failed on a fresh install#

I created a Postgres database, ran a backup and looked at the history. Every local backup on a default install failed with the same error: the backup folder did not exist.

On my laptop the folder had existed for months, so I never saw it. On a clean server the platform tried to write into a path nobody had created.

The fix was small: create the local backup root the first time it is used, and make the location a setting instead of a hardcoded path. The lesson was bigger. A backup feature that works only on the machine you built it on is not a backup feature. I now run every backup and restore on a clean server before I trust it.

2. The firewall panel showed nothing#

I wanted a firewall page that tells you what is actually exposed on a host, not a list of settings. On the VPS it showed an empty table.

The reason: the host used nftables, and the panel only knew how to read the older iptables rules. It looked fine on every machine I had tested, and showed nothing on the one that mattered.

Now the panel detects which backend is active, reads it, and says so when it cannot. An empty security panel is worse than no panel, because it looks like good news.

3. After a reboot the apps were unreachable for about 30 seconds#

Apps are bound to loopback ports and the proxy routes to them. After a reboot, some ports changed, and the proxy kept sending traffic to the old ones until the next refresh.

On the Oracle ARM VM, Docker brought the containers back about 19 seconds after boot and the control plane answered health checks at 33 seconds. But Docker had handed two apps new loopback ports, and the platform kept the old ones for about 30 seconds until its next reconcile. I added a refresh at boot so routes recover as soon as the containers are back.

4. The HTTP to HTTPS redirect dropped the port#

On a server where the proxy listened on a non-standard port, the redirect sent people to the default HTTPS port and the page failed. Small bug, very confusing for the person on the other end.

What the testing taught me#

Test on a clean server, not your own. Three of the four bugs only appear where nothing is pre-configured.
A security or backup panel needs a loud "I cannot tell" state.
Write the failure into the product: the platform now tells you when a backup has failed instead of waiting for you to look.

How I test now#

Before I trust a feature, I run it three ways: on my machine, on a clean cloud VM, and again after a reboot. The reboot is the one people skip. Backups, routes, certificates and firewall state are all things that look fine until the machine restarts and has to rebuild them from nothing. If a feature survives a clean install and a reboot, it has earned a place in the product.

The numbers#

Control plane memory: about 200 MB RSS right after start (196 MiB on the VPS), and the API was ready in 4.2 seconds from an empty data directory.
A Next.js 14 app built and deployed in 2 minutes 46 seconds on a 2 vCPU VPS, with a peak of about 514 MB of extra host memory during the build.
A restart with --wait takes about 12 seconds. Three redeploys under 10 requests per second served 458 requests with no failed responses and a longest gap of 0.36 seconds.
Upgrading the platform in place: 2.19 seconds of downtime, a 179 ms database migration, and all 5 apps, 7 databases and 6 domains still there afterwards.
A new Postgres database was ready in 7.2 seconds, and a restore into a new database took 22 seconds with all 1000 test rows intact.

What Levelrail is#

It is a self-hosted deploy platform. Deploy from Git, a Dockerfile, a compose file or a template. HTTPS and domains, logs, rollbacks, databases with scheduled backups and restore, multiple servers from one control plane, and an audit log. Everything is also available from the CLI, the API and an MCP server. I run my own sites on it.

Repo: https://github.com/glincker/levelrail Site: https://levelrail.com

If you self-host and have a bug you only found on a clean server, I would like to hear it. What did your first failed backup look like?