I redeployed an app three times while sending 4,000 requests. Two cut over, one is a mystery

"Zero downtime" is one of those phrases that sounds finished until you try to prove it. So I built a small load script and a rule: send steady traffic through the real HTTPS ingress, redeploy the app three times, and count every failed request.

The setup#

A tiny web app (a whoami container) behind the platform's proxy, over HTTPS with a local certificate.
About 20 requests per second, for the whole run.
Three redeploys, each by API with a new image tag.
Default drain settings: 15 seconds of draining, then a 3 minute hold so you can roll back, and a stability check before the old container goes away.

The result#

4,313 requests. 0 failed or non-2xx.
Latency: p50 5 ms, p95 68 ms, p99 208 ms.
Longest gap between two successful responses: 0.111 seconds.

I ran the same idea on a real 2 vCPU VPS too, at 10 requests per second: three redeploys, 458 requests, no failed responses, longest gap 0.36 seconds. A restart that waits for the app to be healthy took about 12 seconds each time, and roughly 8 of those seconds are the stability check deliberately making sure the new container is not a one-hit wonder.

The two things that did not go to plan#

The disk guard refused my deploy. On the first attempt, the third deploy came back with a 507. The platform has a guard that refuses a deploy when the disk is too full, and my laptop disk was at 90 percent. The guard was right. I lowered its thresholds for the test and moved on. A deploy platform that says no before it fills your disk is doing its job.

One redeploy never answered. Of the three redeploys, two cut over cleanly. The third was accepted by the API but the new version did not answer within my three minute wait. No request failed, because the old container kept serving, but I do not know why the cutover did not finish. The log ends with a cancelled write at shutdown, and I have not investigated further. It might be the hold window, a slow image pull or a real stall. I would rather tell you that than pretend the test was perfect.

What I learned about testing zero downtime#

Count failures, not vibes. A number like 0 out of 4,313 is the claim. Anything softer is marketing.
Look at the longest gap, not just the average. Averages hide a one second hole.
Test on a nearly full disk at least once. Real servers are rarely tidy.
Keep the unexplained result. It is the next bug.

The script is in the repo if you want to point it at your own setup. The platform is Levelrail, which I build and run my own sites on: https://github.com/glincker/levelrail

How do you check that a deploy really did not drop requests?