The Domain I Forgot to Take With Me
A story about one moved domain, one leftover certificate, and two hours of my sites being down while every app behind them was perfectly healthy.
A story about one moved domain, one leftover certificate, and two hours of my sites being down while every app behind them was perfectly healthy.
I moved a project to a different server.
That's it. New server, new setup, DNS pointed at the new place. Everything worked. I moved on with my life.
What I didn't move was the SSL certificate sitting on the old server.
I didn't even think about it. The domain wasn't there anymore, so in my head, nothing about it was there anymore either.
Certbot disagreed.
What Actually Happened
My dev server runs a handful of apps. Each one lives in its own Docker container, and Nginx sits in front of all of them as a reverse proxy, handling SSL and routing traffic to the right app.
Certbot runs on a timer twice a day to renew certificates. Most of the time it checks everything, sees nothing is due, and goes back to sleep.
But the certificates for the domain I had moved were getting close to expiry. So certbot tried to renew them.
To renew a certificate with the Nginx plugin, certbot temporarily changes the Nginx config and restarts it. Then it asks Let's Encrypt to verify the domain.
So the sequence looked something like this:
-
Certbot wakes up at 6:20 on a Saturday morning
-
Finds certificates due for renewal for a domain that no longer lives on this server
-
Restarts Nginx to run the challenge
-
The restart fails with
Address already in useon ports 80 and 443 -
Renewal fails
-
Nginx stays down
Looking at the logs afterwards, the most likely explanation is the old Nginx process hadn't fully let go of the ports when the new one tried to start. A timing thing. The kind of thing that probably works fine 99 times out of 100.
This was the 100th time.
The Damage
Every single app on that server went down at once.
Not because any of them broke. Every container was up and healthy. Databases fine. APIs fine. Workers fine. Disk at 35%. Memory fine.
The only thing that was dead was the front door.
Nothing on ports 80 or 443 meant nothing could reach any of them. From the outside, it looked like the whole server had disappeared.
And I had no idea.
It was down for about two hours before I noticed. No alert. No email. Nothing. Just sites quietly not loading while I went about my morning.
The fix itself took seconds:
sudo systemctl start nginx
That's it. Everything came back.
Two hours of downtime. Fixed with one command.
The Cleanup Had Its Own Surprise
Once everything was back, I deleted the leftover certificates so certbot would stop trying to renew them.
Then I found another certificate I was sure was dead. It was named after a test subdomain I had shut down a while back. No DNS record. Clearly safe to delete.
Except it wasn't.
When I checked the Nginx config first, one of my live APIs was using that certificate. At some point it had been issued for multiple domains, and the API was one of them. The certificate's name had nothing to do with what it was actually protecting.
If I had deleted it, the next Nginx reload would have failed, and I would have taken everything down a second time. In the same morning. While cleaning up the first outage.
Instead, I issued a new certificate just for the API, pointed Nginx at it, tested the config, and only then deleted the old one.
The lesson there? A certificate's name is just a label. Check what's actually using it before you touch it.
What I Should Have Done
A few things, honestly.
Clean up the old server when you move a domain. Moving a project isn't finished when the new server works. It's finished when the old one has forgotten it ever existed. Certificates, Nginx configs, renewal settings, all of it:
sudo certbot delete --cert-name your-old-domain.com
Make Nginx restart itself. By default, if Nginx fails to start, systemd just shrugs and leaves it dead. A tiny override changes that:
# /etc/systemd/system/nginx.service.d/restart.conf
[Service]
Restart=on-failure
RestartSec=5s
Now if it fails, systemd tries again five seconds later. If this had been in place, that two-hour outage would have been a five-second blip.
Add a safety net to certbot. Certbot supports hooks that run after every renewal attempt. I added one that makes sure Nginx is actually running when certbot is done with it:
#!/bin/sh
# /etc/letsencrypt/renewal-hooks/post/ensure-nginx.sh
systemctl is-active --quiet nginx || systemctl start nginx
Set up uptime monitoring. This is the one that stings the most. Everything was healthy. The fix took one command. The only reason it lasted two hours was that nobody told me. An external monitor checking my sites every few minutes would have turned this into a non-event.
The Real Lesson
The most annoying outages aren't caused by the thing you're working on. They're caused by the thing you finished months ago and stopped thinking about.
I moved that domain cleanly. The new server worked perfectly. As far as I was concerned, that job was done.
But servers don't forget things just because you did. Every certificate, every config, every timer you leave behind keeps running on its own schedule. And eventually one of them will do something at 6:20 on a Saturday morning.
Now, when I move anything off a server, I treat the cleanup as part of the move. Not a nice-to-have. Part of the job.
I write about real engineering experiences. The things that went wrong, what I learned, and how I build now. If you've ever been taken down by something you forgot was still running, you already know: it's never the thing you're watching.