The Doors I Forgot Were Open

Notes on tending a server that quietly grew into doing everything.

It began with a feeling, which is the worst possible way to begin anything technical.

The server felt like it kept falling over. Every so often the load average would drift past two, the terminal would repaint its banner, and some old reflex would mutter there it goes again, rebooted. I believed that reflex for longer than I'd like to admit.

The honest first move in any optimization is to stop trusting the symptom and go look at the number underneath it. Load average might be the most misread figure in all of Linux. People read it as a CPU gauge. It isn't. On a twelve-core machine, a load of two means the box is half asleep — better than eighty percent of its cores idle, waiting for something to do. The figure that actually mattered was idle time, and idle time sat at ninety-four percent. The machine wasn't drowning. It had never been drowning. And those reboots I was so sure about? The uptime counter climbed the entire time. Machines that reboot don't count days; they start again from zero.

So there was no performance problem. That's a real result, not a disappointment. "Nothing is wrong here" is a legitimate answer, and hunting for phantom gains on a healthy machine is just a more sophisticated way of breaking it.

But once you stop asking why is it slow and start asking what is this thing actually doing, a different picture shows up. This was a server that had grown, without my quite noticing, into doing everything — a handful of websites, a git forge, an IRC bouncer, a small app or two I'd forgotten I installed, a sediment of services laid down over years. And every one of them had, at some point, opened a door.

So I listed what was listening, and it was not a short list. A database bound to 0.0.0.0 — every interface, the public one included. A second database doing the same. A container-management dashboard, the kind of thing that hands you the entire host if its password ever leaks, answering cheerfully on a public port. File sharing. A proxy I'd set up for some reason that had long since died and taken its justification with it. Each of these had been opened deliberately, once, for a purpose that no longer existed, and then simply left.

Here's the part that catches almost everyone, and it caught me: a firewall is not enough when you run Docker.

The mental model most of us carry is that the firewall is the bouncer at the door and nothing gets past it. But Docker writes its own packet-forwarding rules to make published ports reachable, and those rules are consulted before your tidy firewall policy ever gets a vote. You can have a deny-by-default firewall, feel completely safe, and still have a dozen container ports wide open to the internet. I did. The policy said one thing; the actual path a packet would take said another. That gap is the single most useful thing I relearned this week: trust the path, not the policy.

The fix wasn't clever, only disciplined. Anything that already sat behind the reverse proxy didn't need a public port of its own — the proxy reaches it over loopback, on the same machine — so I rebound those services to 127.0.0.1 and the outside door quietly closed. Nothing changed for visitors; they were always arriving through the domain and the proxy. The direct shortcut, the one only an attacker would ever bother with, just stopped existing.

Everything else — the databases, the dashboards, the things only I ever touch — moved onto the mesh. I already run a WireGuard-style mesh VPN across my machines, and once you have one of those, the public internet stops being the place you administer anything. A service bound to its mesh address is invisible to the world and a single hop away from me. The database that had been sitting exposed for who-knows-how-long became reachable from my own devices and from nowhere else on earth.

Then I deleted things, which is badly underrated. The app I never opened. The proxy with no purpose. A couple of accounts belonging to people who'd moved on long ago. Every one of them was surface area I'd been defending for no reason. Removing a service you don't use is worth more than hardening one you do — there is no more secure version of a thing than its absence.

The remainder was housekeeping of the small, dignified sort. A login banner that had been quietly lying about half its fields, because a language setting had broken the way it parsed memory and processor, got rewritten to read straight from the kernel instead of from translated labels. Stale accounts trimmed. The kind of work nobody thanks you for and everybody quietly benefits from.

And then, near the very end, the original ghost finally showed its face.

The thing that had convinced me the server was falling over wasn't the server at all. It was the IRC link — my client, at home behind an ordinary router, talking to the bouncer across the open internet. Home connections are restless animals. The router's translation table forgets an idle conversation; the next packet lands on a socket that's already dead; the link snaps; the client reconnects; and from the outside it looks exactly like a machine that keeps collapsing. It wasn't. The machine's uptime was measured in serene, uninterrupted days. The road to it was what kept washing out.

The remedy was the same idea as all the rest: stop driving on the public road. I pointed the client at the bouncer through the mesh instead of across the open net. The tunnel keeps itself awake, the translation-table timeouts stopped mattering, and the connection that used to die on the hour simply held. One steady thread where there had been a churn of dying ones.

None of this made the server faster, because the server was never slow. That was the lesson folded inside the whole exercise. We reach for the word optimization and picture squeezing more speed out of something, but most of the real work on a mature system is subtraction: closing doors you forgot you opened, deleting what no longer earns its keep, and learning to read the instrument that tells you the truth rather than the one that merely tells you something.

The machine had been fine the entire time. I just hadn't been looking at it honestly.

A server, in the end, isn't so much optimized as it is tended. And most of tending is knowing what to take away.