How murad.chat fits the fediverse on one VPS
GoToSocial, PeerTube, Lemmy, Matrix, a Bluesky PDS and Nostr on one 8-core VPS. How I tamed Pixelfed's Horizon (79% to 16% of its memory limit), cut Celery workers, capped every container and then loosened the limits that were starving the apps.
murad.chat is the VPS where most of my federated stuff lives. The name comes from the Matrix server, but the box runs a lot more than chat: GoToSocial at murad.social, a PeerTube instance, Lemmy at lemmy.club, a Bluesky PDS, a Nostr relay with its web client, my mail server and a handful of small sites. It's a Contabo machine on Debian 12 with 8 virtual cores (AMD EPYC), 23 GB of RAM and 788 GB of disk, 7% of it used.
This is the story of how all of that fits on one box, including the part where it didn't.
The heavy era #
A few months ago the server also ran Funkwhale (music), BookWyrm (books), Pixelfed (photos) and Iceshrimp (microblogging). Every one of these apps has background workers that handle federation: fetching remote posts, delivering activities to other servers, processing media. Put four of them on the same machine and they all compete for memory and CPU, even when nobody is using them. The load average was the first thing I went after.
Funkwhale and BookWyrm are Python apps that run their background jobs through Celery. Celery's default is one worker process per CPU core, so on an 8 core box each app could start eight workers of its own. I cut those worker counts down. A personal instance doesn't need eight workers per app.
Pixelfed was the bigger problem. It's a Laravel app, and its queues are run by Horizon. Out of the box, Horizon uses an "auto" balancing strategy: it watches the queues and starts more PHP processes when they back up, up to a maximum for each group of queues. Federation traffic comes in bursts, so every burst meant more processes, and every PHP process holds its own chunk of memory.
The fix was two settings: balance set to false and maxProcesses set to 3. With balancing off, Horizon stops spawning new processes and works with a small fixed pool. A burst takes a little longer to drain. Horizon's memory use dropped from 79% of its limit to 16%:
before ################################ 79%
after ###### 16%Then most of it went away #
Later I deleted Pixelfed. Funkwhale, BookWyrm and Iceshrimp are gone too. GoToSocial now handles the microblog side at murad.social, and it lives in /opt/murad-shrimp, a folder name left over from the Iceshrimp days.
GoToSocial is a single Go binary with SQLite or Postgres behind it, so it asks for a lot less than the apps it replaced.
Memory and CPU limits #
Everything on the box runs in Docker except mail, the Nostr signers and the network plumbing (nginx, Tailscale, a local DNS resolver). Almost every container has a memory limit, and most of them have a CPU cap too. For most apps the limits sit in a separate file, docker-compose.limits.yml, next to the main compose file. That means recreating a container needs both files:
docker compose -f docker-compose.yml -f docker-compose.limits.yml up -dForget the second file and the container comes back without those limits.
These are the ceilings right now:
- GoToSocial: 4 CPUs, 3 GB
- PeerTube: 4 GB, with 1 GB for its Postgres and 512 MB for Redis
- Bluesky PDS: 2 CPUs, 2 GB
- Matrix Synapse: 1 GB, plus 512 MB for its Postgres
- Lemmy backend: 0.75 CPU, 768 MB
- Lemmy UI: 1 CPU, 512 MB
- Lemmy Postgres: 0.5 CPU, 768 MB
- pict-rs (Lemmy's image server): 1 CPU, 512 MB
- Lemmy's nginx: 0.25 CPU, 128 MB
- Nostr relay: 512 MB
- nostrudel: 96 MB
The Lemmy containers also have a swap ceiling a bit above the memory limit (768 MB of RAM and 1 GB total, for example). A container that goes over gets a small cushion of swap before the kernel kills it.
Where the limits were too tight #
Limits protect the neighbors, but they can also starve the app they sit on. On November 4 I checked the throttling numbers and found two cases. Lemmy UI, capped at half a CPU, had been throttled for about eight hours over eleven days. GoToSocial, at 2 CPUs and 1.5 GB, was hitting its ceiling while loading the timeline, and requests were timing out. Meanwhile the host itself was mostly idle.
So I doubled the caps where containers were being throttled: Lemmy UI and pict-rs went to 1 CPU, PeerTube's Redis to 1 CPU, the Bluesky PDS to 2 CPUs. GoToSocial went to 4 CPUs and 3 GB, with a 300 MiB in-memory cache.
One gotcha from that day: after restarting Docker, a container can stay stopped with "Exited (128)" and an "AlreadyExists" error. An orphan containerd-shim is still holding its ID. Kill that shim process and run docker start on the container, and it comes back.
Logs and transcoding #
Logs cost CPU. GoToSocial and PeerTube log at the warn level now, and PeerTube doesn't log every HTTP request anymore. nginx still keeps the access log in the journal.
Video transcoding is heavy work, so PeerTube's is kept small: only 360p and 480p, two threads, one job at a time, and it never transcodes a copy at the original resolution.
How it looks today #
At the time of writing, the server had been up for 17 days with 20 containers running. Out of 23 GB of RAM, about 4.4 GB was in use, 13 GB was cache and 19 GB was available. Swap had 3 MB used out of 4 GB, and memory pressure was zero. The load average sat between 4 and 6 on 8 cores, with the CPU about 83% idle.
Memory per container at that moment, the biggest ones:
peertube ############## 695 MiB
gotosocial ########### 574 MiB
bluesky-pds ### 126 MiB
synapse ## 115 MiB
lemmy-ui ## 108 MiB
coppermine-db ## 94 MiBPeerTube was also the top CPU user, at 29%, with its Redis second at 12.5%.
That coppermine-db line belongs to Coppermine, a PHP 7.4 photo gallery from another era. It's the next thing on the list to go if I don't find a use for it.