sysadmin horror stories
A real-world AlmaLinux 10 SELinux migration from fully disabled to enforcing on a server running Apache, Remi PHP-FPM, BookStack, a Laravel controller, MariaDB, custom SSH port 65535, cron jobs and custom writable paths — with troubleshooting commands and a practical SELinux cheatsheet.
A real-world BookStack upgrade from v25.02.2 to v26.05.3: service-account Git ownership, runuser, backups, Composer/PHP version mismatches, Laravel migrations, writable storage, and the checks that tell you the upgrade actually worked.
A real-world, step-by-step AlmaLinux 8 → 9 → 10 in-place upgrade guide using ELevate: backups, Leapp inhibitors, VDO checks, NetworkManager migration, third-party packages, MariaDB, old EL packages, and the post-upgrade cleanup nobody tells you about.
Introduction, or “What Could Possibly Go Wrong” So you have a perfectly good server. Two perfectly good SATA SSDs. A perfectly reasonable desire to mirror them so that when one inevitably dies at 3am on a Sunday, your server keeps running. You’ve done this a hundred times. mdadm, RAID1, /boot on the array, done, beer, … Read more
The Problem Internal tools. You know the drill — a mix of VMs that never see the public internet, accessible only over VPN or a local network. No port 80. No HTTP challenge possible. And that wildcard SSL certificate you bought two years ago just expired. You could buy another commercial wildcard. Or you could … Read more
A critical authentication bypass in cPanel & WHM gave attackers root access to over a million publicly exposed servers — silently, without a password, defeating two-factor authentication — for sixty-four days before anyone patched a thing. What Was Actually at Stake cPanel and WHM are not just web applications. They are the administrative nervous system … Read more
We tested 9 local LLM models as spam classifiers on an AMD EPYC 4545P running Ollama. Here’s what we found. After setting up the SpamAssassin + Ollama integration (see the previous post), the obvious next question was: is qwen2.5:7b actually the best choice, or did we just get lucky picking it first? So we ran … Read more
In which I spend a Sunday morning asking a local AI to tell me whether emails about cheap Viagra are, in fact, about cheap Viagra. Spoiler: the 0.5b model cannot. The Problem SpamAssassin is great. Bayes is great. RBLs are great. But spam has gotten weird. Greek-language product spam from hacked domains with randomized subfolders … Read more
Sysadmin Horror Stories — Vol. 1 Ubuntu 24.04 & RAID1 EFI: A Comedy of Errors Platform: AMD AM5 / B650 Disks: 2× Samsung 960GB SATA SSD Mood: Progressively deteriorating A completely true account of what happens when you try to do something perfectly reasonable — installing Ubuntu Server 24.04 on a proper software RAID1 … Read more
Every piece of security infrastructure needs a real-world test. Unit tests are fine. Staging environments are fine. But nothing validates your upload scanning pipeline quite like an actual threat actor uploading a PHP webshell to your server while you’re mid-development. Allow us to introduce our tester: ~XBumbbleB33~. The Setup For context: CFM is our homegrown … Read more