sysadmin horror stories
Introduction, or “What Could Possibly Go Wrong” So you have a perfectly good server. Two perfectly good SATA SSDs. A perfectly reasonable desire to mirror them so that when one inevitably dies at 3am on a Sunday, your server keeps running. You’ve done this a hundred times. mdadm, RAID1, /boot on the array, done, beer, … Read more
“Wait, can you even do that?” It started, as the best bad ideas do, with a question I didn’t expect a real answer to. I’d been hearing about MCP — the Model Context Protocol, the thing that lets an AI assistant actually talk to your stuff instead of just hallucinating about it. And I had … Read more
The Problem Internal tools. You know the drill — a mix of VMs that never see the public internet, accessible only over VPN or a local network. No port 80. No HTTP challenge possible. And that wildcard SSL certificate you bought two years ago just expired. You could buy another commercial wildcard. Or you could … Read more
Part of the CFM server security platform. This post covers what kernsec does, how it works, how to use it, and how it relates to the Linux kernel’s proposed Killswitch primitive. The problem it solves A stock AlmaLinux or CloudLinux install ships with a running kernel that supports hundreds of subsystems almost no hosting server … Read more
A grounded look at kernel attack surface reduction for production hosting servers — no silver bullets, just honest defense. The kernel LPE wave of 2025–2026 was a good reminder that most Linux servers are running with more attack surface exposed than they need. Dirty Frag, Copy Fail (CVE-2026-31431), the ksmbd parade, watch_queue, Dirty Cred — … Read more
A critical authentication bypass in cPanel & WHM gave attackers root access to over a million publicly exposed servers — silently, without a password, defeating two-factor authentication — for sixty-four days before anyone patched a thing. What Was Actually at Stake cPanel and WHM are not just web applications. They are the administrative nervous system … Read more
There was a point where it became obvious that flowenricher had outgrown its original shape. It started as an engine: ingesting NetFlow/IPFIX, enriching traffic with ASN, GeoIP, BGP path, PTR and SNMP data, running detections, and handling mitigation/blackhole logic. The backend side was already doing serious work. But the operator experience around it still belonged … Read more
We tested 9 local LLM models as spam classifiers on an AMD EPYC 4545P running Ollama. Here’s what we found. After setting up the SpamAssassin + Ollama integration (see the previous post), the obvious next question was: is qwen2.5:7b actually the best choice, or did we just get lucky picking it first? So we ran … Read more
In which I spend a Sunday morning asking a local AI to tell me whether emails about cheap Viagra are, in fact, about cheap Viagra. Spoiler: the 0.5b model cannot. The Problem SpamAssassin is great. Bayes is great. RBLs are great. But spam has gotten weird. Greek-language product spam from hacked domains with randomized subfolders … Read more
Scenario: You’re handing over a server to someone else to administer, but you retain control of the network. You want to prevent them from changing the server’s IP address, adding extra IPs, or bypassing your network rules — also give management without giving them VPN access to your management subnet. Stack used: RouterOS 7.x (CCR2004), … Read more