sysadmin horror stories
In which I spend a Sunday morning asking a local AI to tell me whether emails about cheap Viagra are, in fact, about cheap Viagra. Spoiler: the 0.5b model cannot. The Problem SpamAssassin is great. Bayes is great. RBLs are great. But spam has gotten weird. Greek-language product spam from hacked domains with randomized subfolders … Read more
Scenario: You’re handing over a server to someone else to administer, but you retain control of the network. You want to prevent them from changing the server’s IP address, adding extra IPs, or bypassing your network rules — also give management without giving them VPN access to your management subnet. Stack used: RouterOS 7.x (CCR2004), … Read more
Sysadmin Horror Stories — Vol. 1 Ubuntu 24.04 & RAID1 EFI: A Comedy of Errors Platform: AMD AM5 / B650 Disks: 2× Samsung 960GB SATA SSD Mood: Progressively deteriorating A completely true account of what happens when you try to do something perfectly reasonable — installing Ubuntu Server 24.04 on a proper software RAID1 … Read more
Every piece of security infrastructure needs a real-world test. Unit tests are fine. Staging environments are fine. But nothing validates your upload scanning pipeline quite like an actual threat actor uploading a PHP webshell to your server while you’re mid-development. Allow us to introduce our tester: ~XBumbbleB33~. The Setup For context: CFM is our homegrown … Read more
CFM (Configurable Firewall Manager) started as a modern nftables-first firewall manager designed for high-security hosting and infrastructure operators. Over time, it evolved into a complete security platform: dynamic firewalling, log-driven detection, autoblocking, system hardening, notifications, DNS/GeoIP enrichment, and API integration. After introducing the Unified Web Detector (real-time vhost analytics and suspicious scoring), the next obvious … Read more
A Production-Grade, ML-Ready Traffic Analytics & Abuse-Detection Engine for Nginx, Apache & LiteSpeed CFM (Configurable Firewall Manager) started as a modern nftables-first firewall manager designed for high-security hosting and infrastructure operators.Over time, it evolved into a complete security platform: dynamic firewalling, live log-driven detection, autoblocking, system hardening, notifications, DNS/GeoIP enrichment, and API integration. Today, CFM … Read more
🧠 Building a Hybrid Anomaly Detection Engine for Network Flows 1️⃣ Background flowenricher already enriched NetFlow/IPFIX data with ASN, GeoIP, DNS, etc., and had an Isolation Forest (iForest)–based anomaly detector. We wanted to make the anomaly detection more stable, explainable, and sensitive to different attack patterns without constant retuning — so we added two complementary … Read more
Teaching FlowEnricher to Spot Weirdos: Isolation Forest Joins the Party tl;dr: We added unsupervised anomaly detection to FlowEnricher using an Isolation Forest microservice. It scores per-IP behavior in real time and helps catch stealthy port scans and low-and-slow DoS bursts that signatures miss. Yeap, Machine Learning in netflows. Why Isolation Forest? Rule engines are great … Read more
When CSF (ConfigServer Firewall) announced it was closing, it left a big hole in the Linux hosting world.CSF had been the de-facto standard for years — a reliable mix of iptables, Perl scripts, and clever wrappers that kept countless servers secure. But for those of us who’ve been running modern systems, one question was already … Read more
It all started, as it often does, with a simple hosting setup on dedicated servers, hosting customers on the usual suspects: Hetzner, OVH, and the like. As things grew, the natural evolution was to dive deeper into infrastructure — first a few racks in a datacenter (Nova), then IPs and circuits from a provider, and … Read more