sysadmin horror stories
Running Linux servers in the real world: upgrades, storage, boot loaders, RAID, VMs and the applications that live on them.
A real-world AlmaLinux 10 SELinux migration from fully disabled to enforcing on a server running Apache, Remi PHP-FPM, BookStack, a Laravel controller, MariaDB, custom SSH port 65535, cron jobs and custom writable paths — with troubleshooting commands and a practical SELinux cheatsheet.
A real-world BookStack upgrade from v25.02.2 to v26.05.3: service-account Git ownership, runuser, backups, Composer/PHP version mismatches, Laravel migrations, writable storage, and the checks that tell you the upgrade actually worked.
A real-world, step-by-step AlmaLinux 8 → 9 → 10 in-place upgrade guide using ELevate: backups, Leapp inhibitors, VDO checks, NetworkManager migration, third-party packages, MariaDB, old EL packages, and the post-upgrade cleanup nobody tells you about.
Introduction, or “What Could Possibly Go Wrong” So you have a perfectly good server. Two perfectly good SATA SSDs. A perfectly reasonable desire to mirror them so that when one inevitably dies at 3am on a Sunday, your server keeps running. You’ve done this a hundred times. mdadm, RAID1, /boot on the array, done, beer, … Read more
The Problem Internal tools. You know the drill — a mix of VMs that never see the public internet, accessible only over VPN or a local network. No port 80. No HTTP challenge possible. And that wildcard SSL certificate you bought two years ago just expired. You could buy another commercial wildcard. Or you could … Read more
Sysadmin Horror Stories — Vol. 1 Ubuntu 24.04 & RAID1 EFI: A Comedy of Errors Platform: AMD AM5 / B650 Disks: 2× Samsung 960GB SATA SSD Mood: Progressively deteriorating A completely true account of what happens when you try to do something perfectly reasonable — installing Ubuntu Server 24.04 on a proper software RAID1 … Read more
server { listen 80; server_name kibana; error_log /var/log/nginx/kibana.error.log; access_log /var/log/nginx/kibana.access.log; location / { rewrite ^/(.*) /$1 break; proxy_ignore_client_abort on; proxy_pass http://localhost:5601; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; } }
Growing a Linux VM disk can usually be done online, including the root filesystem. First identify what sits between the virtual disk and the mount point: a plain partition, or LVM with ext4 or XFS on top. The original version of this post was the short growpart + resize2fs recipe. That works for ext4 on … Read more
The problem looks like the device ‘disappears’ from the bus, and becomes inaccessible to the driver. If it happens early – the driver will not load, if it happens later – it may fail with sporadic access errors cat /sys/module/pcie_aspm/parameters/policy grubby –update-kernel ALL –args “pcie_aspm.policy=performance pcie_port_pm=off pcie_aspm=off” dracut -f ethtool -K eno1 tso off gso … Read more
Configure Automatic Security Update (Unattended Upgrades) on Ubuntu Server First, install the unattended-upgrades package. sudo apt update sudo apt install unattended-upgrades You need to install the update-notifier-common package in order to set up automatic reboot. sudo apt install update-notifier-common Then edit the 50unattended-upgrades file. sudo nano /etc/apt/apt.conf.d/50unattended-upgrades In this file, you can configure what packages should be automatically updated. By default, … Read more