A Critical PHP Patch, Seven CloudLinux Servers, and a Dependency That Didn’t Exist Yet

PHP shipped a CRITICAL security release. Patching it took one command on every box – except the ones running the OS we pay to be the stable part of the stack.

Twenty-nine days. That’s how long it has been since PHP2Shell Day, when half the internet took its sites offline and we merged a WAF rule. The fix back then was PHP 8.2.35 / 8.3.36 / 8.4.27 / 8.5.12. As of yesterday afternoon, those are the vulnerable versions. Here we go again. Except this time the exploit wasn’t the hard part. The package manager was.

TL;DR

  • CloudLinux’s cl-ea4 repo shipped Passenger 6.2.2, which requires criu-lve >= 4.2-6. The newest criu-lve anywhere was 4.2-4.
  • dnf refuses the whole transaction, so ~540 patched alt-php/ea-php packages per server sat waiting behind a Ruby app server module.
  • --exclude='*passenger*' is not enough on EL8. You also need --exclude='ea-ruby27-ruby-wrapper*'.
  • Bonus round: after a yum clean all, the repo’s metadata pointed to files that returned 404 on every mirror.

The plan fit in a tweet

Thursday, November 19th, 14:00 UTC. PHP 8.5.13, 8.4.28, 8.3.37 and 8.2.36 land, rated CRITICAL: use-after-frees that end in RCE, an out-of-bounds read/write in CGI’s PATH_INFO handling, a local privilege escalation in FPM, and three session bugs that also end in RCE.

The plan: yum update on eight cPanel servers, restart, verify, sleep.

One of the eight runs plain AlmaLinux. The other seven run CloudLinux. Guess which one patched on the first try.

Act 1: Seven out of seven

All seven CloudLinux servers stopped at the same error:

nothing provides criu-lve >= 4.2-6 needed by
ea-apache24-mod-passenger-1:6.2.2-1.el9.cloudlinux.x86_64 from cl-ea4

On EL8 it was the same story with ea-ruby27-rubygem-passenger, ea-ruby27-mod_passenger and ea-ruby27-ruby-wrapper. I checked the repo: criu-lve 4.2-1, 4.2-2, 4.2-3, 4.2-4. No 4.2-6. Not on EL8, not on EL9.

So a package that depends on a version that has not been published was published. Through a repo with 21 “Gradual Rollout Slots”. Gradual, apparently, in every direction except the one where you check that dependencies resolve.

The part that actually hurts: dnf treats a transaction as all or nothing. One broken Passenger and the whole update aborts, including the CRITICAL PHP fixes sitting in the same run. A Ruby app server module, which most of our customers will never touch, was holding PHP security patches hostage on a few hundred hosting accounts.

Act 2: Exclude it. No, the other one too.

The obvious fix is to keep Passenger on 6.2.1 for now. Our customers do run Passenger apps through cPanel’s Application Manager, so removing it was never an option.

dnf -y update --exclude='*passenger*'

That worked on the EL9 boxes. On EL8, three out of three came back with a new error: ea-ruby27-ruby-wrapper 6.2.2 requires Passenger 6.2.2, and Passenger is “filtered out by exclude filtering”. The wrapper is part of the Passenger stack, but it doesn’t have “passenger” in its name. Of course it doesn’t.

The version that actually works, after a dry run with --assumeno to confirm it resolves:

dnf -y update --exclude='*passenger*' --exclude='ea-ruby27-ruby-wrapper*'

550 upgrades, 539 of them PHP, zero conflicts.

Act 3: The mirrors that knew nothing

Once PHP was patched, I ran a yum clean all && yum update on one box, just to be sure. That’s when error number three showed up:

Status code: 404 for https://.../cloudlinux-x86_64-server-8.10/repodata/
f3a2e3f0...-primary.xml.gz
Error: Failed to download metadata for repo 'cloudlinux-x86_64-server-8'

Every mirror returned 404: Athens, US East, Singapore, Finland. The repomd.xml already pointed to a new primary.xml.gz and filelists.xml.gz that had not been uploaded yet.

Publishing a repo has exactly one rule: upload the packages and metadata files first, then switch repomd.xml to point at them. Doing it the other way round means every client that refreshes in between gets a repo that points to nothing. This is the order anyone learns the first time they run createrepo and rsync by hand.

Three update attempts, three different errors, on the platform that sells itself as the stable one. A few hours later criu-lve 4.2-6 finally showed up, and Passenger 6.2.2 installed on the second or third try. Gradually.

Meanwhile, everywhere else

Same evening, same PHP release, the rest of the fleet:

Stack How it went
cPanel on plain AlmaLinux yum update, done. First try.
DirectAdmin CustomBuild Update the versions, let it compile, FPM restarts on its own. Boring, in the best way.
Remi on EL8 / EL10 FPM restarted itself on upgrade. EL10’s php82 build was still on its way, and the mirrors were slow, but nothing was broken.
Debian security 8.4.28 was in debian-security the same evening.
CloudLinux See Acts 1 to 3.

 

If you run CloudLinux, read this before the next PHP release

  1. Dry-run first. dnf update --assumeno shows you the dependency mess before you are halfway through a maintenance window.
  2. Exclude the whole stack, not just the name. *passenger* plus ea-ruby27-ruby-wrapper* on EL8. Check with dnf repoquery --whatrequires what else hangs off the package you’re holding back.
  3. Don’t yum clean all in a hurry. The metadata in your cache may be the only copy that still works while the mirrors catch up.
  4. Make sure FPM is really running the new binary. Check readlink /proc/<master pid>/exe. If it ends in (deleted), the master is still running the old binary. We found one like that on a box where the package had updated fine.
  5. Check /boot before kernel season. Run rpm -qa 'kernel-debug*'. You might be surprised.

Thank you, CloudLinux

By midnight, every server was on 8.2.36 / 8.3.37 / 8.4.28 / 8.5.13. The cPanel boxes took about an hour, and most of that hour was spent working around a package we don’t use, waiting for a package that didn’t exist, from mirrors that hadn’t been told.

PHP releases run on a four-week cycle. See you in December, CloudLinux. Same time, same slot.