LetsEncrypt

Wildcard SSL with Let’s Encrypt: One Cert, Multiple Internal Hosts

by

The Problem

Internal tools. You know the drill — a mix of VMs that never see the public internet, accessible only over VPN or a local network. No port 80. No HTTP challenge possible. And that wildcard SSL certificate you bought two years ago just expired.

You could buy another commercial wildcard. Or you could do it properly, for free, with Let’s Encrypt — and never think about it again.

This post walks through the exact setup I use across a small hosting infrastructure:

  • SDNS — an AlmaLinux 9 VM running BIND, authoritative for myip.gr. Internet-facing.
  • rosso — an internal VM running Apache httpd with multiple subdomains. VPN-only, port 80 blocked.
  • gitlab — an internal VM running GitLab CE (Omnibus). VPN-only.

The model is pull-based: SDNS issues and stages the certificate, each internal host pulls it on a daily cron. SDNS never needs to reach into your internal network.

Architecture

Let's Encrypt
      |
      | ACME DNS-01 (writes _acme-challenge TXT to BIND)
      v
   [ SDNS ]  <-- authoritative NS, internet-facing
      |  certbot renews ~30d before expiry
      |  deploy hook copies to /home/certpull/
      |
      |  <-- rsync over SSH (pull, read-only, rrsync restricted)
      |
   [ rosso ]          [ gitlab ]          [ future-host ]
   Apache httpd       GitLab Omnibus      whatever
   cron 03:00         cron 03:00          cron 03:00
   reload httpd       gitlab-ctl hup      ...

The key security property: trust flows outward. Internal hosts reach out to SDNS to grab a cert. SDNS has zero credentials pointing inward. If SDNS is ever compromised (it is internet-facing, after all), the attacker can read a certificate — not pivot into your internal network.

Prerequisites

  • BIND running on SDNS, authoritative for your domain
  • certbot and python3-certbot-dns-rfc2136 installed on SDNS
  • rsync installed on SDNS and all target hosts
  • SSH access between hosts (custom port is fine)

Part 1 — SDNS Setup

1.1 TSIG Key for BIND

We create a dedicated TSIG key that certbot will use to write the DNS challenge TXT record. The key is restricted via update-policy — it can only write _acme-challenge.myip.gr, nothing else in the zone.

tsig-keygen -a hmac-sha256 acme-key > /etc/named/acme.key
chmod 640 /etc/named/acme.key
chown root:named /etc/named/acme.key

Add to named.conf:

include "/etc/named/acme.key";

zone "myip.gr" {
    type master;
    file "/var/named/myip.gr.zone";
    update-policy {
        grant acme-key name _acme-challenge.myip.gr. TXT;
    };
};

rndc reload

1.2 Certbot credentials file

Extract the secret from the key file you just generated:

cat /etc/named/acme.key

Create /etc/letsencrypt/rfc2136.ini:

dns_rfc2136_server = 127.0.0.1
dns_rfc2136_port = 53
dns_rfc2136_name = acme-key
dns_rfc2136_secret = <secret from acme.key>
dns_rfc2136_algorithm = HMAC-SHA256

chmod 600 /etc/letsencrypt/rfc2136.ini

1.3 Issue the wildcard

certbot certonly \
  --dns-rfc2136 \
  --dns-rfc2136-credentials /etc/letsencrypt/rfc2136.ini \
  -d '*.myip.gr' -d 'myip.gr' \
  --preferred-challenges dns-01 \
  --agree-tos -m you@myip.gr --non-interactive

Note: *.myip.gr does not cover the apex myip.gr — you need both -d flags. Also note: wildcards only cover one level deep (sub.myip.gr yes, a.b.myip.gr no).

Certbot sets up a systemd timer for automatic renewal:

systemctl enable --now certbot-renew.timer
systemctl status certbot-renew.timer

1.4 certpull user and staging directory

Each internal host will SSH in as certpull with a restricted key. This user owns the staging directory — nothing else.

useradd -r -m -s /bin/bash certpull
mkdir -p /home/certpull/.ssh
chmod 700 /home/certpull/.ssh
touch /home/certpull/.ssh/authorized_keys
chmod 600 /home/certpull/.ssh/authorized_keys
chown -R certpull:certpull /home/certpull/.ssh

1.5 Install rrsync

rrsync is the tool that restricts an SSH key to rsync-only, read-only, in a specific directory. On AlmaLinux it ships with the rsync package but not in the PATH:

cp /usr/share/doc/rsync/support/rrsync /usr/local/bin/rrsync
chmod 755 /usr/local/bin/rrsync

1.6 Deploy hook

Certbot calls scripts in /etc/letsencrypt/renewal-hooks/deploy/ after every successful renewal. This hook copies the fresh certs into the certpull staging directory:

cat > /etc/letsencrypt/renewal-hooks/deploy/stage-wildcard.sh << 'EOF'
#!/bin/bash
install -m 644 -o certpull -g certpull \
  /etc/letsencrypt/live/myip.gr/fullchain.pem /home/certpull/fullchain.pem
install -m 640 -o certpull -g certpull \
  /etc/letsencrypt/live/myip.gr/privkey.pem /home/certpull/privkey.pem
EOF
chmod +x /etc/letsencrypt/renewal-hooks/deploy/stage-wildcard.sh

Run it once manually to populate the staging directory now:

bash /etc/letsencrypt/renewal-hooks/deploy/stage-wildcard.sh
ls -la /home/certpull/

You should see fullchain.pem (644) and privkey.pem (640), both owned by certpull.

Part 2 — rosso (Apache httpd)

2.1 Generate the pull key

On rosso:

ssh-keygen -t ed25519 -f /root/.ssh/certpull_rosso \
  -C "certpull rosso -> sdns wildcard" -N ""
cat /root/.ssh/certpull_rosso.pub

2.2 Register the key on SDNS

On SDNS, add the public key to /home/certpull/.ssh/authorized_keys. The forced command pins this key to rrsync read-only — it cannot do anything else:

command="rrsync -ro /home/certpull",no-pty,no-agent-forwarding,no-X11-forwarding,no-port-forwarding ssh-ed25519 AAAA...rosso_pubkey... certpull rosso -> sdns wildcard

One line, no line breaks.

2.3 Accept the SDNS host key

The first connection needs to be interactive to accept the SDNS fingerprint. BatchMode=yes in the pull script would block this:

ssh -i /root/.ssh/certpull_rosso -p 65535 certpull@sdns.myip.gr
# type 'yes' at the fingerprint prompt
# you'll get an rrsync error immediately after -- that's fine

2.4 Update the Apache vhosts

All vhosts on rosso were pointing to a commercial cert split across three files (myipgr.crt, myipgr.key, myipgr.ca). Let’s Encrypt’s fullchain.pem combines the certificate and chain in one file, so SSLCertificateChainFile goes away.

One sed pass updates all vhosts at once:

sed -i \
  -e 's|SSLCertificateFile.*|SSLCertificateFile /etc/ssl/wildcard.myip.gr/fullchain.pem|' \
  -e 's|SSLCertificateKeyFile.*|SSLCertificateKeyFile /etc/ssl/wildcard.myip.gr/privkey.pem|' \
  -e '/SSLCertificateChainFile/d' \
  /etc/httpd/conf.d/*.conf

# verify
grep -h "SSLCertificate" /etc/httpd/conf.d/*.conf | sort -u

# syntax check before reload
httpd -t

2.5 Pull script

cat > /usr/local/sbin/pull-wildcard.sh << 'EOF'
#!/bin/bash
set -euo pipefail

KEY=/root/.ssh/certpull_rosso
SDNS=certpull@sdns.myip.gr
PORT=65535
DEST=/etc/ssl/wildcard.myip.gr
LOG=/var/log/pull-wildcard.log

log() { echo "$(date '+%F %T') $*" >> "$LOG"; }

before=$(md5sum "$DEST/fullchain.pem" 2>/dev/null || echo "none")

rsync -az --include='*.pem' --exclude='*' \
  -e "ssh -i $KEY -p $PORT -o BatchMode=yes -o ConnectTimeout=10" \
  "$SDNS":/ "$DEST"/

chown root:root "$DEST/fullchain.pem" "$DEST/privkey.pem"
chmod 644 "$DEST/fullchain.pem"
chmod 600 "$DEST/privkey.pem"

after=$(md5sum "$DEST/fullchain.pem")

if [[ "$before" != "$after" ]]; then
  systemctl reload httpd
  log "cert updated + httpd reloaded"
else
  log "cert unchanged, skipped reload"
fi
EOF
chmod 750 /usr/local/sbin/pull-wildcard.sh

Test it:

mkdir -p /etc/ssl/wildcard.myip.gr
bash /usr/local/sbin/pull-wildcard.sh
systemctl reload httpd

Verify the live certificate:

echo | openssl s_client -connect rosso.myip.gr:443 -servername rosso.myip.gr 2>/dev/null \
  | openssl x509 -noout -issuer -dates

Expected: issuer=... Let's Encrypt ..., notAfter=Aug 27 ...

2.6 Cron

echo "0 3 * * * root /usr/local/sbin/pull-wildcard.sh" \
  > /etc/cron.d/pull-wildcard

Part 3 — gitlab (GitLab CE Omnibus)

GitLab Omnibus manages its own nginx. Two differences from rosso:

  • Cert files must be named after the hostname and live in /etc/gitlab/ssl/
  • Reload is gitlab-ctl hup nginx, not systemctl reload nginx

3.1 Generate the pull key

On the gitlab VM:

ssh-keygen -t ed25519 -f /root/.ssh/certpull_gitlab \
  -C "certpull gitlab -> sdns wildcard" -N ""
cat /root/.ssh/certpull_gitlab.pub

3.2 Register the key on SDNS

Add a second line to /home/certpull/.ssh/authorized_keys on SDNS:

command="rrsync -ro /home/certpull",no-pty,no-agent-forwarding,no-X11-forwarding,no-port-forwarding ssh-ed25519 AAAA...gitlab_pubkey... certpull gitlab -> sdns wildcard

Each host gets its own key and its own line. rrsync handles multiple keys fine — each is independently restricted to read-only on the same staging directory.

3.3 Accept the SDNS host key

ssh -i /root/.ssh/certpull_gitlab -p 65535 certpull@sdns.myip.gr
# type 'yes' — rrsync error is expected and harmless

3.4 Pull script

cat > /usr/local/sbin/pull-wildcard.sh << 'EOF'
#!/bin/bash
set -euo pipefail

KEY=/root/.ssh/certpull_gitlab
SDNS=certpull@sdns.myip.gr
PORT=65535
STAGE=/etc/ssl/wildcard.myip.gr
DEST=/etc/gitlab/ssl
LOG=/var/log/pull-wildcard.log

log() { echo "$(date '+%F %T') $*" >> "$LOG"; }

before=$(md5sum "$DEST/gitlab.myip.gr.crt" 2>/dev/null || echo "none")

rsync -az --include='*.pem' --exclude='*' \
  -e "ssh -i $KEY -p $PORT -o BatchMode=yes -o ConnectTimeout=10" \
  "$SDNS":/ "$STAGE"/

install -m 644 "$STAGE/fullchain.pem" "$DEST/gitlab.myip.gr.crt"
install -m 600 "$STAGE/privkey.pem"   "$DEST/gitlab.myip.gr.key"

after=$(md5sum "$DEST/gitlab.myip.gr.crt")

if [[ "$before" != "$after" ]]; then
  gitlab-ctl hup nginx
  log "cert updated + gitlab nginx reloaded"
else
  log "cert unchanged, skipped reload"
fi
EOF
chmod 750 /usr/local/sbin/pull-wildcard.sh

Test and verify:

mkdir -p /etc/ssl/wildcard.myip.gr
bash /usr/local/sbin/pull-wildcard.sh

echo | openssl s_client -connect gitlab.myip.gr:443 -servername gitlab.myip.gr 2>/dev/null \
  | openssl x509 -noout -issuer -dates

3.5 Cron

echo "0 3 * * * root /usr/local/sbin/pull-wildcard.sh" \
  > /etc/cron.d/pull-wildcard

Adding a New Host

The pattern is always the same five steps:

  1. ssh-keygen -t ed25519 -f /root/.ssh/certpull_<hostname> -C "certpull <hostname> -> sdns wildcard" -N ""
  2. Add pub key to SDNS /home/certpull/.ssh/authorized_keys with the rrsync -ro forced command
  3. Accept the SDNS host key interactively once: ssh -i /root/.ssh/certpull_<hostname> -p <port> certpull@sdns.myip.gr
  4. Write the pull script — copy from an existing host, adjust KEY, DEST, and the reload command
  5. Add cron.d entry

Five minutes per host. The SDNS side never changes after initial setup.

How Renewal Works End-to-End

  1. certbot-renew.timer runs twice a day on SDNS
  2. When the cert is within 30 days of expiry, certbot renews via DNS-01 (writes _acme-challenge TXT to BIND locally, waits 60s, validates with Let’s Encrypt)
  3. On success, the deploy hook runs: fresh fullchain.pem and privkey.pem land in /home/certpull/
  4. Next morning at 03:00, each host’s cron fires, rsync detects the changed checksum, pulls the new cert, reloads its web server
  5. Log entry: cert updated + <service> reloaded

On all other days: rsync runs, checksum matches, log says cert unchanged, skipped reload. Zero unnecessary service restarts.

Security Notes

  • The rrsync -ro forced command means the certpull key can only read from /home/certpull/. Even if the private key leaks from a host, an attacker gains read access to a certificate that’s already being served publicly — nothing more.
  • The TSIG key for BIND is restricted via update-policy to write only _acme-challenge.myip.gr TXT. It cannot modify any other record in the zone.
  • SDNS holds no credentials pointing into the internal network. Compromise of the DNS box does not equal compromise of internal hosts.
  • no-pty,no-agent-forwarding,no-X11-forwarding,no-port-forwarding on every authorized_keys entry — belt and suspenders.
Post Views: 2