I use certificates obtained from a self-hosted certificate authority (CA) on my Proxmox Backup Server (PBS), with Certbot taking care of renewing them automatically. This worked well enough with a simple renewal hook to copy the newly issued certificate into place and reload proxmox-backup-proxy.

Something like this:

renew_hook = cp /etc/letsencrypt/live/pbs.lan/fullchain.pem /etc/proxmox-backup/proxy.pem; cp /etc/letsencrypt/live/pbs.lan/privkey.pem /etc/proxmox-backup/proxy.key; chown root:backup /etc/proxmox-backup/proxy.pem /etc/proxmox-backup/proxy.key; chmod 640 /etc/proxmox-backup/proxy.pem /etc/proxmox-backup/proxy.key; systemctl reload proxmox-backup-proxy

And this worked fine. Mostly.

Except when using a self-signed CA, I think it’s best practice to have your certs expire on a fairly short window, like every 24 hours. And cycling proxmox-backup-proxy every 24 hours was a pain point because it caused any running Verify or Garbage Collection job to fail.1 That’s annoying.

A certificate renewal happening in the middle of a multi-hour verify job shouldn’t result in the verify job getting killed. Or maybe it should? I didn’t architect PBS so who is to say, although I can say that backup jobs from Proxmox VE fail to PBS when the cert is expired so perhaps there’s a good reason.

So, off to make certificate renewal a little smarter on my end.

Note: a bunch of the fun diagrams below are from me and ChatGPT figuring this out together. I’m smart enough to know what I don’t know and dumb enough to think AI will save me some time solving it. Although, spoiler, AI did save me some time!

The Problem

Proxmox Backup Server serves its HTTPS certificate from:

/etc/proxmox-backup/proxy.pem
/etc/proxmox-backup/proxy.key

After those files change, proxmox-backup-proxy has to be reloaded for the running service to begin serving the new certificate.

Proxmox docs recommends a reload rather than a restart specifically because restarting the proxy can interrupt active backup jobs.

Unfortunately, reload isn’t entirely free either. In particular, PBS verification and garbage collection jobs can be disrupted by it. And those jobs were taking 4+ hours to chew through about 1TB of backups on a spinning disk.

My Certbot post-renew hook didn’t care what PBS was doing:

Certificate renews
      ↓
Copy certificate
      ↓
Reload PBS proxy
      ↓
¯\_(ツ)_/¯

If a garbage collection or verification job happened to be running at that moment, well, too bad, so sad.

What I really wanted was:

Certificate renews
      ↓
Stage certificate
      ↓
GC or Verify running? ── Yes ──→ Wait
      │                            │
      No                           │
      ↓                            │
Install certificate ←──────────────┘
      ↓
Reload PBS proxy

Although, that Wait is a risky step! More on that later.

Certbot should still renew normally, on its schedule. The only thing being deferred is activating the new certificate.

First Attempt: Copy Now, Reload Later

The first attempt at solving this looked like:

  1. Copy the new certificate into /etc/proxmox-backup/.
  2. Check whether PBS had a verification or garbage collection job running.
  3. Reload immediately if none of those things were running.
  4. Otherwise, have a systemd timer periodically check again.

That almost worked.

I started seeing this:

WARNING: certificate fingerprint does not match expected fingerprint!
expected:    6a:9a:10:…
certificate validation failed - Certificate fingerprint was not confirmed.
Error: client error (Connect)

That problem was the order of operations.

I had replaced:

/etc/proxmox-backup/proxy.pem

with the new certificate, but deliberately hadn’t reloaded proxmox-backup-proxy yet.

The state of the machine therefore looked like this:

/etc/proxmox-backup/proxy.pem
        ↓
   NEW certificate

AND

proxmox-backup-proxy
        ↓
   OLD certificate

Then my script tried to run:

proxmox-backup-manager task list

to find out whether it was safe to reload the cert (safe == no GC or Verify jobs running).

PBS complained that the certificate fingerprint it expected and the certificate actually being given back didn’t match. Because they didn’t match. Because I updated the one on disk. Darn it.

The fix was to stop modifying the live certificate files until the exact moment I was ready to reload the proxy.

Stage the Certificate Instead

The Certbot deploy hook now copies newly renewed certificates into staging files:

/etc/proxmox-backup/proxy.pem.pending
/etc/proxmox-backup/proxy.key.pending

The currently active ones stay as:

proxy.pem
proxy.key

This means the certificate on disk continues to match the certificate being served by the currently running proxy while we wait for those GC and Verify jobs to wrap up. (Foreshadowing.)

A systemd timer periodically checks PBS. Once no verification or garbage collection jobs are running, it moves the pending certificate into place and immediately reloads the proxy.

There are a few pieces needed to make this work.

The Certbot Deploy Script

First, I created:

/usr/local/sbin/pbs-cert-deploy

with:

#!/bin/bash
set -euo pipefail

CERT_DIR=“/etc/proxmox-backup”

install -o root -g backup -m 0640 \
    /etc/letsencrypt/live/pbs.lan/fullchain.pem \
    “${CERT_DIR}/proxy.pem.pending”

install -o root -g backup -m 0640 \
    /etc/letsencrypt/live/pbs.lan/privkey.pem \
    “${CERT_DIR}/proxy.key.pending”

# Try activating the certificate immediately. If PBS is busy,
# the systemd timer will try again later.
systemctl start pbs-cert-reload-if-safe.service

And made it executable:

chmod 755 /usr/local/sbin/pbs-cert-deploy

The use of install here takes care of copying the file, setting its owner/group, and setting its permissions in one operation. This is a cool command I didn’t know about before.

PBS wants ownership of root:backup and permissions 0640.

Most importantly, nothing touches the certificate currently used by the proxy. We’ve got those nice *.pending files for the newly renwed certs.

Change the Certbot Renewal Hook

The giant Certbot renewal command can now be replaced with:

renew_hook = /usr/local/sbin/pbs-cert-deploy

That definitely reads nicer! When Certbot successfully renews the certificate it stages the new certificate and asks systemd to see whether it can activate it. That script doesn’t need to know anything about PBS jobs itself.

Checking PBS for Running Jobs

PBS provides its running task list with:

proxmox-backup-manager task list

It can also return JSON if you add --output-format json, which we will do.

Rather than waiting until PBS has absolutely nothing running, I only care about the jobs I’ve found can be disrupted by the proxy reload: verification and garbage collection.

Using jq, I can look for those worker types:

proxmox-backup-manager task list --output-format json \
    | jq -e ‘any(.[]; .worker_type == “garbage_collection”
                      or .worker_type == “verificationjob”
                      or .worker_type == “verify”)’

If any of those workers exist, it isn’t time to reload yet.

This allows normal PBS activity, including backups, to continue without unnecessarily delaying certificate activation.

The Safe Reload Script

The actual decision-making happens in:

/usr/local/sbin/pbs-cert-reload-if-safe

Which contains:

#!/bin/bash
set -euo pipefail

CERT_DIR=“/etc/proxmox-backup”
PENDING_CERT=“${CERT_DIR}/proxy.pem.pending”
PENDING_KEY=“${CERT_DIR}/proxy.key.pending”

# Nothing waiting to be activated.
[[ -f “$PENDING_CERT” && -f “$PENDING_KEY” ]] || exit 0

# Don’t reload the proxy while a garbage collection or
# verification task is running.
if proxmox-backup-manager task list --output-format json \
    | jq -e ‘any(.[]; .worker_type == “garbage_collection”
                      or .worker_type == “verificationjob”
                      or .worker_type == “verify”)’ >/dev/null; then

    logger -t pbs-cert-reload-if-safe \
        "GC or verification task running; deferring TLS certificate activation"

    exit 0
fi

logger -t pbs-cert-reload-if-safe \
    "No GC or verification tasks running; activating renewed TLS certificate"

# Move the staged certificate into place.
mv “$PENDING_CERT” “${CERT_DIR}/proxy.pem”
mv “$PENDING_KEY” “${CERT_DIR}/proxy.key”

# Immediately make the running proxy use it.
systemctl reload proxmox-backup-proxy

logger -t pbs-cert-reload-if-safe \
    "Renewed TLS certificate activated successfully"

Then:

chmod 755 /usr/local/sbin/pbs-cert-reload-if-safe

This version uses the existence of the two .pending files as the indicator that a certificate needs to be activated.

Originally I used a separate marker under /run, but there’s no reason to maintain a second piece of state. More importantly, /run is ephemeral. If PBS rebooted while a certificate was waiting, the marker would disappear while the pending certificate remained.2

The pending certificate itself is a much better marker.

No pending certificate? Nothing to do.

Pending certificate? Keep checking until it’s safe to reload.

The systemd Service

Next comes a very small oneshot systemd service:

/etc/systemd/system/pbs-cert-reload-if-safe.service

containing:

[Unit]
Description=Activate renewed PBS TLS certificate when safe

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/pbs-cert-reload-if-safe

There isn’t much happening here. systemd runs the script once and the script either:

  • Finds no pending certificate and exits.
  • Finds a GC/verify job and exits without changing anything.
  • Finds a pending certificate and no GC/verify job, installs the certificate, and reloads the proxy.

The systemd Timer

Finally, let’s run that service on a timer:

/etc/systemd/system/pbs-cert-reload-if-safe.timer

with:

[Unit]
Description=Check for pending PBS TLS certificate activation

[Timer]
OnBootSec=5min
OnUnitActiveSec=5min

[Install]
WantedBy=timers.target

Enable it:

systemctl daemon-reload
systemctl enable --now pbs-cert-reload-if-safe.timer

Five minutes seems reasonable.

Let’s Encrypt certificates are renewed well ahead of expiration. (More foreshadowing.) If a verification job has another three hours to run, continuing to serve the existing perfectly valid certificate for another three hours is fine. Assuming, of course, the GC or Validation jobs finish within that 3 hour window.

How It All Fits Together

With everything installed, the entire process looks like this.

Certbot renews:

/etc/letsencrypt/live/pbs.lan/fullchain.pem
/etc/letsencrypt/live/pbs.lan/privkey.pem

Its renewal hook runs:

/usr/local/sbin/pbs-cert-deploy

That creates:

/etc/proxmox-backup/proxy.pem.pending
/etc/proxmox-backup/proxy.key.pending

Then it immediately starts:

pbs-cert-reload-if-safe.service

The service asks PBS for its running tasks.

If there’s no GC or verification running:

pending certificate
       ↓
live certificate
       ↓
reload proxmox-backup-proxy
       ↓
done

If there’s a verify job running:

pending certificate
       ↓
verify running
       ↓
do nothing
       ↓
timer waits five minutes
       ↓
check again
       ↓
verify still running?
       ↓
keep waiting

Eventually the verify finishes:

timer fires
       ↓
no GC/verify
       ↓
pending → live
       ↓
reload proxy
       ↓
new certificate active

All the while, PBS continues serving the old certificate and its certificate files remain internally consistent.

Testing It

I first tested the PBS task detection by running:

proxmox-backup-manager task list --output-format json \
    | jq ‘.[] | select(
        .worker_type == “garbage_collection”
        or .worker_type == “verificationjob”
        or .worker_type == “verify”
      )’

With no matching job running, there should be no output. Now start a verify or garbage collection job and run it again. The matching worker should appear.

You can also manually test the safe reload service:

systemctl start pbs-cert-reload-if-safe.service

And see what it decided with:

journalctl -t pbs-cert-reload-if-safe

Or check the whole service:

journalctl -u pbs-cert-reload-if-safe.service

The timer itself can be checked with:

systemctl status pbs-cert-reload-if-safe.timer

or:

systemctl list-timers pbs-cert-reload-if-safe.timer

Verifying the Certificate

It’s also useful to compare the certificate PBS has on disk with what the running proxy is actually serving.

The certificate on disk:

openssl x509 \
    -in /etc/proxmox-backup/proxy.pem \
    -noout -fingerprint -sha256

And the certificate currently being served:

openssl s_client \
    -connect 127.0.0.1:8007 \
    -servername pbs.lan \
    </dev/null 2>/dev/null |
openssl x509 -noout -fingerprint -sha256

Those should match during normal operation.

This was also a handy way to understand my original fingerprint error. After copying the renewed certificate over proxy.pem but before reloading the proxy, these commands returned two different fingerprints.

The End Result

This ended up being slightly more complicated than simply doing:

systemctl reload proxmox-backup-proxy

But not that much more complicated, once you understand the two different scripts, the systemd service, and the timer. It seems like a lot but I suppose it’s not.

The certificate renewal process and the certificate activation process are now deliberately separate. That’s a benefit.

A problem with short-lived certificates and my garbage collection schedule

Here’s where all that foreshadowing above leads.

I did this to myself. By having certificates that expire every 24 hours, and by having my GC job run every 4 hours, I ended up in a situation where:

  1. GC job starts
  2. Cert renewal happens, but we can’t reload the cert yet because a GC job is running
  3. GC job takes ~4.5 hours to finish
  4. Another GC job begins (every 4 hours)
  5. Cert renewal never happens because a GC job is always running.

And… now the old cert expires after 24 hours and my backup jobs start failing. Not good. More alerts hitting my phone.

The solution was simple, just change the GC job frequency to once per week. Each GC job was freeing up only ~4GB of space, so moving it from 6 times per day to 1 time per week should not cost me much in terms of free space on the drive.

The Use of AI in this project

This solution was just given to me by an AI chatbot. At least most of it was. I read it, understood it, tried it out, and gave the chatbot some feedback so we could land on a better solution. It helped me diagnose some errors along the way. I helped it understand what my goals were and provide some more context about the solution. This whole thing took me 2-3 hours messing around, versus the week I might have otherwise spent reading forum posts to figure it out.

I’m a working dad now, I could use the time back.

  1. A fun thing I learned during all this is I was running garbage collection way too often. I’m down to a much more sensible once a week now. ↩︎

  2. Editor’s note, it took writing this blog post to realise I actually left the /run implementation in place on the live machine. Fixing that now! ↩︎