Using Step-CA Certificates with Traefik

In my homelab, I have a handful of services which are never exposed to the open internet and therefore do not need a domain name such as myservice.mydomainname.com. Instead, I can get by with a domain name that only resolves within my home network such as myservice.local.¹ Let’s talk about how we set this up.

tl;dr

You can use a self-hosted certificate authority with Traefik to generate SSL certificates for homelab domains such as myservice.local.

traefik.yaml:

certificateResolvers:
  stepca:
    acme:
      email: __REPLACE_WITH_YOUR_EMAIL__
      storage: "/etc/traefik/acme/stepca-acme.json"
      caServer: "https://__REPLACE_WITH_YOUR_STEP_CA_DOMAIN__/acme/acme/directory"
      certificatesDuration: 24
      tlsChallenge: true
      httpChallenge:
        entryPoint: insecure

Then in some dynamic config:

labels:
    - "traefik.enable=true"
    - "traefik.http.routers.myservice.rule=Host(`myservice.local`)"
    - "traefik.http.routers.myservice.entrypoints=secure"
    - "traefik.http.routers.myservice.tls=true"
    - "traefik.http.routers.myservice.tls.certresolver=stepca"

Ensure your CA is trusted by the host machine and the docker container running Traefik, e.g.:

volumes:
    - /etc/ssl:/etc/ssl:ro

A traffic machine

If the above tl;dr whet your appetite, or you want to see how the sausage is made, read on.

I am using Traefik as a reverse proxy within my homelab, so that I can host services such as my Mastodon instance toot.mattedwards.org. The basic flow is:

1. A request is made from the open internet to toot.mattedwards.org
2. My DNS provider translates toot.mattedwards.org to an IP address which points to my homelab²
3. The router in my homelab gets the request to toot.mattedwards.org on port 80/443 and passes that along based on it’s port forwarding config to a specific box in my homelab
4. The box receiving this request is running Traefik, and has dynamic config to know where to send requests to toot.mattedwards.org
5. The VM servicing Mastodon at toot.mattedwards.org receives the request and replies

Or more simply: Internet -> Homelab Router -> Traefik box -> Mastodon box.

Traefik will also perform SSL termination. Meaning Traefik gets an encrypted request (e.g. https://toot.mattedwards.org, note the “s” in “https”), decrypts it, sends it along to the Mastodon box, and finally re-encrypts the response. This allows all Internet-exposed communication to be encrypted, while not needing to setup an SSL certificate on each box inside my homelab.

For services which have a public domain like toot.mattedwards.org, I can rely on Let’s Encrypt and Traefik’s implementation of it to get and periodically renew the appropriate SSL certificates that will be trusted by my devices, my wife’s devices, and anyone else’s devices.³ More on how I set this up below.

SSL for local domains

While Traefik’s default implementation of Let’s Encrypt works great for services I identify with public domain names, what about requests to local-only domains such as myservice.local? I can’t create a DNS entry at Cloudflare against myservice.local, and therefore Let’s Encrypt cannot issue a certificate against it.

This is where two tools come together:

• Traefik’s ACME validation
• Smallstep’s Step CA

To keep this blog post to a reasonable length, I will not cover how to setup your own Step Certificate Authority inside your homelab. However, here is an excellent tutorial for doing it.

I can configure Traefik to fetch certificates using my homelab hosted Certificate Authority. Neat!

Setup of a Traefik box

I have Traefik running as a docker container. Here is an abridged version of that compose file (I am excluding stuff specific to my own network setup).

services:
  traefik:
    image: "traefik:v2.11"
    restart: always
    ports:
      - "80:80"
      - "443:443"
    environment:
      - CF_API_EMAIL=${CF_API_EMAIL}
      - CF_DNS_API_TOKEN=${CF_DNS_API_TOKEN}
      - CF_ZONE_API_TOKEN=${CF_ZONE_API_TOKEN}
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik.yaml:/etc/traefik/traefik.yaml:ro
      - ./fileProviders:/etc/traefik/fileProviders:ro
      - /etc/ssl:/etc/ssl:ro
      - ./le-certs:/etc/traefik/acme
      - ./logs:/logs
    networks:
      - internal
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefikdashboard.rule=Host(`traefik.local`)"
      - "traefik.http.routers.traefikdashboard.service=api@internal"
      - "traefik.http.routers.traefikdashboard.entrypoints=secure"
      - "traefik.http.routers.traefikdashboard.tls=true"
      - "traefik.http.routers.traefikdashboard.tls.certresolver=stepca"

And the accompanying .env file, which is in Ansible template form:

NAMECHEAP_API_USER={{ traefik_namecheap_api_user }}
NAMECHEAP_API_KEY={{ traefik_namecheap_api_key }}

CF_API_EMAIL={{ traefik_cf_api_email }}
CF_DNS_API_TOKEN={{ traefik_cf_dns_api_token }}
CF_ZONE_API_TOKEN={{ traefik_cf_zone_api_token }}

If you’re using Namecheap, or Cloudflare, or any other service for which you need to authenticate you can set up your own values in .env.

Let’s discuss two major parts of that docker-compose.yaml file. First, the volumes I have mapped.

• - /etc/ssl:/etc/ssl maps the local machines SSL certificates into the Traefik container. This will be important later, when we have bootstrapped the machine using Smallstep’s step utility.
• - ./le-certs:/etc/traefik/acme providers a bind mount for Traefik to put certificate data into.

Second, the Traefik docker compose labels:

• - "traefik.http.routers.traefikdashboard.rule=Host(`traefik.local`)" creates an http router in Traefik so that traefik knows how to handle any incoming traffic to the host traefik.local.
• - "traefik.http.routers.traefikdashboard.tls=true" tells Traefik that TLS should be enabled on these connections.
• - "traefik.http.routers.traefikdashboard.tls.certresolver=stepca" identifies which certificate resolver should be used on this router to get certificates.

In my traefik.yaml static configuration file (a portion is shown below), I define my certificate resolvers.

certificatesResolvers:
  cloudflare:
    acme:
      email: __REPLACE_WITH_YOUR_EMAIL__
      storage: "/etc/traefik/acme/acme.json"
      dnsChallenge:
        provider: cloudflare
        resolvers:
          - "1.1.1.1:53"
          - "8.8.8.8:53"
  stepca:
    acme:
      email: __REPLACE_WITH_YOUR_EMAIL__
      storage: "/etc/traefik/acme/stepca-acme.json"
      caServer: "https://__REPLACE_WITH_YOUR_STEP_CA_DOMAIN__/acme/acme/directory"
      certificatesDuration: 24
      tlsChallenge: true
      httpChallenge:
        entryPoint: insecure

I have two different certificate resolvers defined. The first is cloudflare. This resolver is used in the dynamic configuration of services such as my Mastodon server toot.mattedwards.org. Traefik knows to perform a DNS challenge against Cloudflare in order to get a valid certificate from Let’s Encrypt for that domain.

The second is stepca. This resolver will be used in the dynamic configuration of locally hosted services, e.g. myservice.local. Traefik will reach out to my Smallstep Certificate Authority in order to obtain a certificate for those .local domains I setup in my homelab.

There are a couple lines in the file above which you would need to customize:

• email: __REPLACE_WITH_YOUR_EMAIL__. You can’t use mine!
• Note that storage: is different for each certificate resolver. Don’t use the same JSON file for everything!
• caServer: "https://__REPLACE_WITH_YOUR_STEP_CA_DOMAIN__/acme/acme/directory" you would put your Certificate Authority ACME endpoint in here. Note that I did not need to define caServer for the cloudflare resolver because Traefik inherently knows where Let’s Encrypt hosts it.
• certificatesDuration: 24 means 24 hours, which is how I’ve configured my certificate authority. Yours may have been differently configured and you should update this value to match.
• httpChallenge: as there is a limitation in Traefik⁴ preventing you from having more than one DNS challenge providers in use at the same time.

Finally, in the docker-compose.yaml file above there is some environment variable mapping going on.

environment:
    - CF_API_EMAIL=${CF_API_EMAIL}
    - CF_DNS_API_TOKEN=${CF_DNS_API_TOKEN}
    - CF_ZONE_API_TOKEN=${CF_ZONE_API_TOKEN}

These variables are how Traefik knows to authenticate with the cloudflare certificate resolver I’ve defined in traefik.yaml. Learn more about which providers (other than Cloudflare) are supported in Traefik’s docs.

The dynamic configuration setup for each Traefik service can now ask for one or the other certificate resolver defined in traefik.yaml.

A dynamic file provided config for Plex may look like:

http:
  routers:
    plexrouter:
      rule: "Host(`plex.local`)"
      entryPoints:
        - secure
      tls:
        certResolver: stepca
      service: plex

  services:
    plex:
      loadBalancer:
        servers:
          - url: "http://192.168.2.10:32400"

Or a docker labels dynamic configuration for Mastodon:

labels:
    - "traefik.enable=true"
    - "traefik.http.routers.mastodon.rule=Host(`toot.mattedwards.org`)"
    - "traefik.http.routers.mastodon.entrypoints=secure"
    - "traefik.http.routers.mastodon.tls=true"
    - "traefik.http.routers.mastodon.tls.certresolver=cloudflare"

Note that I’m using the cloudflare resolver for public-facing domains *.mattedwards.org and I’m using stepca for internal/local domains *.local.

Traefik host machine bootstrapping

Recall the volume mapping in the compose file above: - /etc/ssl:/etc/ssl:ro. That is taking the host machine’s certificate store and allowing the Traefik docker container to read them. Why?

As part of standing up a Certificate Authority in your homelab, you would have created a root and intermediate signing key that would sign all the certificate you generate. Your devices would then need to trust that a public certificate signed with that intermediate signing key. Since these custom created certificates are not known to “the world”, no one would trust them… except you and your devices you’ve setup to do just that.

We need to install the root certificate from your certificate authority into the host machine running docker, and then map that trusted root cert into the Traefik container. This way, your host machine and your Traefik container will all trust it. This is what that volume mapping of /etc/ssl is accomplishing.

To get your host machine to trust the root cert, we have to bootstrap it. We can do that by installing the step command line utility (Github link) from Smallstep, and then running:

step ca bootstrap --ca-url "${CA_URL}" --fingerprint "${CA_FINGERPRINT}" --install --force && \
update-ca-certificates

You would have your CA_URL and CA_FINGERPRINT from your install of the Smallstep Certificate Authority.

Docs from Smallstep about this setup.

Wrap up

Now, Traefik will trust your certificate authority and you can generate new certificates for your locally hosted services!

A couple of caveats and reminders:

• You need a self-hosted certificate authority to make the above work. I didn’t explain how to create one, but Smallstep has some good documentation about doing it.
• The root certificate from your homelab certificate authority needs to be trusted before the SSL certs being generated will be trusted. If you have a wife, kid, friend, or neighbor accessing your self-hosted services you’ll need to install the cert on their machine before that nifty lock icon and green address bar will show up in a web browser for them. If you don’t, they will get SSL errors!
• You need some DNS in your homelab that is translating myservice.local into an IP address of your Traefik box. If you go outside your homelab network, myservice.local won’t resolve unless you tunnel back into your home network a la Tailscale/Wireguard.

I think this is super neat and a fun thing to play around with. While it’s by no means required to have SSL in your homelab for locally-hosted services which are never exposed to the internet, there is a huge feeling of accomplishment that comes from setting this up and getting it to work.

Thanks for reading!