LetsEncrypt and Apache Macros

If you're like me, you've probably started using LetsEncrypt pretty heavily for your TLS certificate needs. Free, short-lived certificates that have a decent auto-renewal pattern are just a no-brainer when it comes to modern DevOps.

That said, there can be some challenges. While many of you will have moved onto using nginx, Apache still holds a considerable market share for a lot of reasons. I'm still on the Apache stack, and like most good Apache admins, my config files are full of Macros. After all, a VirtualHost config should be a single line and should be enabled with a graceful reload and not a full server stop/start. 

But LetsEncrypt doesn't play nice with Macros, so you've probably started splitting out your configs into something that looks like this: 


    ServerName www.example.com 

    Use SSLDefaults

    Use VirtualHostTemplateMacro example.com

    SSLCertificateFile /etc/letsencrypt/live/www.example.com/cert.pem

    SSLCertificateKeyFile /etc/letsencrypt/live/www.example.com/privkey.pem

    SSLCertificateChainFile /etc/letsencrypt/live/www.example.com/chain.pem


This type of thing is sufficient to get the certbot-auto apache plugin working but its really quite terse and less than ideal. One ServerName per file, and you have to reconfigure things after certbot runs the first time, usually involving a full server restart.

There's a better way and this method also helps you keep multiple servers in sync in a load balancing setup. 

The first method is using mod_proxy in the http (not https) site config. First enable mod proxy (probably via a2enmod if you're on Ubuntu) ... and inside a VirtualHost config, you add the following code: 

ProxyPass /.well-known/acme-challenge/ http://internal.letsencrypt.example.ca/.well-known/acme-challenge/

ProxyPassReverse /.well-known/acme-challenge/ http://internal.letsencrypt.example.ca/.well-known/acme-challenge/

You then setup a virtualhost for the internal.letsencrypt subdomain (it doesnt have to be externally reachable)


    DocumentRoot /var/www/html/

    ServerName internal.letsencrypt.example.ca

    <Directory "/var/www/html/.well-known/acme-challenge">

        Options None

        AllowOverride None

        Require all granted

        AddDefaultCharset off



Then you can use the webroot method of issuing letsencrypt certificates.

certbot-auto certonly --webroot -w /var/www/html -d www.example.org,example.org

Any http-01 type challenges will hit the proxy and forward over to your subdomain. In this way you can have any endpoint server forward to a box dedicated to LetsEncrypt internally and then rsync out the certs when they're generated. 

If you're using mod_rewrite and the ProxyPass isn't engaging (because your RewriteRule engages first), add  

RewriteCond %{REQUEST_URI} ^/\.well\-known

RewriteRule . - [L] 

To your rules, which will turn them off for the .well-known directory. 

Once this is working you can go back to using Macro's properly without worrying about certbot not understanding how to parse them. 

If you've already generated your certificates, you can update your /etc/letsencrypt/renewal/example.ca.conf configs to properly use the webroot method. (Just look at another domain you generated for example of how to set the configs correctly). You can use --dry-run or --force-renew to test the new config will work on renewal.

After that, to keep them updated, all you need to do is add to your cron.

/usr/local/bin/certbot-auto --no-self-upgrade renew --webroot -w /var/www/html

And then you can just 

/usr/bin/rsync -ar --delete /etc/letsencrypt/./ otherappserver.example.org:/etc/letsencrypt/

To sync between app servers. Keeping in mind that with --delete it will delete any files on the otherappserver that don't match the machine you're running the command from. Once you have a script working, you can issue, renew and sync your LetsEncrypt certs centrally.

If you use HAProxy, you can also just forward http requests for .well-known/acme-challenge/ to your new internal webserver.

To make life easier and test everything before running the letsencrypt client, you can also just: 

mkdir -p /var/www/html/.well-known/acme-challenge

echo test > /var/www/html/.well-known/acme-challenge/test.txt  


curl http://www.example.ca/.well-known/acme-challenge/test.txt

If you can see the test file, then LetsEncrypt should be able to see the challenge files created there as well. 

Some webserver configs also limit access to dot-files, a typically smart move, but one that can restrict access to the .well-known directory. The following can replace your .dotfiles rules, usually found in apache2.conf

<FilesMatch "^\.(?!well-known)">

    Require all denied


<DirectoryMatch "^\.(?!well-known)|\/\.(?!well-known)">

    Require all denied


Which translated means deny to anything starting with a ., except if it has a negative lookahead to .well-known, in which case the rule doesn't apply. 

Hope thats helpful.


Subscribe to RSS - letsencrypt