Let’s Encrypt for Private Networks


My Goal

I want to have Chrome stop telling me that the other SSL-encrypted devices on my private network are insecure. Chrome is unhappy because these devices use self-signed certificates. What I’d like to do is get certificates that are signed by a recognized certificate authority. While I don’t mind a good technical challenge, I’d rather not spend a bunch of cash on my certificates.

Security Disclaimer

There are probably ways to be even more secure than the methodology I’m going to present here. If you want to go into full-on tinfoil hat mode here, cool. I mean, this is an article about a guy setting up secure SSL certificates for his home network, so I’m not going to judge. However, for the sake of keeping this guide readable, I’m going to assume that hardening occurs after you get your setup working. Don’t blindly follow guides about security you read on the internet – audit your environments and look for other possible intrusion/weaknesses, and ameliorate them appropriately. Check file permissions. Don’t run as root unless you absolutely have to. Read the source code of new things you’re going to install on your machine(s).

At the same time, don’t make your life super hard during the proof of concept phase when you’re still trying to understand the concepts and get the basic setup working.

My Approach

In a private network situation, the level of security required is usually pretty low. In fact, you could argue that spending any sort of time or effort on this sort of thing is pointless on a private network, and you might be right! But hey, Chrome is annoying and this was a learning opportunity for me.

Usually getting an ‘legit’ signed certificate requires paying money. In some cases, a lot of money. These kind of certificates are really intended for large businesses to buy and deploy to secure web applications that require security when customer data and actual cash are on the line. Making this even remotely feasible requires that we are able to get SSL certificates for free in some easy-to-automate fashion. Thankfully, back in 2016, Let’s Encrypt launched. There are several benefits to this (relatively) new SSL Certificate Authority – in particular, they offer signed SSL certificates for free! They also support a wide range of automation around issuing and renewing certificates that allows people like me to do common SSL-related tasks without spending a lot of time on maintenance.

The downside to Let’s Encrypt is that their tools need to be able to verify that you own a domain in order to issue or renew a certificate. You’re not going to easily get a certificate that works on machines on a private network without some effort. In this article, I’ll explain how I solved these problems and managed to deploy a Let’s Encrypt certificate on a set of SSL-enabled hosts on my private network.

The Tools

To solve this problem, I used:

  • A domain name. Since most private networks use the same IP ranges it’s not feasible to get a certificate for a private network IP address. Instead, we’ll use a domain name for the certificate, which will guarantee uniqueness.
  • DNS service. Via DNS, we can use subdomains to resolve to our private network IPs. Many domain name registrars offer DNS service, and sometimes the cost of the DNS service is included in the domain name purchase. I opted for Google Domains since I already pay for other services from Google.
  • A Linux host on the public internet, ideally with a static IP. There are many ways of solving this problem. I already rent a “Nanode” VPS from Linode so this was an obvious choice for me. This won’t need 100% uptime, nor does it require much in the way of CPU, RAM, or disk space. You could maybe use some host on your private (home?) network for this, but I won’t be covering the port forwarding and potential security concerns with this approach.

For the rest of this guide, I’ll be using:

  • EXAMPLE.COM as the domain name (since it is literally designed to be an example domain name)
  • PUBLIC_HOST to refer to the Linux host on the public internet
  • PUBLIC_IP_ADDR as the IP address of the Linux host on the public internet.

A few simplifications I’m going to be assuming to make things work for most people without overly complicating this guide:

  • Your DNS hosting service doesn’t natively support ACME DNS TXT challenges. We’ll cover what those are in a minute, but since some service providers don’t support this, and there may be security implications to allowing API calls to modify your DNS records, we’ll assume you need/want the setup I’m describing.
  • I’ll only be setting things up for IPv4, so if you are an IPv6 user you may need to modify certain areas of this guide to suit your needs.
  • The PUBLIC_HOST is available for running at least monthly cron jobs. You could spin it up and down only when you need it, but that’s more complicated than I want to get into today.
  • You don’t want to use subdomains of your subdomains. For example, you’re only using names like myname.EXAMPLE.COM and not thisname.thatname.EXAMPLE.COM. A wildcard only matches up to the next . in the domain name, so you’d need additional certificates to handle sub-subdomains.
  • You don’t want to run a SSL-encrypted service on PUBLIC_HOST. This is actually not that hard to add support for, but I’m omitting it in the guide for brevity’s sake.
  • You don’t want to host something at your bare domain, ie https://EXAMPLE.COM. Another fairly simple extension, but I’m leaving it out for now.

In order to create a Let’s Encrypt certificate that will work for our private IPs, we’re going to need to use the PUBLIC_HOST to authorize the creation of a wildcard certificate. That is, a certificate for *.EXAMPLE.COM. Once we have this, we can create DNS records for hosts that resolve to private IPs, and this one certificate will work for all of them.

Let’s Encrypt leverages a standard called Automated Certificate Management Environment (ACME). The trick to this is that while there are several methods of verifying a domain supported by ACME, the only one that works for wildcard certificates is DNS “TXT” record modification. So, in order to automatically issue and renew our Let’s Encrypt SSL certificates, we’re going to need an ACME DNS-compatible environment. Luckily, there are open source projects that can help with this!

Alternative Approaches

This guide is going to assume that you can’t automate the process of changing your DNS TXT records via your DNS provider. I made the choice to use Google Domains before I knew they didn’t offer an API. You can probably simplify your setup by using DNS service from another company. Cloudflare offers free DNS service, even if you don’t purchase a domain through them. Using a DNS provider that has an API that Let’s Encrypt supports will skip a few steps in this guide.

Like I said earlier, some providers support this, and some don’t. If you picked a DNS provider that supports changing DNS TXT records via an API, you might want to not use this guide, and instead have a look at acme.sh support for DNS APIs.

The downside to this approach is that you’re going to be putting an API key for your DNS provider on a host on the public internet. If PUBLIC_HOST gets compromised, the attackers can change whatever DNS settings that API key gives them access to. This guide partitions your DNS so that most of the DNS handling is done by your provider (hopefully securely!) and only the certificate issuing related bits are done on PUBLIC_HOST.

DNS Setup

Getting your domain name and DNS set up are outside the scope of this guide, and really quite dependent on the service providers you choose. I’ll assume you can get to the control panel for your domain/DNS provider(s) and enter DNS records there by this point.

Once you’ve acquired your domain name and associated the DNS service with it, the first step is to set up some records that delegate the work of serving the ACME DNS records from your DNS provider to a service running on PUBLIC_HOST.

First, we have the A record to establish the domain name for our DNS server:

  • Type: A – this is an A record, the type of record that maps a domain name to an IPv4 address.
  • Name: ns1 – this is the name to map, relative to the top-level domain. So this actually means ns1.EXAMPLE.COM.
  • Data: PUBLIC_IP_ADDR – the IP address we want this domain name to resolve to.
  • TTL: 60 – cache this record for 60 seconds. for testing purposes, it’s OK to set this fairly low – if we screw up, we don’t want to wait hours for DNS changes to propagate. Later, once everything’s set up properly, we can up this value and save some bandwidth.

Next, we need the NS record to delegate nameserver duties for our TXT record verification to this host:

  • Type: NS – this is a nameserver record – used for delegating resolving subdomain names to a particular server.
  • Name: acme – any domains under the subdomain acme.EXAMPLE.COM are resolved by the DNS server in this record.
  • Data: ns1.EXAMPLE.COM. – The name of the nameserver to delegate to. Note the trailing ., which means this is an absolute domain name rather than relative (we used a relative name in the previous record)
  • TTL: 60 – cache this record for 60 seconds. (see the same note as above)

The combination of these two DNS records essentially says “there is a host at PUBLIC_IP_ADDR that I want to refer to as ns1.EXAMPLE.COM. If anyone asks to resolve something like *.acme.EXAMPLE.COM, send it to that host.”

If we were to try and resolve something like myhost.acme.EXAMPLE.COM right now, it would probably fail, because we haven’t set up the acme-dns server to listen to these DNS requests… so that’s our next step.

Setting up acme-dns

In order to serve up our DNS TXT records for validation, we’ll use acme-dns. The instructions on that page should get you to the point where you have the acme-dns binary installed, and a config file at /etc/acme-dns/config.cfg. You can safely ignore most of the rest of the instructions and docs for now. We’ll need to make a few changes to the config file, though.

In the [general] section, change:

  • listen = PUBLIC_IP_ADDR – we want the DNS server to listen on the public IP of the Linux host.
  • domain = acme.EXAMPLE.COM – this nameserver is serving requests for acme.EXAMPLE.COM – just like we told our DNS host in the NS record in the previous section.
  • nsname = ns1.EXAMPLE.COM – the DNS name of this host is ns1.EXAMPLE.COM – again, matching what we told our DNS host in the A record in the previous section.
  • records = [] – you can comment out or delete the other static records, unless you have some reason beyond the scope of this guide why you need other records.

In the [api] section, change:

  • ip = 127.0.0.1 – the certificate registration/renewal requests will be coming from this machine, so to keep things secure, let’s just listen locally.
  • port = 5353 – you can choose any unused port here, just make a note of it.
  • tls = none – since we’re only listening on 127.0.0.1, we don’t really need encryption. No packets are being transmitted. Setting up SSL for the thing we’re using to set up SSL has a bit of an ouroboros quality to it. If it bothers you, you can always use the certificate we’re about to generate once we’ve generated it.

Once the configuration file is saved, manually launch acme-dns from the command line on PUBLIC_HOST as root. We’re going to need a bit of its output during this next step.

Setting up certbot and acme-dns-certbot

Now we’re ready to start playing with certbot and getting it to play nice with our acme-dns installation. certbot is Let’s Encrypt’s frontend for certificate issuing and renewal. The installation instructions are on the EFF website. If you can, install the package that works with the OS you’re using. Since certbot doesn’t technically require root privileges, and we’re not auto-configuring a web server with SSL using it, I prefer to create another user to run my certbot tasks. My certbot command lines will reflect this.

We do need one more piece, though – a connector between certbot and acme-dns. The author of acme-dns has provided such a script called acme-dns-auth.py. Put this in the certbot user’s $HOME/acme-dns directory.

At the top of the file are a set of configuration options, check/edit the following:

  • ACMEDNS_URL = "http://127.0.0.1:5353" – Note that this should match the port from the [api] section above. If you opted to set up SSL, you would want https here instead of http.
  • STORAGE_PATH = "/home/certbot/acme-dns/acmedns.json" – This file should be writable by the user invoking certbot. If you are opting to run as root, or you want to play with permissions, you could use something under /etc if you wanted.

The certbot executable needs some working directories, so let’s make some:

mkdir -p $HOME/letsencrypt/cfg $HOME/letsencrypt/log $HOME/letsencrypt/work

Now that’s ready to go, so finally it’s time to run certbot for the first time!

(Psst. Hey. acme-dns is still running in another terminal, right? You can see the output? Just checking.)

As the certbot user on PUBLIC_HOST:

certbot certonly --manual --manual-auth-hook $HOME/acme-dns/acme-dns-auth.py \
    --config-dir $HOME/letsencrypt/cfg --logs-dir $HOME/letsencrypt/log --work-dir $HOME/letsencrypt/work/ \
    --preferred-challenges dns --debug-challenges \
    -d *.EXAMPLE.COM

Okay. With any luck, you’ll see some output on the acme-dns terminal (not the certbot terminal!) that looks like:

DEBU[0005] TXT updated  subdomain=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee

This is important, and we need to know it to complete the loop here. Back over to the DNS provider’s control panel!

See, before the ACME protocol will allow the issuance of *.EXAMPLE.COM, it wants a TXT record updated under the domain _acme-challenge.EXAMPLE.COM. We need to direct that to the name that acme-dns has assigned right there. So what we want is one more DNS record in the DNS provider’s control panel:

  • Type: CNAME – this record is like an alias – a name that points to another name, rather than to an IP
  • Name: _acme-challenge – the ACME protocol expects this name to be used
  • Data: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.acme.EXAMPLE.COM. – trailing . again, because this is an absolute domain name. This being a subdomain of acme.EXAMPLE.COM means that the request for this record will be sent to PUBLIC_HOST, where acme-dns is running.
  • TTL: 60 – again, when you’re done debugging, turn this up.

You can test this record with something like dig _acme-cahllenge.acme.deon.app TXT – you should see traffic on the acme-dns terminal, and you should get back a response pretty quickly. If not, debug your setup.

At this point, certbot should be usable for automatic renewal of the wildcard certificate!

As the certbot user on PUBLIC_HOST:

certbot renew --dry-run --manual-public-ip-logging-ok --manual-auth-hook=$HOME/acme-dns/acme-dns-auth.py \
    --config-dir $HOME/letsencrypt/cfg --logs-dir $HOME/letsencrypt/log --work-dir $HOME/letsencrypt/work/

If you get back output that includes “Congratulations, all renewals succeeded.” – you’re good to go!

Next Steps

Now we have some much more straightforward things to do.

Make acme-dns a Service

The documentation for acme-dns has instructions for this, and additionally it includes a systemctl file for itself.

Automate Renewals

We’ll want to run the certbot renew command on a regular basis, since Let’s Encrypt certs are only good for 90 days. Once a month is probably a good place to start.

I created a script called /home/certbot/letsencrypt/renew.sh with:

certbot renew --manual-public-ip-logging-ok --manual-auth-hook=$HOME/acme-dns/acme-dns-auth.py \
    --config-dir $HOME/letsencrypt/cfg --logs-dir $HOME/letsencrypt/log --work-dir $HOME/letsencrypt/work/

Then in the certbot user’s crontab:

0 5 1 * * /home/certbot/letsencrypt/renew.sh > /home/certbot/letsencrypt/output.log 2>&1

Configure Private Network Subdomains

Using the DNS provider’s control panel, we can set up subdomains of our domain that point to the private IPs on our private network. Those records will be of the form:

  • Type: A – a record that resolves a name to an IPv4 address
  • Name: myserver – However you want to refer to your private machine.
  • Data: 192.168.0.20 – Whatever the private IP is of your machine on the private network.
  • TTL: 60 – again, for debug set this low, for production set it high. If you use static IPs on your private network, you can set this to days if you want.

Deploy Certificates to the Private Network Hosts

The methodology here will vary depending on what your private network looks like, and what services you have running. As a starting point, you can rsync the /home/certbot/letsencrypt/cfg/live/EXAMPLE.COM directory to a central machine on your private network, then copy them (with the correct user/permissions) to where they need to go. Keep in mind that you might need to make special configuration allowances or restart services when the certificates change.

Also remember that you need to start referring to the hosts on your private networks using their DNS names (myserver.EXAMPLE.COM rather than 192.168.0.20) in order for this to work. The SSL certificate is valid for *.EXAMPLE.COM, remember? If you have bookmarks, etc, make sure you update them.

Consider Additional Privacy and Security Concerns

  • Would you prefer to run acme-dns only during the renewals? That leaves one less open port to worry about, but you’ll have to control permissions on starting/stopping the acme-dns server, since it must be run as root.
  • Are you sensitive to disclosing the IPs of hosts on your private network? Some people may consider this an issue. I don’t, personally, as it’s hard to list all the subdomains of a domain, and knowledge of the names/ips of the hosts on my private network is not particularly interesting to anyone else.
  • How have you controlled permissions to the certificate files on PUBLIC_HOST and the copies that are made elsewhere in your private network? Double check your permissions, and make sure that it is as difficult as possible to get access to these files.