I want to have Chrome stop telling me that the other SSL-encrypted devices on my private network are insecure. Chrome is unhappy because these devices use self-signed certificates. What I’d like to do is get certificates that are signed by a recognized certificate authority. While I don’t mind a good technical challenge, I’d rather not spend a bunch of cash on my certificates.
There are probably ways to be even more secure than the methodology I’m going to present here. If you want to go into full-on tinfoil hat mode here, cool. I mean, this is an article about a guy setting up secure SSL certificates for his home network, so I’m not going to judge. However, for the sake of keeping this guide readable, I’m going to assume that hardening occurs after you get your setup working. Don’t blindly follow guides about security you read on the internet – audit your environments and look for other possible intrusion/weaknesses, and ameliorate them appropriately. Check file permissions. Don’t run as root unless you absolutely have to. Read the source code of new things you’re going to install on your machine(s).
At the same time, don’t make your life super hard during the proof of concept phase when you’re still trying to understand the concepts and get the basic setup working.
In a private network situation, the level of security required is usually pretty low. In fact, you could argue that spending any sort of time or effort on this sort of thing is pointless on a private network, and you might be right! But hey, Chrome is annoying and this was a learning opportunity for me.
Usually getting an ‘legit’ signed certificate requires paying money. In some cases, a lot of money. These kind of certificates are really intended for large businesses to buy and deploy to secure web applications that require security when customer data and actual cash are on the line. Making this even remotely feasible requires that we are able to get SSL certificates for free in some easy-to-automate fashion. Thankfully, back in 2016, Let’s Encrypt launched. There are several benefits to this (relatively) new SSL Certificate Authority – in particular, they offer signed SSL certificates for free! They also support a wide range of automation around issuing and renewing certificates that allows people like me to do common SSL-related tasks without spending a lot of time on maintenance.
The downside to Let’s Encrypt is that their tools need to be able to verify that you own a domain in order to issue or renew a certificate. You’re not going to easily get a certificate that works on machines on a private network without some effort. In this article, I’ll explain how I solved these problems and managed to deploy a Let’s Encrypt certificate on a set of SSL-enabled hosts on my private network.
To solve this problem, I used:
- A domain name. Since most private networks use the same IP ranges it’s not feasible to get a certificate for a private network IP address. Instead, we’ll use a domain name for the certificate, which will guarantee uniqueness.
- DNS service. Via DNS, we can use subdomains to resolve to our private network IPs. Many domain name registrars offer DNS service, and sometimes the cost of the DNS service is included in the domain name purchase. I opted for Google Domains since I already pay for other services from Google.
- A Linux host on the public internet, ideally with a static IP. There are many ways of solving this problem. I already rent a “Nanode” VPS from Linode so this was an obvious choice for me. This won’t need 100% uptime, nor does it require much in the way of CPU, RAM, or disk space. You could maybe use some host on your private (home?) network for this, but I won’t be covering the port forwarding and potential security concerns with this approach.
For the rest of this guide, I’ll be using:
EXAMPLE.COMas the domain name (since it is literally designed to be an example domain name)
PUBLIC_HOSTto refer to the Linux host on the public internet
PUBLIC_IP_ADDRas the IP address of the Linux host on the public internet.
A few simplifications I’m going to be assuming to make things work for most people without overly complicating this guide:
- Your DNS hosting service doesn’t natively support ACME DNS TXT challenges. We’ll cover what those are in a minute, but since some service providers don’t support this, and there may be security implications to allowing API calls to modify your DNS records, we’ll assume you need/want the setup I’m describing.
- I’ll only be setting things up for IPv4, so if you are an IPv6 user you may need to modify certain areas of this guide to suit your needs.
PUBLIC_HOSTis available for running at least monthly cron jobs. You could spin it up and down only when you need it, but that’s more complicated than I want to get into today.
- You don’t want to use subdomains of your subdomains. For example, you’re only using names like
thisname.thatname.EXAMPLE.COM. A wildcard only matches up to the next
.in the domain name, so you’d need additional certificates to handle sub-subdomains.
- You don’t want to run a SSL-encrypted service on
PUBLIC_HOST. This is actually not that hard to add support for, but I’m omitting it in the guide for brevity’s sake.
- You don’t want to host something at your bare domain, ie
https://EXAMPLE.COM. Another fairly simple extension, but I’m leaving it out for now.
In order to create a Let’s Encrypt certificate that will work for our private IPs, we’re going to need to use the
PUBLIC_HOST to authorize the creation of a wildcard certificate. That is, a certificate for
*.EXAMPLE.COM. Once we have this, we can create DNS records for hosts that resolve to private IPs, and this one certificate will work for all of them.
Let’s Encrypt leverages a standard called Automated Certificate Management Environment (ACME). The trick to this is that while there are several methods of verifying a domain supported by ACME, the only one that works for wildcard certificates is DNS “TXT” record modification. So, in order to automatically issue and renew our Let’s Encrypt SSL certificates, we’re going to need an ACME DNS-compatible environment. Luckily, there are open source projects that can help with this!
This guide is going to assume that you can’t automate the process of changing your DNS TXT records via your DNS provider. I made the choice to use Google Domains before I knew they didn’t offer an API. You can probably simplify your setup by using DNS service from another company. Cloudflare offers free DNS service, even if you don’t purchase a domain through them. Using a DNS provider that has an API that Let’s Encrypt supports will skip a few steps in this guide.
Like I said earlier, some providers support this, and some don’t. If you picked a DNS provider that supports changing DNS TXT records via an API, you might want to not use this guide, and instead have a look at acme.sh support for DNS APIs.
The downside to this approach is that you’re going to be putting an API key for your DNS provider on a host on the public internet. If
PUBLIC_HOST gets compromised, the attackers can change whatever DNS settings that API key gives them access to. This guide partitions your DNS so that most of the DNS handling is done by your provider (hopefully securely!) and only the certificate issuing related bits are done on
Getting your domain name and DNS set up are outside the scope of this guide, and really quite dependent on the service providers you choose. I’ll assume you can get to the control panel for your domain/DNS provider(s) and enter DNS records there by this point.
Once you’ve acquired your domain name and associated the DNS service with it, the first step is to set up some records that delegate the work of serving the ACME DNS records from your DNS provider to a service running on PUBLIC_HOST.
First, we have the
A record to establish the domain name for our DNS server:
A– this is an A record, the type of record that maps a domain name to an IPv4 address.
ns1– this is the name to map, relative to the top-level domain. So this actually means
PUBLIC_IP_ADDR– the IP address we want this domain name to resolve to.
60– cache this record for 60 seconds. for testing purposes, it’s OK to set this fairly low – if we screw up, we don’t want to wait hours for DNS changes to propagate. Later, once everything’s set up properly, we can up this value and save some bandwidth.
Next, we need the
NS record to delegate nameserver duties for our TXT record verification to this host:
NS– this is a nameserver record – used for delegating resolving subdomain names to a particular server.
acme– any domains under the subdomain
acme.EXAMPLE.COMare resolved by the DNS server in this record.
ns1.EXAMPLE.COM.– The name of the nameserver to delegate to. Note the trailing
., which means this is an absolute domain name rather than relative (we used a relative name in the previous record)
60– cache this record for 60 seconds. (see the same note as above)
The combination of these two DNS records essentially says “there is a host at
PUBLIC_IP_ADDR that I want to refer to as
ns1.EXAMPLE.COM. If anyone asks to resolve something like
*.acme.EXAMPLE.COM, send it to that host.”
If we were to try and resolve something like myhost.acme.EXAMPLE.COM right now, it would probably fail, because we haven’t set up the acme-dns server to listen to these DNS requests… so that’s our next step.
Setting up acme-dns
In order to serve up our DNS TXT records for validation, we’ll use acme-dns. The instructions on that page should get you to the point where you have the
acme-dns binary installed, and a config file at
/etc/acme-dns/config.cfg. You can safely ignore most of the rest of the instructions and docs for now. We’ll need to make a few changes to the config file, though.
[general] section, change:
listen = PUBLIC_IP_ADDR– we want the DNS server to listen on the public IP of the Linux host.
domain = acme.EXAMPLE.COM– this nameserver is serving requests for
acme.EXAMPLE.COM– just like we told our DNS host in the
NSrecord in the previous section.
nsname = ns1.EXAMPLE.COM– the DNS name of this host is
ns1.EXAMPLE.COM– again, matching what we told our DNS host in the
Arecord in the previous section.
records = – you can comment out or delete the other static records, unless you have some reason beyond the scope of this guide why you need other records.
[api] section, change:
ip = 127.0.0.1– the certificate registration/renewal requests will be coming from this machine, so to keep things secure, let’s just listen locally.
port = 5353– you can choose any unused port here, just make a note of it.
tls = none– since we’re only listening on 127.0.0.1, we don’t really need encryption. No packets are being transmitted. Setting up SSL for the thing we’re using to set up SSL has a bit of an ouroboros quality to it. If it bothers you, you can always use the certificate we’re about to generate once we’ve generated it.
Once the configuration file is saved, manually launch
acme-dns from the command line on
PUBLIC_HOST as root. We’re going to need a bit of its output during this next step.
Setting up certbot and acme-dns-certbot
Now we’re ready to start playing with
certbot and getting it to play nice with our
certbot is Let’s Encrypt’s frontend for certificate issuing and renewal. The installation instructions are on the EFF website. If you can, install the package that works with the OS you’re using. Since
certbot doesn’t technically require root privileges, and we’re not auto-configuring a web server with SSL using it, I prefer to create another user to run my
certbot tasks. My
certbot command lines will reflect this.
We do need one more piece, though – a connector between
acme-dns. The author of
acme-dns has provided such a script called acme-dns-auth.py. Put this in the certbot user’s
At the top of the file are a set of configuration options, check/edit the following:
ACMEDNS_URL = "http://127.0.0.1:5353"– Note that this should match the port from the
[api]section above. If you opted to set up SSL, you would want
httpshere instead of
STORAGE_PATH = "/home/certbot/acme-dns/acmedns.json"– This file should be writable by the user invoking
certbot. If you are opting to run as root, or you want to play with permissions, you could use something under
/etcif you wanted.
certbot executable needs some working directories, so let’s make some:
mkdir -p $HOME/letsencrypt/cfg $HOME/letsencrypt/log $HOME/letsencrypt/work
Now that’s ready to go, so finally it’s time to run
certbot for the first time!
acme-dns is still running in another terminal, right? You can see the output? Just checking.)
certbot user on PUBLIC_HOST:
certbot certonly --manual --manual-auth-hook $HOME/acme-dns/acme-dns-auth.py \ --config-dir $HOME/letsencrypt/cfg --logs-dir $HOME/letsencrypt/log --work-dir $HOME/letsencrypt/work/ \ --preferred-challenges dns --debug-challenges \ -d *.EXAMPLE.COM
Okay. With any luck, you’ll see some output on the
acme-dns terminal (not the
certbot terminal!) that looks like:
DEBU TXT updated subdomain=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
This is important, and we need to know it to complete the loop here. Back over to the DNS provider’s control panel!
See, before the ACME protocol will allow the issuance of
*.EXAMPLE.COM, it wants a TXT record updated under the domain
_acme-challenge.EXAMPLE.COM. We need to direct that to the name that
acme-dns has assigned right there. So what we want is one more DNS record in the DNS provider’s control panel:
CNAME– this record is like an alias – a name that points to another name, rather than to an IP
_acme-challenge– the ACME protocol expects this name to be used
.again, because this is an absolute domain name. This being a subdomain of
acme.EXAMPLE.COMmeans that the request for this record will be sent to PUBLIC_HOST, where
60– again, when you’re done debugging, turn this up.
You can test this record with something like
dig _acme-cahllenge.acme.deon.app TXT – you should see traffic on the
acme-dns terminal, and you should get back a response pretty quickly. If not, debug your setup.
At this point,
certbot should be usable for automatic renewal of the wildcard certificate!
certbot user on PUBLIC_HOST:
certbot renew --dry-run --manual-public-ip-logging-ok --manual-auth-hook=$HOME/acme-dns/acme-dns-auth.py \ --config-dir $HOME/letsencrypt/cfg --logs-dir $HOME/letsencrypt/log --work-dir $HOME/letsencrypt/work/
If you get back output that includes “Congratulations, all renewals succeeded.” – you’re good to go!
Now we have some much more straightforward things to do.
Make acme-dns a Service
The documentation for
acme-dns has instructions for this, and additionally it includes a
systemctl file for itself.
We’ll want to run the
certbot renew command on a regular basis, since Let’s Encrypt certs are only good for 90 days. Once a month is probably a good place to start.
I created a script called
certbot renew --manual-public-ip-logging-ok --manual-auth-hook=$HOME/acme-dns/acme-dns-auth.py \ --config-dir $HOME/letsencrypt/cfg --logs-dir $HOME/letsencrypt/log --work-dir $HOME/letsencrypt/work/
Then in the
certbot user’s crontab:
0 5 1 * * /home/certbot/letsencrypt/renew.sh > /home/certbot/letsencrypt/output.log 2>&1
Configure Private Network Subdomains
Using the DNS provider’s control panel, we can set up subdomains of our domain that point to the private IPs on our private network. Those records will be of the form:
A– a record that resolves a name to an IPv4 address
myserver– However you want to refer to your private machine.
192.168.0.20– Whatever the private IP is of your machine on the private network.
60– again, for debug set this low, for production set it high. If you use static IPs on your private network, you can set this to days if you want.
Deploy Certificates to the Private Network Hosts
The methodology here will vary depending on what your private network looks like, and what services you have running. As a starting point, you can
/home/certbot/letsencrypt/cfg/live/EXAMPLE.COM directory to a central machine on your private network, then copy them (with the correct user/permissions) to where they need to go. Keep in mind that you might need to make special configuration allowances or restart services when the certificates change.
Also remember that you need to start referring to the hosts on your private networks using their DNS names (
myserver.EXAMPLE.COM rather than 192.168.0.20) in order for this to work. The SSL certificate is valid for
*.EXAMPLE.COM, remember? If you have bookmarks, etc, make sure you update them.
Consider Additional Privacy and Security Concerns
- Would you prefer to run
acme-dnsonly during the renewals? That leaves one less open port to worry about, but you’ll have to control permissions on starting/stopping the
acme-dnsserver, since it must be run as root.
- Are you sensitive to disclosing the IPs of hosts on your private network? Some people may consider this an issue. I don’t, personally, as it’s hard to list all the subdomains of a domain, and knowledge of the names/ips of the hosts on my private network is not particularly interesting to anyone else.
- How have you controlled permissions to the certificate files on
PUBLIC_HOSTand the copies that are made elsewhere in your private network? Double check your permissions, and make sure that it is as difficult as possible to get access to these files.