The relative proximity of the words "done right" and "split-horizon DNS" makes my insides hurt a little bit.<p>Use DNS validation to allow these internal services to pull ACME certs. There's so much less headache, long-term.<p>Split-horizon DNS (and the tedious make-work it can create when you start needing to mirror public-accessibly records in the private DNS) has always been something to aspire to move away from in my experience.
Once dns-persist-01 becomes available/usable[1], it should make dns validation even easier.<p>[1]: <a href="https://letsencrypt.org/2026/02/18/dns-persist-01" rel="nofollow">https://letsencrypt.org/2026/02/18/dns-persist-01</a>
Also be careful when using split DNS and Tailscale, which increasingly won't work without MagicDNS enabled.
Came here to say the same. I use DNS-Challenge rather than HTTP-challenge, and that makes internal servers trivial.<p>You need a DNS provider which supports API calls (I use DNSimple) but the core is all very straightforward.<p>To prevent having to include DNSimple authentication on the client's internal server I have a small API server on the web which does the Acme work.
I do this exactly, using ACME DNS:<p><a href="https://github.com/acme-dns/acme-dns" rel="nofollow">https://github.com/acme-dns/acme-dns</a><p>CAUTION, though, the last time I downloaded a binary release, ClamAV triggered on it, so I kept my old version which worked. I was using the 1.0 series (without any problems!), and now it seems the project has picked up development again with a 2.0 series.
Just LE a wild card cert and slap it everywhere.
Hmm. I don't really care enough about leaking home network host names because they are all super generic names like 'router', 'laptop', 'tv', 'nas'. So I use my public zone on cloudflare. I just use internal ip addresses (eg: nas.example.com = 10.1.2.3) on the public zone and DNS01 challenge for let's encrypt. Anyone can resolve the ip for any of my hosts, but obviously you'd need to be on the wireguard vpn to hit them.<p>This means that I can always use public DNS servers like 1.1.1.1, 8.8.8.8, nextDNS etc<p>This is not "done right" by any stretch but it's extremely low effort to set up and has never once failed me, unlike countless complex meshy things.
I do the same thing. I'm not worried about them seeing my FQDNs.<p>I use the form of hostname.int.example.com for everything inside my home network. None of which is accessible to the outside world. I use LetsEncrypt with DNS validation to get the certificates.
Fair, but what about names that are specific enough to give an attacker a clue to a potential attack surface, like "authelia.example.com" - now they know you've likely got an Authelia setup, and can start digging for exploitable CVEs etc. I'm in the process of removing all my individual certs and replacing with a wildcard cert served by Traefik. Is that a bad idea?
Can they dig for exploitable CVEs if they're not on the Wireguard network? It is a clue to your infrastructure, but I personally think the simplicity is worth it.
Do the names resolve to publicly routeable IPs? If not, I wouldn't worry about it.
My IaC is on public GitHub. They could do a network scan to find software then fingerprint to find version anyway.<p>Removing attack surface is better than trying to hide it.
This is similar to what I do, except I have my own authoritative DNS servers instead of Cloudflare.<p>I'd prefer this over split DNS, any day.
The real answer here is that configuring HTTPS clients to trust a self-signed cert (or signed by an internal CA) shouldn't be as difficult as it is. I find it extremely annoying that every programming language has it's own idea of where certificates should live instead of just checking the os trust store.
Is there a commonly used language other than Java that doesn’t just defer to the OS trusted CAs by default?
Even if all applications look at the OS trust store, in my experience there's always a gap distributing the CA to every consumer, leading to time spent on debugging from time to time... Maybe that's not the case in perfectly homogeneous or sufficiently small environments where every team uses the same infra / stack.
It's hard to get right when OSs, programming languages, browsers and sometimes other applications have their own opinions about trust stores. I understand why our IT department want corporate devices to use internal CA certs on paper but it just breaks stuff in the real world.
My preferred procedure is to use DNS-01 validation and have no publicly accessible "A" or "AAAA" record for internal services.<p>Or even a more extreme example: <a href="https://crt.sh/?id=27555237869" rel="nofollow">https://crt.sh/?id=27555237869</a> (sorry for any possible crt.sh downtime) - the domain name in question never existed in public or private DNS by itself. It is used only for a WPA3-Enterprise network, as the CN that WiFi clients expect to be present in the RADIUS server certificate, but never resolve. In the public DNS, only the "_acme-challenge" TXT record exists.
Split horizon DNS is not something I'm willing to do.
I'll just rock out with .internal or .home.arpa, have step-ca and bind communicate to each other, either in step-issuer in kube or maybe even rfc2136 if i feel like a bit of the dns-01 strategy is in order. I slap the internal ca root certs everywhere, and keep my home infra out of the crt.sh logs.<p>I get it, I could just do *.mydomain.com and slap that wildcard cert everywhere, but it's still in the public logs..
I use the acme dns-1 challenge on my public domain. That gives you certificates you can use as you see fit, without needing to expose anything else to the public internet.<p>I also use Tailscale so I configure my DNS to use my Tailscale IP addresses. If you don’t want to expose them on a public DNS server you can add them only to an internal DNS server.
> <i>I use the acme dns-1 challenge on my public domain.</i><p>See also perhaps DNS aliasing in case you are not able to dynamically update your 'primary' domain, but can update a secondary or sub-domain:<p>* <a href="https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mode" rel="nofollow">https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mo...</a><p>So if "example.com" is control by Corporate IT, and they don't want 'random' folks fiddling with it, then you can create a "dnsauth.example.com" and point the dns-1 challenge record from "…foo.example.com" to "foo.dnsauth.example.com" (or a completely different domain, like "…example.net").<p>There are DNS servers written strictly focused on this use case:<p>* <a href="https://github.com/acme-dns/acme-dns" rel="nofollow">https://github.com/acme-dns/acme-dns</a><p>Also code that handles a bunch of DNS provider APIs so you don't have to roll your own for ACME client hooks:<p>* <a href="https://github.com/dns-lexicon/dns-lexicon" rel="nofollow">https://github.com/dns-lexicon/dns-lexicon</a>
Moreover you can delegate domain to improve security.<p><a href="https://www.eff.org/deeplinks/2018/02/technical-deep-dive-securing-automation-acme-dns-challenge-validation" rel="nofollow">https://www.eff.org/deeplinks/2018/02/technical-deep-dive-se...</a>
This is crazy. If you have a home network with a few internal services, or some sort of network where you don't control the endpoints, just use DNS validation. That's why it exists.<p>But on hosts you control, you should absolutely provision them with an identity and join the local CA. You're going to need it for a multitude of other reasons.
Personally, I hate split horizon DNS. I prefer the "BeyondCorp" model. I MUCH prefer putting an mTLS cert in my trusted devices over relying on VPNs in same devices. I've yet to see a "clever" DNS setup not cause annoyances.<p>Specifically grafana is nice to be able to see on the phone, and split horizon DNS and corp VPN is a hassle, to say the least, on phones.<p>I bet you can do it with HA-Proxy, but I use <a href="https://github.com/ThomasHabets/sni-router" rel="nofollow">https://github.com/ThomasHabets/sni-router</a>
I was under impression that I understand networking and DNS resolution. It was really hard to follow, the OP did worked hard, just not sure what exact problem was being solved with the proposed solution that isnt already been solved.
I don't agree that tunneling everything through some external facing proxy is "TLS certificates for internal services done right".
I am looking forward to finally using DNS-PERSIST-01 for validation. No more dynamic DNS updates, DNS credentials or forwarding necessary.
I don’t know much in this space, but I find myself wishing there was a dead simple self hosted CA solution and also that trust on first use (à la ssh) was A Thing for self-managed root certs in client implementations. TOFU is such an elegant, good-enough solution for these use cases. Fixed deployment is always still an option, but in this day and age it feels so much like we are unnecessarily still dealing with solved problems
> TLS certificates for internal services* done right<p>* "internal services" = on a single server that is publicly routable
A Github Action running acme.sh that pushes certs to S3 solves the split dns issue for hosts, which can cause all sorts of weirdness after a while. You can then grab a cert on a schedule and even make them wildcard if you want. Then you will get NXDOMAIN if you are not on the VPN so ideally no public traffic.
"Right" without use case can be wrong. And by use case I include scale. For a small team, few machines, some in-place infrastructure may worth it. Smaller than that may be overkill, bigger than that may not be enough, or end being cumbersome, insecure or not work for everyone.
I wonder if the author realizes that getting public certificates results in them being recorded in CT logs.
Why not just map the domain to an internal IP and call it a day? Then the only way it can be accessed is through a VPN. Then use a wildcard so none of leaks into cert transparency logs
I've documented how to securely set up TLS certificates for internal services without creating TLS issues for http clients downstream. All thanks to split-horizon DNS, WAF and ACME protocol. All for free!
to help with passive reconnaissance, here are tuxnet.dev's SSL certificates and associated subdomains:<p><a href="https://www.certkit.io/tools/ct-logs/?query=tuxnet.dev" rel="nofollow">https://www.certkit.io/tools/ct-logs/?query=tuxnet.dev</a>
I use a registered domain with DNS validation and then CNAMEs that I resolve locally. Basically:<p><pre><code> 1. Register a domain ("server.com") and put it on some public DNS that can do DNS validation with acme.sh.
2. Use DNS validation to get a certificate on your domain from Let's Encrypt. You can just grab a wildcard one ("*.server.com").
3. CNAME all of your services on a public DNS to an internal address ("email.server.com" → "server.internal", "plex.server.com" → "server.internal").
4. Resolve your internal address on a local DNS server with an A record ("server.internal" → 192.168.0.123). This can often just be done on your router.
</code></pre>
Since you use DNS validation, you just API keys for your public DNS service that acme.sh can use. No need to have any VPN network interfaces for getting your certificate. Your wildcard certificate also doesn't leak any details about your services.
this is all fine and good, if you are okay broadcasting your internal hostnames.
I suppose it's a trade off some might make.
Don't do this. Public certs are for public services.
[dead]