Revisiting JH0project’s DNS for DN42

2026-01-02

In my previous article about JH0project's DNS for DN42, I described how BIND9, ExaBGP, and BGP anycast provide DNS across our servers. DNS records were stored in static zone files and distributed with the container images, which worked well when those records rarely changed.

Over time, maintaining the zones became less convenient, especially when temporary records were needed for certificate validation. I wanted to make those changes easier while keeping the availability of the existing anycast setup.

Why change the original setup?

The original approach was straightforward: keep the BIND zone files in Git, sign them with DNSSEC, and include them in the container image deployed to each server.

At the time, I didn't have distributed storage across the infrastructure. Git and Docker images provided a convenient way to distribute the same zones to every node while I was getting everything running. Records rarely changed, so the extra work was acceptable.

However, every record update required editing the zone, signing it again, building an image, and deploying it across the DNS nodes. DNSSEC signatures also expire, so even unchanged zones needed periodic maintenance.

This became inconvenient when I wanted to automate DNS updates. A temporary TXT record for certificate validation should not require a container deployment.

By then, Garage S3 was already running across the infrastructure. With shared object storage available, it made sense to revisit how DNS records were distributed.

Why PowerDNS and Lightning Stream?

I wanted an authoritative DNS server with an API for managing records and built-in DNSSEC handling. Maturity and community adoption also mattered, since DNS is something the rest of the infrastructure depends on.

BIND9, Unbound, and PowerDNS were the names I already knew. Having used BIND9 and Unbound, I focused on what PowerDNS offered for authoritative DNS.

PowerDNS's choice of storage backends was particularly useful. Compared with the file-based configurations I had been managing, it offered a way to separate record updates from configuration deployment. Its LMDB backend supports both DNSSEC and record management through the API.

My initial thought was to use a SQL backend with CockroachDB, which was already deployed in my infrastructure. However, that would require checking compatibility between PowerDNS and CockroachDB. It would also make the shared database another dependency for serving authoritative queries.

I wanted each DNS node to hold enough local data to keep answering queries independently. An interruption to synchronization should delay record updates without immediately taking existing DNS answers offline.

While looking through the backend options, I came across Lightning Stream. It synchronizes LMDB databases between servers through S3-compatible object storage, including the DNSSEC keys used by PowerDNS.

That fitted the infrastructure I already had. Each PowerDNS node could serve queries from its own LMDB database, while Lightning Stream exchanged changes through Garage.

It also avoided configuring direct replication relationships between DNS nodes. With a conventional BIND primary-secondary setup, I would need to configure the primary's address or provide a way to discover it. Here, the nodes synchronize through the shared object storage instead.

An S3 sync outage affects record propagation while PowerDNS and its local LMDB remain healthy. PowerDNS continues using the local data it already has, and Lightning Stream can catch up when the S3 endpoint becomes reachable again.

The tradeoff is asynchronous replication. An update is not immediately visible everywhere, and concurrent changes to the same data can conflict. For the relatively infrequent authoritative record changes in this setup, that was an acceptable compromise.

Moving authoritative DNS to PowerDNS

I moved the authoritative zones into PowerDNS and kept BIND9 for recursive DNS. The two services now have separate anycast addresses and can be configured and deployed independently.

BIND continues to handle recursive resolution, caching, DNSSEC validation, and encrypted DNS. PowerDNS handles the authoritative zones and DNSSEC signing.

Both retain the existing ExaBGP and exacheck integration, so health checks continue to control their anycast advertisements.

Using separate anycast addresses also means BIND9 does not depend on the PowerDNS instance on its own node. If that PowerDNS instance fails, its authoritative route is withdrawn after the health check detects the failure. Once routing converges, BIND9's authoritative queries can reach another healthy node while its own recursive service continues running.

For a query under jh0project.dn42, the recursive resolver follows the DN42 delegation when it is not already cached, then queries the authoritative anycast address. PowerDNS answers from its local LMDB data. A cached answer can be returned directly by BIND.

On each authoritative node, PowerDNS reads from local LMDB storage. Lightning Stream handles synchronization through Garage in the background. If synchronization becomes unavailable, existing nodes can continue serving the data they already have, although new changes will not reach other nodes until replication recovers.

Record updates now go through the PowerDNS API, without rebuilding or redeploying the DNS containers.

Automating DN42 certificate renewal

After moving to this architecture, DN42 HTTPS was still using old, expired certificates. The DNS migration provided the missing piece for restoring automated issuance and renewal.

With the static zone workflow, creating and removing a DNS-01 challenge record would have involved signing and deploying zone changes. PowerDNS's API allows acme.sh to manage those temporary TXT records directly.

The process creates the challenge record, waits for it to propagate, and lets burble's DN42 ACME certificate authority validate it. The CA checks the TXT record through DNS; the PowerDNS API is used by acme.sh to publish it. After issuance, the certificates are distributed to Traefik across the nodes.

Because replication is asynchronous, the challenge record needs time to reach the authoritative servers before validation begins. The automation includes a DNS propagation wait for this.

The certificates expire after 30 days, so renewal is scheduled at 25 days, leaving roughly five days to recover from a failed attempt before expiration.

Conclusion

DNS records can now be updated without building a new container image, and temporary TXT records are practical enough to automate DN42 certificate renewal. BIND9 still handles recursive queries, while PowerDNS serves authoritative data locally and Lightning Stream distributes changes through Garage.

Git was a convenient way to distribute zones when I didn't have shared storage. Reusing Garage later let me remove those deployment steps while keeping each DNS node able to answer independently. The compromise is that updates take time to reach every node. Concurrent changes still need coordination, and certificate renewal can fail if the propagation wait is too short. Health checks and routing also need time to react to a failed node. For our relatively infrequent record changes, this is a tradeoff I can accept.

A useful next step would be to check the challenge TXT record on the authoritative nodes before asking the CA to validate it. I'd also like to make replication delays easier to spot, since a node answering DNS queries doesn't necessarily have the latest records.