Multi-Hosts TLS Certificate

It is sometimes convenient to have a domain distributed over two or more machines. This technique, as old as DNS, is interesting to spread the load between multiple hosts, or to provide a bit of high availability. Indeed, if a host becomes inaccessible, at least half of the requests will continue to be successful.

However, since TLS connections have become the norm, and certificates should be renewed automatically, it could be hard to control the validation and the distribution.

I will present you a technique which, with the help of a finely configured web server, allows to get a different certificate on each machine, but usable for the same subdomain.

The trick is, when receiving the verification request, to either serve the file if you have it, or to transmit the request to the next server: it will return the file if it has it, or will return a 404 error.

You have to be careful not to create loops, because you can quickly end up in a situation where no machine has the requested file and therefore calls the next one which does not have it either…

Use case

This trick is helpfull in a small setup: with more than 3 or 4 servers, it is better to use a deployment strategy between hosts, for example based on ACME client hooks: after validation, the client initiates a connection to the other servers to update their certificate and notify their respective web servers.

I made this installation for my minio servers: each one is accessible through minio.nemunai.re but also individually on minio0.nemunai.re and minio1.nemunai.re. The two servers are in a distinct geographic area, and are not able to execute commands on each other.

So I tried to create two independent certificates, one for each machine:

• the first host must obtain a certificate for minio0.nemunai.re and minio.nemunai.re ;
• the second host must obtain a certificate for minio1.nemunai.re and minio.nemunai.re.

At the end, each of the two servers will be able to answer the requests of minio.nemunai.re, our goal, whatever the status of the other machine.

Let’s Encrypt doesn’t revoke certificates already issued for a domain, so both certificates are valid at the same time for the same domain.

DNS Configuration

Since we want to keep the architecture simple, we are leaving the load balancing to the DNS: good resolver implementations will do round-robin over our records, ensuring a distribution of clients between each host.

The first step is to create our sub-domains as follows:

minio0.nemunai.re.   3600 IN A 198.51.100.10
minio1.nemunai.re.   3600 IN A 198.51.100.11

This registers a dedicated domain for each host (so we can access it easily when needed). Then we register the common domain with which we will do load-balancing:

minio.nemunai.re.    3600 IN A 198.51.100.10
                     3600 IN A 198.51.100.11

We do the same for AAAA records, with IPv6 address.

Reverse Proxy Configuration

We know that when using the ACME http-01 test, we’ll receive a request on minio.nemunai.re/.well-known/acme-challenge. But the problem we have to solve is that when one of the servers will ask for a certificate, the validation request has a 50% chance to arrive on the other host.

This is where we will configure the reverse proxy in order to send the request to the next server if it does not have the answer of the challenge.

On both servers, we’ll handle acme-challenge that way:

  location /.well-known/acme-challenge {
      alias /var/lib/acme/webroot;

      try_files $uri $uri/ @minio_neighbor;
  }

And we also need that the default server handle acme-challenges in the same directory:

server {
  listen 80 default;

  [...]

  location /.well-known/acme-challenge {
      alias /var/lib/acme/webroot;
  }
}

With the try_files instruction, our reverse proxy will first look for the file in the local directory, then, if it doesn’t exists, it forwards the request to its neighbor server @minio_neighbor.

We declare @minio_neighbor in another block:

  location @storage_neighbor {
      proxy_pass http://198.51.100.11;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }

On the other server, we’ll obviously use the address of the other host.

This configuration avoid infinite proxy loop, as if the challenge response is not find localy, we ask the other host, but it’ll not pass through the try_files file as this is the default server that’ll respond. If it’s the other host that have the challenge response, the first server asked will answer it thanks to the proxy, otherwise, the proxify answer will be a 404 error.

And that’s it: we can launch our ACME client on both host, each one can handle the challenge reponse of the other.