Sep 16, 2020

Nginx, Let's Encrypt, IPv6, HTTP/2 and healthcheck

While working on the generic configuration set for Nginx/PHP/MySQL web applications in Docker, the most difficult problem I met was writing scripts and configuration files for Nginx. I need a simple, generic, small and reliable solution to issue and renew Let's Encrypt certificates in Docker, reload Nginx, support HTTP2 and IPv6.

Each of these tasks alone has a pretty straightforward solution. It became much more complicated when I started writing a generic solution one could copy-paste among different sites and domains.

A requirement for a solution to be generic means I don't want to build my own Nginx image, I want to use an official image from the Docker Hub: nginx:alpine. I use Acme.sh that just works everywhere.

A requirement to be simple means that I don't a separate container to issue certificates, no http proxy to process HTTP requests from Let'sEncrypt. 

And a requirement to be reliable means I don't pass a docker unix socket to a container to reload Nginx when the certificate is renewed

I use a posix sh script running in background in the Nginx container with a plain "webroot" method to pass Let's Encrypt authorization. I implement a healthcheck for Docker.

It has a limitation of scalability and replication: this solution is for a single Nginx service setup.

Here is a standalone pattern for Nginx configs:

https://github.com/grikdotnet/docker-patterns/tree/master/1.nginx-acme.sh-healthcheck

Jun 7, 2020

Fast deployment of Nginx/PHP/MySQL with LetsEncrypt, HTTP2 and IPv6 using Docker Swarm.

Periodically I deploy simple web sites with Nginx, PHP and MySQL. It is usually one or two virtual servers, all with similar requirements and configs. I came to a "setup-and-forget" set of configs to deploy a site and upgrade in a few years.

Here is my "Infrastructure As A Code" for LEMP sites you can deploy pretty fast.

Functional requirements:
  1. HTTPS with ACME-issued certificate (Let'sEncrypt, BuyPass, and others)
  2. Local deployment with a self-issued certificate
  3. Nginx with HTTP2
  4. IPv6 support with a single IP, as most virtual servers have
  5. Simple upgrade and migration of services among server providers
  6. Use environment variables for secrets such as a database password
  7. Healthcheck for services
  8. Works in Linux as production environment
  9. Works in Docker Desktop for both Mac and Windows as development environment
Non-functional requirements:
  1. Lightweight to fit a cheap virtual server with 1Gb RAM.
  2. Use official well-known repositories and images, no third-party dependencies
  3. Single site per server. VPS are cheap enough. It can serve multiple domains, of course.


Why Docker Swarm?

Kubernetes could be a great choice, but it consumes gigabytes of RAM. It just won't work in a tiny VPS. Ansible/Vagrant are popular tools among system administrators, but they don't provide service decoupling. I like upgrading PHP, Nginx and MySQL by downloading new images for containers with a single command.

Problems:
  1. Docker Swarm does not provide cron jobs scheduling. ACME renewal should be done each 2 months. I have a dedicated article describing this solution.
  2. Swarm does not support IPv6 options, and Docker documentation asks for a /80 IPv6 range.
Keypoints:

* You can see an "acme.sh" package used instead of Certbot. Acme.sh in the "Nginx mode" configures Nginx to issue certificates. This way certificates can be obtained before starting Nginx with production configs. Local deployment generates a self-signed certificate.

* Recent Nginx and MySQL images support init scripts in "/docker-entrypoint.d/". You can see scripts mounted to the Nginx and MySQL containers installing openssl and acme.sh packages, initializing database when container starts. This way you can avoid maintaining a custom image.

* Official PHP images do not support init scripts, so I substitute the entyrypoint script with a custom one, and still use an official docker image for PHP.

* There is no simple solution to work with IPv6 in Docker Swarm. Docker uses IPv4 NAT to route traffic, and IPv6 is just something Docker not designed for.
Docker documentation offers is to use Compose-file version 2, ask a provider for a /80 range, and set the "enable_ipv6" flag for the docker engine. This requires too much manual work.
Another way is to add an IPv6 NAT service in a Docker container. But the best way I found is to run Nginx in a "host" network.

* A host network can't be used in Docker desktop in Windows and Mac. This is solved with YAML inheritance in a "docker-compose.override.yml" file.
A local deployment with a "docker-compose up" uses both "docker-compose.yml" and "docker-compose.override.yml" files.
In production the command "docker stack up -c docker-compose.yml mystack" reads just the "docker-compose.yml" file and runs Nginx in a host network mode.
 
* Environment variables take precedence over values in the "mysql/db.env" file. This trick allows using values from a .env file in a local deployment, and environment variables are used in production.

* A "clear_env = no" clause in "fpm.conf" file allows passing environment variables to PHP scripts. Only variables listed in "docker-compose.yml" are passed, so it's safe.

* PHP scripts access MySQL service using a separate network, while Nginx communicates with PHP over Unix socket. Database can be easily moved and replicated to separate servers in a cluster with minor configuration changes.

A single-instance PHP site is quite productive if done right, it may handle traffic at a million daily users scale, and is very cost-effective. Just keep in mind that a single-instance architecture has SPOFs, does not provide failover and adds problems for implementing the blue-green deployment.

That's it, now I can deploy simple PHP scripts with Nginx, MySQL, TLS with ACME certificates, HTTP2 and IPv6 using a single command.