Healthchecks.io Hosting Setup, 2022 Edition

By / February 17, 2022

Here’s the summary of the hardware and the software that powers Healthchecks.io.

Hardware

Since 2017, Healthchecks.io runs on dedicated servers at Hetzner. The current lineup is:

  • HAProxy servers: 4x AX41-NVMe servers (Ryzen 3600, 6 cores)
  • Web servers: 3x AX41-NVMe servers (Ryzen 3600, 6 cores)
  • PostgreSQL servers: 2x AX101 servers (Ryzen 5950X, 16 cores)

All servers are located in the Falkenstein data center park, scattered across the FSN-DCx data centers so they are not all behind the same core switch. The monthly Hetzner bill is €484.

Software

  • Ubuntu 20.04 on all machines.
  • Systemd manages services that need to run continuously (haproxy, nginx, postgresql, etc.)
  • Wireguard for private networking between the servers. Tiered topology: HAProxy servers cannot talk to PostgreSQL servers.
  • Netdata agent for monitoring the machines and the services running on them. Connected to Netdata Cloud for easy overview of all servers.
  • HAProxy 2.2 for terminating TLS connections, and load balancing between app servers. Enables easy rolling updates of application servers.
  • PostgreSQL 13, streaming replication from primary to standby. No automatic failover: I can trigger failover with a single command, but the decision is manual.

On app servers:

  • uWSGI runs the Healthchecks Python application (web frontend, management API).
  • hchk, a small application written in Go, handles ping API (hc-ping.com) and inbound email.
  • NGINX handles rate limiting, static file serving, and reverse proxying to uWSGI and hchk.

SaaS Tools

  • AWS S3 for storing encrypted database backups.
  • Braintree for accepting payments and managing subscriptions.
  • Cloudflare for hosting DNS records.
  • Elastic Email for sending transactional email.
  • Fastmail for sending and receiving support email.
  • GitHub for version control and tracking issues, and GitHub Actions for running tests on every commit.
  • Hardypress for blog.healthchecks.io (static WordPress blog as-a-service).
  • HetrixTools for uptime monitoring.
  • IcoMoon for authoring icon fonts.
  • pgDash for monitoring PostgreSQL servers. Here’s a blog post about setting it up.
  • PingPong for powering status.healthchecks.io (service status, incidents, planned downtimes, performance metrics).
  • SSLMate for provisioning certificates from command-line.
  • Syften for getting notifications when Healthchecks is mentioned on HN, Twitter, Reddit and elsewhere.
  • Twilio for sending SMS, WhatsApp and phone call notifications.

Cron Jobs

Healthchecks.io, the cron job monitoring service, uses cron jobs itself for the following periodic tasks:

  • Once a day, make a full database backup, encrypt it with gpg, and upload it to AWS S3.
  • Once a day, send “Your account is inactive and is about to be deleted” notifications to inactive users.
  • Once a day, send “Your subscription will renew on …” for annual subscriptions that are due in 1 month.

Bonus – Development and Deployment Setup

  • My main dev machine is a desktop PC with a single 27″ 1440p display.
  • Ubuntu 20.04, GNOME Shell.
  • Sublime Text for editing source code. A combination of meld, Sublime Merge and command-line git for working with git.
  • Yubikeys for signing git commits and logging into servers.
  • Fabric scripts for deploying code and running maintenance tasks on servers.
  • sops for storing secrets.
  • A dedicated laptop inside a dedicated backpack, for dealing with emergencies while away from the main PC.

Comments, questions, ideas? Let me know via email or on Twitter!