Since a little more than two months ago, Healthchecks.io has been sending transactional email (~300’000 emails per month) through its own SMTP server. Here are my notes on setting it up.

The Before

Before going self-hosted, Healthchecks sent email using 3rd-party SMTP relays: AWS SES and later Elastic Email.

The reason for switching from AWS to Elastic Email was GDPR compliance: at the time United States did not have an EU adequacy decision, but Canada (the registration country of Elastic Email Inc.) did.

The primary reason I kept looking for alternatives to Elastic Email was also GDPR compliance: a country with EU adequacy decision is good, but being based in the EU is even better. Another reason was their poor communication during service outages: some outages were not acknowledged on their status page, there were no timely updates via support chat or otherwise, and there were no post-mortems published after outages. To their credit, Elastic Email did fix the outages reasonably quickly, and I was overall happy with the service in terms of functionality and pricing.

The EU-based SMTP Relay Options

There are few EU-based SMTP relay services. None of the big names (AWS SES, Sendgrid, Mailgun, Mailchimp, Postmark) are EU-based. I tested a few options:

• EmailLabs: OK in terms of functionality and pricing. Judging by the mix of Polish and English in the user interface and documentation seemed geared primarily to the Polish market.
• SMTPeter: OK in terms of functionality and pricing. It was probably just bad timing, but had a major outage while I was testing it. Small shop.
• Brevo (formerly Sendinblue): the most prominent EU SMTP relay service. Has open and click tracking enabled by default, and refused to turn it off before seeing live production traffic, so a non-starter for me.

None of the options seemed like an upgrade over what I already had, and I kept circling back to the idea of self-hosting. The common wisdom is that self-hosting email means endless deliverability problems, but maybe-maybe?

The Self-Hosting Options

In May 2023, I spent several weeks researching and trialing self-hosted SMTP servers: mox, Postal, Haraka, Zone MTA, OpenSMTPD, and maddy. My brain was getting fried from jumping between documentation sites, trying to make sense of the feature sets, and the pros and cons of each project. One thing that helped immensely was reading Email explained from first principles – it filled many gaps in my knowledge of email delivery.

Maddy

After experimenting with and strongly considering OpenSMTPD, I ultimately picked maddy. I iterated on a test configuration until I got it to do the required things:

• Accept email on port 465 from authenticated users.
• Rewrite its envelope sender from “@healthchecks.io” to “@mail.healthchecks.io” (required for routing bounce messages back to the maddy server).
• Sign it using DKIM protocol.
• Put outgoing messages in a queue, attempt to deliver them, and retry with exponential backoff.
• Deliver messages to remote MTAs from a single, specific IP.
• When delivery fails, send a webhook notification to a designated webhook handler. For permanent failures, the handler can take appropriate action – unsubscribe a specific user from email reports, or mark a specific email integration as disabled.
• Listen for incoming email on port 25.
• When a remote MTA sends a DSN (delivery status notification, “bounce message”), deliver it to the same webhook.
• Use an automatically provisioned LetsEncrypt certificate for TLS encryption on port 465 and port 25.

I wrote the provisioning scripts for deploying Maddy and its configuration to a server. I added and updated the required DNS entries for SPF and DKIM. I implemented, tested, and deployed the webhook handler that would receive bounce notifications from maddy.

IP Warm-Up

I spent several weeks gradually switching outgoing email traffic from Elastic Email to the self-hosted maddy server. IP warm-up serves two purposes:

• Slowly builds up the reputation of the sending IP address. Switching the entire sending volume to a new IP address all at once risks getting blocked by the receiving servers.
• It lets me test email delivery in the production environment and fix any potential problems with fewer negative consequences.

The Failover IP Oopsie

One issue I discovered during the IP warm-up phase was that the brand-new mail server (a Hetzner AX41 dedicated server) experienced minute-long network hiccups a couple of times per day. The cause could be a faulty NIC, a faulty switch, or a noisy neighbor, and the easiest fix is ordering another server and hoping for better luck. In anticipation of such a scenario, I had ordered a failover IP so I could keep using the already warmed-up IP with the new server.

I set up a new server, switched the failover IP to it, and after a few days of testing, no more network hiccups! So I went ahead and canceled the original machine. Then, a few days later, around 2 AM local time, my monitoring notifications went off: email delivery was broken. I had assumed that “failover IP” is more or less what other providers call “floating IP.” Dazed and confused in front of a blue screen in the middle of the night, I realized my misunderstanding and mistake: the failover IP is owned by a specific server. Canceling the server also cancels the failover IP with all its sender reputation.

To fix the immediate problem, I temporarily switched the web servers back to using Elastic Email as the SMTP relay. I asked Hetzner support if there was any way I could get the released IP back. Minutes later, I got a reply stating in perfect German calmness that my request would need to be handled by a different department during business hours. The next morning, Hetzner added the lost failover IP back to my account. Phew!

Local Relay-Only MTAs for Reliability

The internet-facing SMTP server (mail.healthchecks.io) runs on a single machine. Each app server also runs a local maddy instance which accepts outgoing email messages from local clients only, and hands them off to mail.healthchecks.io. If mail.healthchecks.io is unavailable (for example, during server restart), the local maddy instances queue the messages and retry them later.

Summary, Pros, and Cons

The self-hosted maddy server has been handling all Healthchecks.io transactional email for over two months. I am keeping an eye on bounce notifications, the outbound email queue size, and blocklists. So far, there have been no significant deliverability issues–fingers crossed!

Cons of going self-hosted:

• Self-hosted SMTP server is another service to maintain. It uses up the limited time and mental bandwidth I have.
• The inevitable deliverability problems will be my problems.
• In the case of maddy, no pre-built graphical management and monitoring dashboard.

And pros:

• Complete control of subprocessors with access to customer data (just Hetzner in my case).
• Complete control over server configuration.
• Fixed direct costs (as long as a single server can keep up with the sending volume).
• I learned a bunch of new stuff!

Thanks for reading,
Pēteris

Healthchecks recently gained a new feature: check auto-provisioning. When you send a ping request to a slug URL, and a check with the specified slug does not exist, Healthchecks can now automatically create the missing check. This feature requires opt-in: to use it, add a ?create=1 query parameter to the ping URL.

Here’s check auto-provisioning in action (the -I parameter tells curl to send HTTP HEAD requests so that we can see HTTP response status codes easily):

$ curl -I https://hc-ping.com/fixme-ping-key/does-not-exist
HTTP/2 404
[...]

$ curl -I https://hc-ping.com/fixme-ping-key/does-not-exist?create=1
HTTP/2 201
[...]

$ curl -I https://hc-ping.com/fixme-ping-key/does-not-exist?create=1
HTTP/2 200 
[...]

• The first request returns HTTP 404 (“Not Found”) because a check with a slug does-not-exist does, in fact, not yet exist.
• The second request has a “?create=1” added to the URL to enable auto-provisioning. The server creates a new check and returns HTTP 201 (“Created”).
• The third request is the same as the second, but a matching check now exists. The server accepts the ping and returns HTTP 200 (“OK”).

When is this useful? Whenever you are working with a dynamic infrastructure, and want your monitoring clients to be able to register with Healthchecks.io automatically. If you distribute the Ping Key to monitoring clients, each client can pick its own slug (for example, derived from the server’s hostname), construct a ping URL (https://hc-ping.com/<ping-key>/<slug-chosen-by-client>?create=1), and Healthchecks.io will auto-create a new check on the first ping.

Auto-Provisioned Checks Use Default Configuration

With the current auto-provisioning implementation, clients can create new checks on the fly, but they cannot yet specify the period, the grace time, the enabled integrations, or any other parameters. The new checks will be created with default parameters (period = 1 day, grace time = 1 hour, all integrations enabled). If you need to change any parameters, you will need to do this either manually from the web dashboard, or from a script that calls Management API.

Auto-Provisioning and Account Limits

Each account has a specific limit of how many checks it is allowed to create: 20 checks for free accounts; 100 or 1000 checks for paid accounts. To reduce friction and the risk of silent failures, the auto-provisioning functionality is allowed to temporarily exceed the account’s check limit up to two times. Meaning, if your account is already maxed out, auto-provisioning will still be able to create new checks until you hit two times the limit. If your account goes over the limit, you will start to see warnings in the dashboard and email:

As soon as you get the number of checks in your account below the limit (either by upgrading to higher limits, or by removing unneeded checks), the warning will go away. If you do not resolve the warning for more than a month, you will start seeing an “Account marked for deletion” notice in the dashboard. After another month of inaction, the account will be deleted.

Slugs and Names Are Now Separate

In the initial slug implementation check slugs were tied to check names. Changing a check’s name also updated its slug. With the introduction of auto-provisioning, check names and slugs are now decoupled. You can hand-pick a custom slug for each check. You can also rename a check but keep its existing slug.

The “Name and Tags” dialog has gained a new, editable “Slug” field:

Similarly, the Create a Check and Update an Existing Check API calls now support a new slug field.

Happy monitoring,
–Pēteris

A Healthchecks user sent me a code snippet for sending HTTP pings from Arduino. This prompted me to do some Arduino experimenting on my own. I ordered Arduino Nano 33 IoT board:

I picked this board because I wanted an easy entry into Arduino development. As a first-party Arduino hardware, it should be easy to get it working with Arduino IDE. It has an on-board WIFI chip, so I would not need to hook up additional WiFi or Ethernet hardware.

The Nano 33 IoT has a micro USB port. After connecting to my PC running Ubuntu, Arduino’s power LED lit up, and on the computer side a /dev/ttyACM0 device appeared. Arduino IDE detected the connected board, but my initial attempt to upload a sketch failed. This turned out to be a permissions issue. After I added my OS user to the dialout group, I could upload a “Hello World” sketch to the board:

Sending a Raw HTTP Request

Arduino Nano 33 IoT has an on-board WiFi module. To use it, Arduino provides the WiFiNINA library. The library comes with example code snippets. One of the examples shows how to connect to a WiFi network and make an HTTP request. I adapted it to make an HTTPS request to hc-ping.com:

#include <WiFiNINA.h>
#include "arduino_secrets.h"

char ssid[] = SECRET_SSID;
char pass[] = SECRET_PASS; 
int status = WL_IDLE_STATUS;             
WiFiSSLClient client;

void setup() {
  pinMode(LED_BUILTIN, OUTPUT);
  Serial.begin(9600);
  while (!Serial);

  Serial.print("Connecting ...");
  WiFi.begin(ssid, pass);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.print("\nConnected, IP address: ");
  Serial.println(WiFi.localIP());  
}

void ping() {
  Serial.println("Pinging hc-ping.com...");
  if (client.connect("hc-ping.com", 443)) {
    Serial.println("Connected to server.");
    client.println("GET /da840100-3f58-405e-a5ee-e7e6e4303e82 HTTP/1.0");
    client.println("Host: hc-ping.com");
    client.println("Connection: close");
    client.println();
    Serial.println("Request sent.");
  }

  while (client.connected()) {
    while (client.available()) {
      char c = client.read();
      Serial.write(c);
    }
  }
  Serial.println("\nClosing connection.");
  client.stop();
}

void loop() {
  ping();

  // Blink LED for 10 seconds:
  Serial.print("Waiting 10s: ");
  for (int i=0; i<10; i++) {
    Serial.print(".");
    digitalWrite(LED_BUILTIN, HIGH);  
    delay(500);                      
    digitalWrite(LED_BUILTIN, LOW);  
    delay(500);  
  } 
  Serial.println();
}

After uploading this sketch to Arduino, here’s the output on serial console:

Connecting ...
Connected, IP address: 192.168.1.77
Pinging hc-ping.com...
Connected to server.
Request sent.
HTTP/1.1 200 OK
server: nginx
date: Thu, 30 Mar 2023 12:33:25 GMT
content-type: text/plain; charset=utf-8
content-length: 2
access-control-allow-origin: *
ping-body-limit: 100000
connection: close

OK
Closing connection.
Waiting 10s: ..........
Pinging hc-ping.com...
Connected to server.
Request sent.
HTTP/1.1 200 OK
server: nginx
date: Thu, 30 Mar 2023 12:33:41 GMT
content-type: text/plain; charset=utf-8
content-length: 2
access-control-allow-origin: *
ping-body-limit: 100000
connection: close

OK
Closing connection.
Waiting 10s: .......
[...]

Quite impressively, this works over HTTPS out of the box – the WiFiNINA library and the chip takes care of performing TLS handshake and verifying the certificates. All I had to do was specify port 443, and the rest was handled automagically.

ArduinoHttpClient

After getting the minimal example working, I found the ArduinoHttpClient library. It offers a higher-level interface for making GET and POST requests, and for parsing server responses. It works with several different network libraries, including WifiNINA.

#include <ArduinoHttpClient.h>
#include <WiFiNINA.h>
#include "arduino_secrets.h"

char ssid[] = SECRET_SSID;
char pass[] = SECRET_PASS; 
int status = WL_IDLE_STATUS;             
WiFiSSLClient wifi;
char host[] = "hc-ping.com";
char uuid[] = UUID;
HttpClient client = HttpClient(wifi, host, 443);

void setup() {
  pinMode(LED_BUILTIN, OUTPUT);
  Serial.begin(9600);
  while (!Serial);

  Serial.print("Connecting ...");
  WiFi.begin(ssid, pass);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.print("\nConnected, IP address: ");
  Serial.println(WiFi.localIP());  
}

void loop() {
  client.get("/" + String(uuid));
  Serial.print("Status code: ");
  Serial.println(client.responseStatusCode());
  Serial.print("Response: ");
  Serial.println(client.responseBody());

  // Blink LED for 10 seconds:
  Serial.print("Waiting 10s: ");
  for (int i=0; i<10; i++) {
    Serial.print(".");
    digitalWrite(LED_BUILTIN, HIGH);  
    delay(500);                      
    digitalWrite(LED_BUILTIN, LOW);  
    delay(500);  
  } 
  Serial.println();
}

Output in the serial console:

Connecting ...
Connected, IP address: 192.168.1.77
Status code: 200
Response: OK
Waiting 10s: ..........
Status code: 200
Response: OK
Waiting 10s: ..........
[...]

ESP8266

After having good results with Arduino Nano 33 IoT, I wanted to try the same on an ESP8266 board I had lying around:

This board from AliExpress has a few goodies in addition to the ESP8266 chip: a relay, and multiple powering options: 220V AC, 7-12V DC, 5V DC. It has a USB port, but this port can be used for supplying power only, there is no USB-UART interface onboard. There are clearly labeled GND, 5V, RX, TX pins that I can hook a USB-UART converter (also from AliExpress) to:

The yellow jumper connects GPIO 0 to the ground, this puts ESP8266 in programming mode. At this point I can plug the USB-UART converter in the PC and check for signs of life using esptool:

$ apt-get install esptool
$ esptool chip_id
esptool.py v2.8
Found 2 serial ports
Serial port /dev/ttyUSB0
Connecting...
Detecting chip type... ESP8266
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: a8:48:fa:ff:15:45
Enabling default SPI flash mode...
Chip ID: 0x00ff1545
Hard resetting via RTS pin...

Arduino IDE does not support ESP8266 chips out of the box, but there is esp8266/Arduino project which adds support for different flavors of ESP boards.

The esp8266/Arduino project also comes with a WiFi library, which provides an interface to the WiFi functionality on the chip. For simple use cases, the esp8266wifi library is a drop-in replacement for the WiFiNINA library:

#include <ArduinoHttpClient.h>
#include <ESP8266WiFi.h>
#include "arduino_secrets.h"

char ssid[] = SECRET_SSID;
char pass[] = SECRET_PASS; 
WiFiClient wifi;
char host[] = "hc-ping.com";
char uuid[] = UUID;
HttpClient client = HttpClient(wifi, host, 80);

void setup() {  
  pinMode(LED_BUILTIN, OUTPUT);

  Serial.begin(115200);
  while (!Serial);  
  Serial.println();

  WiFi.begin(ssid, pass);

  Serial.print("Connecting ...");
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.print("\nConnected, IP address: ");
  Serial.println(WiFi.localIP());
}

void loop() {
  client.get("/" + String(uuid));
  Serial.print("Status code: ");
  Serial.println(client.responseStatusCode());
  Serial.print("Response: ");
  Serial.println(client.responseBody());

  // Blink LED for 10 seconds:
  Serial.print("Waiting 10s: ");
  for (int i=0; i<10; i++) {
    Serial.print(".");
    digitalWrite(LED_BUILTIN, HIGH);  
    delay(500);                      
    digitalWrite(LED_BUILTIN, LOW);  
    delay(500);  
  } 
  Serial.println();  
}

Although the esp8266wifi library does support TLS, the documentation also mentions significant CPU and memory requirements. To keep things simple and quick, I went with port 80 and unencrypted HTTP for this experiment.

I uploaded the sketch, removed the yellow jumper, reset the board, and got this output on the serial console:

Connecting ..........
Connected, IP address: 192.168.1.78
Status code: 200
Response: OK
Waiting 10s: ..........
Status code: 200
Response: OK
Waiting 10s: ..........
[...]

Success!

In summary, my first steps in Arduino development left me with positive impressions. The network libraries provide an easy to use, high-level interface for working with network hardware. They have uniform interfaces, so can be used in sketches interchangeably, with minimal code changes. After the initial hump of getting a board recognized by Arduino IDE, and getting the first sketch to upload and run, the development went smoothly. To be fair, the “development” in my case was mostly copying and tweaking code samples. But it was still good!

Happy tinkering,
–Pēteris

Let’s say you have a script that works when run in an interactive session, but does not produce expected results when run from cron. What could be the problem? Some potential culprits include:

• The cron daemon does not see the new job. You added the job definition to a file that the cron daemon does not read.
• The schedule is wrong. The job will run eventually, but not at your intended schedule.
• cron’s PATH environment variable is different from the one in your interactive shell, so the script could not find a binary.
• cron uses /bin/sh, which may be /usr/bin/dash instead of /usr/bin/bash, and your script relies on a bash-only feature.
• The script uses relative filesystem paths and is intended to be run from a specific directory. When run by cron, the filesystem paths are all wrong.

Or it could be something else. How to troubleshoot this then, and where to start? Instead of trying fixes at random, I prefer to start by looking at logs:

• Look at the system logs to see if cron ran the script at all.
• Inspect the script’s stdout and stderr output for error messages and other clues.

System Logs

To check system logs on modern Linux systems using systemd, use the “journalctl” command:

journalctl -t CRON --since "today"

The “-t CRON” argument tells journalctl to show log entries with the “CRON” tag only. The “–since” parameter accepts timestamps and time durations in various formats. A couple of examples:

journalctl -t CRON --since "30 minutes ago"
journalctl -t CRON --since "2023-01-25"

If cron did run the job, you will find log entries that look like this:

jan 25 15:27:01 foo CRON[511824]: (user) CMD (/home/user/make-backup.sh)
jan 25 15:27:01 foo CRON[511823]: (CRON) info (No MTA installed, discarding output)

The first line shows the command line cron tried to run. The second line shows that the command generated some output, and cron discarded it. If the command completes without producing any output, or if your system has an MTA such as Postfix or sSMTP installed, the second line will be absent.

Checking Script’s Output

If a cron job produces output, cron will attempt to email the command’s output to the email address specified in the MAILTO= line in crontab. For this to work, the system needs to have a configured message transfer agent (MTA). See How to Send Email From Cron Jobs for instructions on how to configure sSMTP as a MTA.

Logging to System Logs

If you want to avoid the hassle of setting up a working MTA, a simpler option is to pipe the script’s output to the system log:

0 4 * * * /home/user/make-backup.sh 2>&1 | logger -t backups

Let’s analyze the above line:

• The cron expression 0 4 * * * means “run this job ar 4:00 every day”.
• /home/user/make-backup.sh is the script we want cron to run.
• 2>&1 redirects sderr to stdout.
• |, the pipe character, pipes output (both stdout and stderr, thanks to the redirection) from the previous command into the following command.
• logger -t backups reads data from stdin and writes it to systemd logs, tagged as “backups”.

After the cron job has run, you can inspect its output with the journalctl utility:

journalctl -t backups --since "today"

To view live logs in the follow mode:

journalctl -t backups -f

Logging to Files

An even simpler option for an ad-hoc debugging session is to write the script’s output to a file. In this example, the log file gets overwritten each time the job runs, so it will only contain the output of the most recent run:

0 4 * * * /home/user/make-backup.sh > /tmp/backups.log 2>&1

• > (right angle bracket) redirects output to a file.
• 2>&1 redirects sderr to stdout.

Note: bash has a shorthand for redirecting stdout and stderr to a file: &> /path/to/file. dash does not support it, so it is safer to use the longer form: > /path/to/file 2>&1.

In this example, logs from each run get appended to a single file, and each line is prefixed with a timestamp:

0 4 * * * /home/user/make-backup.sh 2>&1 | ts >> /tmp/backups.log

• 2>&1 redirects sderr to stdout.
• The combined output is piped to ts, which prefixes each line with a timestamp.
• Finally, >> redirects output to a file in append mode.

Note: The “ts” utility may not be installed by default. For Debian-based systems, it is available in the “moreutils” package.

Logging to Healthchecks.io

And yet another option is to forward the logs to Healthchecks.io and view them in the web browser. With this option, you also get external monitoring and notifications when your job does not complete on schedule or exits with non-zero exit status:

0 4 * * * m=$(/home/user/make-backup.sh 2>&1); curl --data-raw "$m" https://hc-ping.com/<uuid>/$?

• The m=$(...) syntax runs the enclosed command and assigns its output to the variable $m
• The semicolon separates two consecutive commands (without piping one’s output to the other)
• curl’s --data-raw "$m" parameter sends the contents of $m in HTTP request body
• $? is the exit status of the previous command. We tack it onto the URL so Healthchecks knows if the command finished successfully (exit status 0) or unsuccessfully (exit status > 0).

If the script produces a lot of output, you may get an error:

/usr/bin/curl: Argument list too long

In that case, one workaround is to save the output to a temporary file, then tell curl to read request body from the file:

0 4 * * * /home/user/make-backup.sh > /tmp/backups.log 2>&1; curl --data-binary @/tmp/backups.log https://hc-ping.com/<uuid>/$?

As the command starts to get a little unwieldly, you may also consider replacing curl with runitor, a command runner with Healthchecks.io integration:

0 4 * * * runitor -uuid <uuid> -- /home/user/make-backup.sh 

That is all for now,
Happy scripting,
Pēteris

When a cron job does not run on time, Healthchecks can notify you using various methods. One of the supported methods is Signal messages. Signal is an end-to-end encrypted messenger app run by a non-profit Signal Foundation. Signal’s mobile client, desktop client, and server are free and open-source software (with some exceptions–read on!).

No Incoming Webhooks

Unlike most other instant messaging services, Signal does not provide incoming webhooks for posting messages in chats. If you want to send messages on the Signal network, you must run a full client, and follow all the same cryptographic protocols that normal end-user clients follow. This is inconvenient for the integration developer but makes sense: the main feature of Signal is strong cryptography and as little as possible sharing of information with the Signal servers. The servers pass around messages, and help with peer discovery, but (as far as I know) cannot send their own messages on the user’s behalf. Official incoming webhooks would conflict with the overall architecture of the system.

signal-cli

signal-cli is a third-party open-source Signal client. It uses the same signal client libary that the official clients use but offers a programmatic interface for sending and receiving messages. signal-cli supports command-line, DBUS, and JSON-RPC interfaces.

Signal’s official position on the signal-cli client seems to be–they do not support it, but they also have not explicitly banned it. When I asked Signal Support about their stance regarding signal-cli (and also about advice regarding rate-limit issues discussed below), I got just this short response back:

Due to our limitations as a non-profit organization, we can only provide support for the product we provide. Signal-cli is not provided or maintained by us, therefore we cannot provide any support for it.

Using signal-cli in Healthchecks

I coded the initial Signal integration in January 2021. To send messages, it was running a signal-cli send -m 'text goes here' command for every message. Each send took a minimum of one second, as every signal-cli invocation was initializing JVM, and initializing network connections, just to do one small send operation. A more efficient approach was to run signal-cli in daemon mode and talk to it via DBUS or JSON-RPC.

Also in January 2021, I upgraded the integration to talk to signal-cli over DBUS. This took some tinkering to figure out the DBUS interface configuration and to get python code to talk to it. But it worked, and message delivery was now much quicker.

In December 2021, signal-cli added the JSON-RPC interface, and I switched the Healthchecks integration to it. Again, it took a fair bit of tinkering and support from the signal-cli author until I figured out how it all hangs together, how to read and write messages over a UNIX socket, and how to interpret them. There were two important improvements over the previous DBUS code:

• Simpler operations: I did not need the DBUS service with its associated configuration files anymore.
• The Healthchecks project did not need the “dbus-python” dependency anymore.

Rate limiting and CAPTCHAs

Around April 2022 I started to notice that some send operations were failing with an error message asking to solve a CAPTCHA challenge. These errors were infrequent at first and seemed to only affect the very first messages to new recipients. I added code to email me the CAPTCHA challenges, and I added a crude command-line utility to submit the CAPTCHA solutions. As the CAPTCHA challenges came in, I manually solved and submitted them. Signal was using Google reCAPTCHA, and I got plenty of opportunities to demonstrate my intelligence by expertly clicking on fire hydrants, crosswalks, and traffic lights. Sometimes at odd hours, sometimes roadside over a mobile hotspot.

As the frequency of CAPTCHAs gradually increased, I tried to make solving them less annoying:

• I figured out that being logged in gmail.com helps the CAPTCHA solving a lot. Usually just a single click, no fire hydrants.
• I set up my computer to automatically put the CAPTCHA solution in the clipboard.
• I made a web form for submitting CAPTCHA solutions. No need to fire up the terminal, just click a link in the email, and paste the solution.

Now solving a CAPTCHA challenge took just a few clicks, but the end-user experience was still not great. For some users, Signal notifications would not work until I showed up and solved yet another CAPTCHA. I did some spelunking in the signal-server code base. There is a class listing various rate-limiters and their parameters. For any rate limiter, I could trace back where and how it was used. But I still could not pinpoint the piece of code that triggers the specific rate-limit errors I was seeing. Signal-Server has an “abusive-message-filter” module, which is private code, perhaps the logic lives there.

It seemed only the initial messages to new recipients were triggering rate-limit errors. After a single message got through, the following messages would work with no issues. So my next idea was to change the Signal integration onboarding flow:

• After the user has entered the phone number of the Signal recipient, ask them to send a test message
• If the test message generates a rate-limit error, ask the user to initiate a conversation with us from their side, then try again:

My working theory is that users initiating the conversation with Healthchecks will look less abusive to Signal’s abusive message filter, and will help avoid hitting rate limits. But, if the theory fails and we still get rate-limit errors, at least the users will not create dysfunctional integrations (the “Save Integration” button becomes available only after successfully sending a test message).

In Summary

In summary, Healthchecks uses signal-cli to send Signal messages. It talks to signal-cli over JSON-RPC. To avoid rate limits, it asks the user to send the first message from their end. Building and maintaining the Signal integration has taken more effort than any other integration. But that is fine and, aside from the manual CAPTCHA solving, time well spent. I’m glad Healthchecks supports it, and I’m happy to see that the Signal integration is popular among Healthchecks.io users.

Happy monitoring and messaging,
Pēteris