Long time no post! Here are some updates of what’s been going on.
Database Backups
I switched from logical database backups using pg_dump to physical backups using pg_basebackup.
Sometime in July I was starting to get monitoring notifications about delays in the alert sending process. I was getting them once per day, at the same time as when the database backup runs. It was not hard to connect the dots. I investigated the problem and found a query that was getting progressively slower if executed while the backup runs. The query was using a frequently updated index which was not being cleaned up during pg_dump. As the database size was steadily growing, pg_dump runs were taking longer and longer, the index had time to become more and more bloated towards the end of each backup run, and the affected query was becoming more and more inefficient.
I experimented with taking the backups on a standby instead of the primary, but this does not solve anything if the database uses the “hot_standby_feedback” option – the primary still holds off from index maintenance while pg_dump runs on a standby.
I then found that physical backups work around the problem, the query does not slow down while a physical backup is running. I updated the backup scripts to take a snapshot of the database data files using pg_basebackup, then encrypting and uploading them to S3 as before.
The physical backups are a fair bit larger. I could “win back” some of the size increase by using zstd compression, which I was not using before. Currently a single full database backup is 14GB.
User Growth
There’s been a noticeable jump in the rate of new signups. It started in May, and is continuing. In September alone the number of registered users went from 112K to 143K. That’s 27% growth in a single month!
What changed? I’m … not sure. But it’s intriguing.
Upgrade to Ubuntu 26.04
I upgraded both my development system and the app servers from Ubuntu 24.04 to Ubuntu 26.04. Healthchecks.io is now running on Python 3.14, woo-hoo!
minismtpd
The Healthchecks open source project includes a SMTP listener for receiving pings over email (manage.py smtpd). Originally it was powered by the smtpd module from Python’s standard library. When the standard library module became deprecated, I updated the code to use its suggested replacement, aiosmtpd. The development activity of aiosmtpd is low, the last released version is from 2024. I have been able to get a few smaller fixes in, but any bigger changes take a lot of patience.
I had an idea for a “mini” version of aiosmtpd, with almost everything ripped out except the bare essentials. I experimented with this (deleting code is fun!) and got to a point where the test suite completes in 0.3s. I also had a look at the old smtpd module from the standard library, and it was so much simpler. Perhaps a better base for what I was trying to do.
I took the last available version of smtpd, replaced asyncore and asynchat with socketserver, removed a bunch of functionality, added type hints, packaged it up and got minismtpd. The code is less than 500 lines, it does very little, but that’s OK for the Healthchecks use case.
New Office
I moved from a home office to a garden office.
It is a small space but the big windows on two sides makes it feel like I’m almost sitting right in the garden. I love it so far.
That’s all for now, thanks for reading, and – happy monitoring!
–Pēteris