Internet und Intranet

Deltachat Push on any IMAP server

Preface

Deltachat is a decentral instant messenger (vulgo, Whatsapp alternative) that uses SMTP and IMAP as its transport medium. I already tried it several years ago but dismissed it as too experimental back then.

At the 37th Chaos Communication Congress I personally got in touch with the Deltachat people and their wholesome anarchist attitude, marveled at their insanely flawless live onboarding process during their talk in front of about 100 listeners, and been closely following the project since.

In late 2023/early 2024, the project introduced the Chatmail concept that enables anyone to host Postfix-Dovecot based Deltachat IMAP toasters for easy onboarding with real-time push notifications.

(Nota bene, I had already registered a domain but then decided against running a Chatmail instance because I doubt I have enough spare spoons to run an anonymous operation and/or deal with potential requests from authorities.)

Architecture

The moment I heard of push for Chatmail, I knew I wanted to figure out how to make it work on my own domain. Since no specification for Chatmail push seemed to be published, I figured out things from the following files in the Chatmail repository:

  • default.sieve – Not as helpful as expected, but providing an important-ish detail for the later implementation.
  • push_notification.lua – Hints towards the existence of a metadata service. Never heard of it so far.
  • dovecot.conf.j2 – First appearance of the metadata service in Dovecot config. TIL. Enabled METADATA and XDELTAPUSH on my own Dovecot and figured out that the client stores a notify-token in IMAP metadata.
  • notifier.py – Simply posts the notify-token to the Delta notification service.

So essentially, the push system works like this:

  1. Client connects to IMAP and saves its notify-token to the metadata service.
  2. Mail arrives.
  3. push_notification.lua, loaded as the push driver into Dovecot, talks back to the metadata server, which in turn uses notifier.py for sending.
  4. Client wakes up and connects to IMAP.
  5. Push message appears on screen only if there are unseen messages on the IMAP server. Meaning, if another client is connected as well (and, I believe, running visibly in the foreground), the mail on the server will already be marked seen and no message is displayed.

My alternative implementation needs to replace steps 1-3. Let’s do it.

Independent Push implementation

Executing things, figuring out the push token from IMAP metadata, calling a URL, from Sieve, requires a lot of configuration, so I quickly came back to running an additional IMAP IDLE session from somewhere else, as IdlePush did back in 2009. Instead of adapting the old Perl code or rewriting the IMAP IDLE dance from scratch, I looked for pre-existing building blocks to put together:

  • getmail6 (prepackaged on Debian) is a powerful alternative to fetchmail that can IDLE on an IMAP server and retrieve new messages while leaving them unseen (the way IdlePush did with Mail::IMAPClient‘s $imap->Peek(1);). The retrieved messages can be fed into what I call a custom mail delivery agent.
  • A small Python script serves as the custom MDA.
  • systemd, the old horse, can reliably run getmail as a service.

Files

getmail configuration

getmail demands that its configuration files be saved in ~/.config/getmail, so this is where this configuration goes.

The notify token can be found at the top of the client’s logfile.

# ~/.config/getmail/deltachat-example.com
[retriever]
type=SimpleIMAPSSLRetriever
server=imap.example.com
username=example@example.com
password=xxx
mailboxes=("INBOX",)

[destination]
type=MDA_external
path=~/bin/deltachat-push.py
arguments=("notify-token",)
ignore_stderr=true

[options]
verbose=1
read_all=false
delete=false

Custom mail delivery agent

The custom MDA is already referenced in the config above. Ignoring messages with the Auto-Submitted header was adapted from default.sieve in Chatmail.

The script talks to the notification server at most every 10 seconds per notify-token.

#!/usr/bin/env python3
import sys, requests, re, time
from select import select
from email.feedparser import BytesFeedParser
from pathlib import Path

notify_url='https://notifications.delta.chat/notify'

# See if a message is waiting on stdin - https://stackoverflow.com/a/3763257
if select([sys.stdin, ], [], [], 0.0)[0]:
    try:
        mailparser = BytesFeedParser()
        mailparser.feed(sys.stdin.buffer.read())
        message = mailparser.close()
    except Exception as e:
        print(f"While reading message: {e}", file=sys.stderr)
        sys.exit(255)

    # Skip if message contains header:
    if message.get('Auto-Submitted') and re.match('auto-(replied|generated)', message.get('Auto-Submitted')):
        print('Skipping: Auto-Submitted', file=sys.stderr)
        sys.exit(0)

# Issue notifications to the specified notify-token(s)
for token in sys.argv[1:]:
    print(f"Token: {token}", file=sys.stderr)

    ratelimit_file = f"{Path.home()}/.cache/deltachat-ratelimit-{token}"
    now = int(time.time())
    try:
        ratelimit_time = int(Path(ratelimit_file).stat().st_mtime)
    except:
        ratelimit_time = 0

    if now - ratelimit_time < 10:
        print('ratelimited', file=sys.stderr)
        continue

    try:
        r = requests.post(notify_url, token)
    except Exception as e:
        print(f"While talking to {notify_url}: {e}", file=sys.stderr)
        continue

    print(f"Notification request submitted, HTTP {r.status_code} {r.reason}", file=sys.stderr)
    Path(ratelimit_file).touch()

sys.exit(0)

systemd user unit

I use an instantiated unit that specifies the getmail configuration file and the mailbox (INBOX) on which to IDLE.

# ~/.config/systemd/user/deltachat-push@.service
[Unit]
Description=Push emitter for delta chat

[Service]
ExecStart=getmail -r %i --idle INBOX
Restart=always
RestartSec=10

[Install]
WantedBy=default.target

Instantiation

systemctl --user enable --now deltachat-push@deltachat-example.com.service

Compatibility tested

  • Vanilla Dovecot
  • multipop.t-online.de (yes)
  • imap.gmail.com (indeed)

Troubleshooting

journalctl --user-unit deltachat-push@\*.service -f

~/bin/deltachat-push <notify-token>

Limitations

  • First execution of getmail downloads all mail, so I had to start newly configured getmail configurations without a push token by commenting out the arguments option.
  • May be considered abuse of the Chatmail infrastructure? I definitely use it sensibly, on an IMAP mailbox that is dedicated to Deltchat only. Someone is paying actual time and effort for running that notification relay.
  • Impact of plans for encrypting the notify-token unclear.