Tag Archives: postfix

Deltachat – First dos, first don’ts

Deltachat is a decentral instant messenger (vulgo, Whatsapp alternative) that uses SMTP and IMAP as its transport medium. I already tried it several years ago but dismissed it as too experimental back then.

At the 37th Chaos Communication Congress I personally got in touch with the Deltachat people and their wholesome anarchist attitude, marveled at their insanely flawless live onboarding process during their talk in front of about 100 listeners, and been closely following the project since.

Here’s a few initial learnings from half a year of using the messenger.


First things first, E-Mail and SMTP are NOT cursed

E-Mail is one of the oldest distributed systems, with redundancy and queueing built-in since eternity, and will always have to deal with reachability issues. If you can’t reach someone, if someone can’t reach you, if either of you receive error messages, get in touch with your postmistresses and -masters. Everyone who is more than 50% serious about running a mail server will be interested in resolving any issues for their users. I certainly am.

If you operate the mail server yourself, get educated on best practices starting from DNS, PTR, EHLO, STARTTLS and SPF up to DKIM. Should your opinions on how to run things inexplicably differ from best practice, acquaint yourself with the idea of being wrong just this one time.


Attachments

Since the early 2000’s, the commonly accepted limits for message size hardly went up at all. We used to tell our users “e-mail is not a file transfer protocol”, but at this point, the small message size limits have honestly become a bit awful. Google Mail accepts a whopping 150 MB, T-Online 50 MB, and even my own mail exchangers are stuck in the past with 32 MB only.

Some messengers just accept any mindless 4K video upload, this one doesn’t. Attachments on Deltachat are encrypted end-to-end, no service in the middle is going to store them for you, and the messages themselves will be even bigger after encryption and transport-encoding.

When in doubt, upload somewhere and share the link.

If you operate the server yourself, feel free to raise the limit for your own users as far as you like but be aware that they may always encounter issues when interacting with users on other servers and domains.


Groups

Groups in Deltachat are a very unusual kind of beast, unlike every other group I’ve seen before.

They are fully decentral and really just resemble a group of users who automatically send e-mails to all others, with control messages that automatically notify everyone of people who are new to the group and have left the group. There is no administration at all; everyone can add and remove anyone, for everyone.

Currently, Deltachat groups are perfect for fully cooperative groups (friends and family), where this administration style even has its benefits because there’s no need to ask person X to do task Y. In hostile environments where the group may be subverted by trolls, they are not the ideal solution.


Don’t change mail accounts (My worst mistake.)

If the random Chatmail address you started out with doesn’t satisfy your vanity anymore, don’t change the mail account in your profile, but start over with a new profile. Direct contacts will need to be informed about your change in address anyway, and you can invite your new profile to groups using your old profile.

If you change mail addresses nevertheless, your PGP key will always carry the mail address you initially created it on, breaking Autocrypt key exchange. Also, your old mail address will forever linger as a “secondary” address in the profile and all sorts of hard-to-understand confusion will set in if it starts showing up from elsewhere in the future.

Don’t change mail accounts. Really.


Add a secondary device

Add a secondary device, e.g. your desktop system. It’s the most casual way of having a backup of your Deltchat profile.


Get your friends on board

It’s just 2½ steps:

  • Install the app.
  • Create a chatmail account (happens semi-automatically in current versions of the app).
  • Have them scan your QR code.

Try it right here.


Post header image: “Sundarbans web” by European Space Agency, CC-BY-NC-SA

Deltachat Push on any IMAP server

Preface

Deltachat is a decentral instant messenger (vulgo, Whatsapp alternative) that uses SMTP and IMAP as its transport medium. I already tried it several years ago but dismissed it as too experimental back then.

At the 37th Chaos Communication Congress I personally got in touch with the Deltachat people and their wholesome anarchist attitude, marveled at their insanely flawless live onboarding process during their talk in front of about 100 listeners, and been closely following the project since.

In late 2023/early 2024, the project introduced the Chatmail concept that enables anyone to host Postfix-Dovecot based Deltachat IMAP toasters for easy onboarding with real-time push notifications.

(Nota bene, I had already registered a domain but then decided against running a Chatmail instance because I doubt I have enough spare spoons to run an anonymous operation and/or deal with potential requests from authorities.)

Architecture

The moment I heard of push for Chatmail, I knew I wanted to figure out how to make it work on my own domain. Since no specification for Chatmail push seemed to be published, I figured out things from the following files in the Chatmail repository:

  • default.sieve – Not as helpful as expected, but providing an important-ish detail for the later implementation.
  • push_notification.lua – Hints towards the existence of a metadata service. Never heard of it so far.
  • dovecot.conf.j2 – First appearance of the metadata service in Dovecot config. TIL. Enabled METADATA and XDELTAPUSH on my own Dovecot and figured out that the client stores a notify-token in IMAP metadata.
  • notifier.py – Simply posts the notify-token to the Delta notification service.

So essentially, the push system works like this:

  1. Client connects to IMAP and saves its notify-token to the metadata service.
  2. Mail arrives.
  3. push_notification.lua, loaded as the push driver into Dovecot, talks back to the metadata server, which in turn uses notifier.py for sending.
  4. Client wakes up and connects to IMAP.
  5. Push message appears on screen only if there are unseen messages on the IMAP server. Meaning, if another client is connected as well (and, I believe, running visibly in the foreground), the mail on the server will already be marked seen and no message is displayed.

My alternative implementation needs to replace steps 1-3. Let’s do it.

Independent Push implementation

Executing things, figuring out the push token from IMAP metadata, calling a URL, from Sieve, requires a lot of configuration, so I quickly came back to running an additional IMAP IDLE session from somewhere else, as IdlePush did back in 2009. Instead of adapting the old Perl code or rewriting the IMAP IDLE dance from scratch, I looked for pre-existing building blocks to put together:

  • getmail6 (prepackaged on Debian) is a powerful alternative to fetchmail that can IDLE on an IMAP server and retrieve new messages while leaving them unseen (the way IdlePush did with Mail::IMAPClient‘s $imap->Peek(1);). The retrieved messages can be fed into what I call a custom mail delivery agent.
  • A small Python script serves as the custom MDA.
  • systemd, the old horse, can reliably run getmail as a service.

Files

getmail configuration

getmail demands that its configuration files be saved in ~/.config/getmail, so this is where this configuration goes.

The notify token can be found at the top of the client’s logfile.

# ~/.config/getmail/deltachat-example.com
[retriever]
type=SimpleIMAPSSLRetriever
server=imap.example.com
username=example@example.com
password=xxx
mailboxes=("INBOX",)

[destination]
type=MDA_external
path=~/bin/deltachat-push.py
arguments=("notify-token",)
ignore_stderr=true

[options]
verbose=1
read_all=false
delete=false

Custom mail delivery agent

The custom MDA is already referenced in the config above. Ignoring messages with the Auto-Submitted header was adapted from default.sieve in Chatmail.

The script talks to the notification server at most every 10 seconds per notify-token.

#!/usr/bin/env python3
import sys, requests, re, time
from select import select
from email.feedparser import BytesFeedParser
from pathlib import Path

notify_url='https://notifications.delta.chat/notify'

# See if a message is waiting on stdin - https://stackoverflow.com/a/3763257
if select([sys.stdin, ], [], [], 0.0)[0]:
    try:
        mailparser = BytesFeedParser()
        mailparser.feed(sys.stdin.buffer.read())
        message = mailparser.close()
    except Exception as e:
        print(f"While reading message: {e}", file=sys.stderr)
        sys.exit(255)

    # Skip if message contains header:
    if message.get('Auto-Submitted') and re.match('auto-(replied|generated)', message.get('Auto-Submitted')):
        print('Skipping: Auto-Submitted', file=sys.stderr)
        sys.exit(0)

# Issue notifications to the specified notify-token(s)
for token in sys.argv[1:]:
    print(f"Token: {token}", file=sys.stderr)

    ratelimit_file = f"{Path.home()}/.cache/deltachat-ratelimit-{token}"
    now = int(time.time())
    try:
        ratelimit_time = int(Path(ratelimit_file).stat().st_mtime)
    except:
        ratelimit_time = 0

    if now - ratelimit_time < 10:
        print('ratelimited', file=sys.stderr)
        continue

    try:
        r = requests.post(notify_url, token)
    except Exception as e:
        print(f"While talking to {notify_url}: {e}", file=sys.stderr)
        continue

    print(f"Notification request submitted, HTTP {r.status_code} {r.reason}", file=sys.stderr)
    Path(ratelimit_file).touch()

sys.exit(0)

systemd user unit

I use an instantiated unit that specifies the getmail configuration file and the mailbox (INBOX) on which to IDLE.

# ~/.config/systemd/user/deltachat-push@.service
[Unit]
Description=Push emitter for delta chat

[Service]
ExecStart=getmail -r %i --idle INBOX
Restart=always
RestartSec=10

[Install]
WantedBy=default.target

Instantiation

systemctl --user enable --now deltachat-push@deltachat-example.com.service

Compatibility tested

  • Vanilla Dovecot
  • multipop.t-online.de (yes)
  • imap.gmail.com (indeed)

Troubleshooting

journalctl --user-unit deltachat-push@\*.service -f
~/bin/deltachat-push <notify-token>

Limitations

  • First execution of getmail downloads all mail, so I had to start newly configured getmail configurations without a push token by commenting out the arguments option.
  • May be considered abuse of the Chatmail infrastructure? I definitely use it sensibly, on an IMAP mailbox that is dedicated to Deltchat only. Someone is paying actual time and effort for running that notification relay.
  • Impact of plans for encrypting the notify-token unclear.