Category Archives: Internet und Intranet

Internet und Intranet

Deltachat – First dos, first don’ts

Deltachat is a decentral instant messenger (vulgo, Whatsapp alternative) that uses SMTP and IMAP as its transport medium. I already tried it several years ago but dismissed it as too experimental back then.

At the 37th Chaos Communication Congress I personally got in touch with the Deltachat people and their wholesome anarchist attitude, marveled at their insanely flawless live onboarding process during their talk in front of about 100 listeners, and been closely following the project since.

Here’s a few initial learnings from half a year of using the messenger.

First things first, E-Mail and SMTP are NOT cursed

E-Mail is one of the oldest distributed systems, with redundancy and queueing built-in since eternity, and will always have to deal with reachability issues. If you can’t reach someone, if someone can’t reach you, if either of you receive error messages, get in touch with your postmistresses and -masters. Everyone who is more than 50% serious about running a mail server will be interested in resolving any issues for their users. I certainly am.

If you operate the mail server yourself, get educated on best practices starting from DNS, PTR, EHLO, STARTTLS and SPF up to DKIM. Should your opinions on how to run things inexplicably differ from best practice, acquaint yourself with the idea of being wrong just this one time.

Attachments

Since the early 2000’s, the commonly accepted limits for message size hardly went up at all. We used to tell our users “e-mail is not a file transfer protocol”, but at this point, the small message size limits have honestly become a bit awful. Google Mail accepts a whopping 150 MB, T-Online 50 MB, and even my own mail exchangers are stuck in the past with 32 MB only.

Some messengers just accept any mindless 4K video upload, this one doesn’t. Attachments on Deltachat are encrypted end-to-end, no service in the middle is going to store them for you, and the messages themselves will be even bigger after encryption and transport-encoding.

When in doubt, upload somewhere and share the link.

If you operate the server yourself, feel free to raise the limit for your own users as far as you like but be aware that they may always encounter issues when interacting with users on other servers and domains.

Groups

Groups in Deltachat are a very unusual kind of beast, unlike every other group I’ve seen before.

They are fully decentral and really just resemble a group of users who automatically send e-mails to all others, with control messages that automatically notify everyone of people who are new to the group and have left the group. There is no administration at all; everyone can add and remove anyone, for everyone.

Currently, Deltachat groups are perfect for fully cooperative groups (friends and family), where this administration style even has its benefits because there’s no need to ask person X to do task Y. In hostile environments where the group may be subverted by trolls, they are not the ideal solution.

Don’t change mail accounts (My worst mistake.)

If the random Chatmail address you started out with doesn’t satisfy your vanity anymore, don’t change the mail account in your profile, but start over with a new profile. Direct contacts will need to be informed about your change in address anyway, and you can invite your new profile to groups using your old profile.

If you change mail addresses nevertheless, your PGP key will always carry the mail address you initially created it on, breaking Autocrypt key exchange. Also, your old mail address will forever linger as a “secondary” address in the profile and all sorts of hard-to-understand confusion will set in if it starts showing up from elsewhere in the future.

Don’t change mail accounts. Really.

Add a secondary device

Add a secondary device, e.g. your desktop system. It’s the most casual way of having a backup of your Deltchat profile.

Get your friends on board

It’s just 2½ steps:

  • Install the app.
  • Create a chatmail account (happens semi-automatically in current versions of the app).
  • Have them scan your QR code.

Try it right here.

Post header image: “Sundarbans web” by European Space Agency, CC-BY-NC-SA

Internet und Intranet

Deltachat Push on any IMAP server

Preface

Deltachat is a decentral instant messenger (vulgo, Whatsapp alternative) that uses SMTP and IMAP as its transport medium. I already tried it several years ago but dismissed it as too experimental back then.

At the 37th Chaos Communication Congress I personally got in touch with the Deltachat people and their wholesome anarchist attitude, marveled at their insanely flawless live onboarding process during their talk in front of about 100 listeners, and been closely following the project since.

In late 2023/early 2024, the project introduced the Chatmail concept that enables anyone to host Postfix-Dovecot based Deltachat IMAP toasters for easy onboarding with real-time push notifications.

(Nota bene, I had already registered a domain but then decided against running a Chatmail instance because I doubt I have enough spare spoons to run an anonymous operation and/or deal with potential requests from authorities.)

Architecture

The moment I heard of push for Chatmail, I knew I wanted to figure out how to make it work on my own domain. Since no specification for Chatmail push seemed to be published, I figured out things from the following files in the Chatmail repository:

  • default.sieve – Not as helpful as expected, but providing an important-ish detail for the later implementation.
  • push_notification.lua – Hints towards the existence of a metadata service. Never heard of it so far.
  • dovecot.conf.j2 – First appearance of the metadata service in Dovecot config. TIL. Enabled METADATA and XDELTAPUSH on my own Dovecot and figured out that the client stores a notify-token in IMAP metadata.
  • notifier.py – Simply posts the notify-token to the Delta notification service.

So essentially, the push system works like this:

  1. Client connects to IMAP and saves its notify-token to the metadata service.
  2. Mail arrives.
  3. push_notification.lua, loaded as the push driver into Dovecot, talks back to the metadata server, which in turn uses notifier.py for sending.
  4. Client wakes up and connects to IMAP.
  5. Push message appears on screen only if there are unseen messages on the IMAP server. Meaning, if another client is connected as well (and, I believe, running visibly in the foreground), the mail on the server will already be marked seen and no message is displayed.

My alternative implementation needs to replace steps 1-3. Let’s do it.

Independent Push implementation

Executing things, figuring out the push token from IMAP metadata, calling a URL, from Sieve, requires a lot of configuration, so I quickly came back to running an additional IMAP IDLE session from somewhere else, as IdlePush did back in 2009. Instead of adapting the old Perl code or rewriting the IMAP IDLE dance from scratch, I looked for pre-existing building blocks to put together:

  • getmail6 (prepackaged on Debian) is a powerful alternative to fetchmail that can IDLE on an IMAP server and retrieve new messages while leaving them unseen (the way IdlePush did with Mail::IMAPClient‘s $imap->Peek(1);). The retrieved messages can be fed into what I call a custom mail delivery agent.
  • A small Python script serves as the custom MDA.
  • systemd, the old horse, can reliably run getmail as a service.

Files

getmail configuration

getmail demands that its configuration files be saved in ~/.config/getmail, so this is where this configuration goes.

The notify token can be found at the top of the client’s logfile.

# ~/.config/getmail/deltachat-example.com
[retriever]
type=SimpleIMAPSSLRetriever
server=imap.example.com
username=example@example.com
password=xxx
mailboxes=("INBOX",)

[destination]
type=MDA_external
path=~/bin/deltachat-push.py
arguments=("notify-token",)
ignore_stderr=true

[options]
verbose=1
read_all=false
delete=false

Custom mail delivery agent

The custom MDA is already referenced in the config above. Ignoring messages with the Auto-Submitted header was adapted from default.sieve in Chatmail.

The script talks to the notification server at most every 10 seconds per notify-token.

#!/usr/bin/env python3
import sys, requests, re, time
from select import select
from email.feedparser import BytesFeedParser
from pathlib import Path

notify_url='https://notifications.delta.chat/notify'

# See if a message is waiting on stdin - https://stackoverflow.com/a/3763257
if select([sys.stdin, ], [], [], 0.0)[0]:
    try:
        mailparser = BytesFeedParser()
        mailparser.feed(sys.stdin.buffer.read())
        message = mailparser.close()
    except Exception as e:
        print(f"While reading message: {e}", file=sys.stderr)
        sys.exit(255)

    # Skip if message contains header:
    if message.get('Auto-Submitted') and re.match('auto-(replied|generated)', message.get('Auto-Submitted')):
        print('Skipping: Auto-Submitted', file=sys.stderr)
        sys.exit(0)

# Issue notifications to the specified notify-token(s)
for token in sys.argv[1:]:
    print(f"Token: {token}", file=sys.stderr)

    ratelimit_file = f"{Path.home()}/.cache/deltachat-ratelimit-{token}"
    now = int(time.time())
    try:
        ratelimit_time = int(Path(ratelimit_file).stat().st_mtime)
    except:
        ratelimit_time = 0

    if now - ratelimit_time < 10:
        print('ratelimited', file=sys.stderr)
        continue

    try:
        r = requests.post(notify_url, token)
    except Exception as e:
        print(f"While talking to {notify_url}: {e}", file=sys.stderr)
        continue

    print(f"Notification request submitted, HTTP {r.status_code} {r.reason}", file=sys.stderr)
    Path(ratelimit_file).touch()

sys.exit(0)

systemd user unit

I use an instantiated unit that specifies the getmail configuration file and the mailbox (INBOX) on which to IDLE.

# ~/.config/systemd/user/deltachat-push@.service
[Unit]
Description=Push emitter for delta chat

[Service]
ExecStart=getmail -r %i --idle INBOX
Restart=always
RestartSec=10

[Install]
WantedBy=default.target

Instantiation

systemctl --user enable --now deltachat-push@deltachat-example.com.service

Compatibility tested

  • Vanilla Dovecot
  • multipop.t-online.de (yes)
  • imap.gmail.com (indeed)

Troubleshooting

journalctl --user-unit deltachat-push@\*.service -f

~/bin/deltachat-push <notify-token>

Limitations

  • First execution of getmail downloads all mail, so I had to start newly configured getmail configurations without a push token by commenting out the arguments option.
  • May be considered abuse of the Chatmail infrastructure? I definitely use it sensibly, on an IMAP mailbox that is dedicated to Deltchat only. Someone is paying actual time and effort for running that notification relay.
  • Impact of plans for encrypting the notify-token unclear.
Internet und Intranet, UNIX & Linux

IPv6 Privacy Stable Addressing Roundup

“Okay, let’s see whether we can reach your Macbook externally via IPv6. What’s the address?”
Sure, let’s have a look.

$ ifconfig
...
 inet6 2a03:2260:a:b:8aa:22bf:7190:ef36 prefixlen 64 autoconf secured
 inet6 2a03:2260:a:b:b962:5127:c7ec:d2df prefixlen 64 autoconf temporary
...

Everybody knows that one of these is a random IP address according to RFC 4941: Privacy Extensions for Stateless Address Autoconfiguration in IPv6 that changes once in a while so external observers (e.g. accessed web servers) can’t keep track of my hardware’s ethernet MAC address. This is the one we do NOT want if we want to access the Macbook from the internet. We want the stable one, the one that has the MAC address encoded within, the one with the ff:fe in the middle, as we all learned in IPv6 101.
It turns out, all of my devices that configure themselves via SLAAC, namely a Macbook, an iPhone, an iPad, a Linux laptop and a Windows 10 workstation, don’t have ff:fe addresses. Damn, I have SAGE status at he.net, I must figure out what’s going on here!
After a bit of research totally from scratch with the most adventurous search terms, it turns out that these ff:fe, or more professionally, EUI-64 addresses, have become a lot less common than 90% of IPv6 how-to documents and privacy sceptics want us to believe. On most platforms, they have been replaced by Cryptographically Generated Addresses (CGAs), as described in RFC 3972. The RFC is a close relative to RFC 3971, which describes a Secure Neighbor Discovery Protocol (SeND). Together, they describe a cryptographically secure, PKI-based method of IPv6 address generation. However, as of this writing, only a PKI-less stub implementation of RFC 3972 seems to have become commonplace.
Those CGAs, or as some platforms seem to call them, Privacy Stable Addresses, are generated once during the first startup of the system. The address itself, or the seed used to randomize it, may be (and usually is) persistently stored on the system, so the system will come up every time with this same IPv6 address instead of one following the well-known ff:fe pattern.
To stick with the excerpt from my macOS ifconfig output above, the address marked temporary is a Privacy Extension address (RFC 4941), while the one marked secure is the CGA (RFC 3972).
It’s surprisingly hard to come up with discussions on the web where those two types aren’t constantly confused, used synonymously, or treated like ones that both need to be exterminated, no matter the cost. This mailing list thread actually is one of the most useful resources on them.
This blog post is a decent analysis on the behaviour on macOS, although it’s advised to ignore the comments.
This one about the situation on Windows suffers from a bit of confusion, but is where I found a few helpful Windows commands.
The nicest resource about the situation on Linux is this german Ubuntuwiki entry, which, given a bit of creativity, may provide a few hints also to non-german speakers.
So, how to configure this?

  • macOS
    • The related sysctl is net.inet6.send.opmode.
    • Default is 1 (on).
    • Note how this is the only one that refers to SeND in its name.
  • Windows
    • netsh interface ipv6 set global randomizeidentifiers=enabled store=persistent
    • netsh interface ipv6 set global randomizeidentifiers=disabled store=persistent
    • Default seems to be enabled.
    • Use store=active and marvel at how Windows instantly(!) replaces the address.
  • Linux
    • It’s complicated.
    • NetworkManager defaults to using addr-gen-mode=stable-privacy in the [ipv6] section of /etc/NetworkManager/system-connections/<Connection>.
    • The kernel itself generates a CGA if addrgenmode for the interface is set to none and /proc/sys/net/ipv6/conf/<interface>/stable_secret gets written to.
    • NetworkManager and/or systemd-networkd take care of this. I have no actual idea.
    • In manual configuration, CGA can be configured by using ip link set addrgenmode none dev <interface> and writing the stable_secret in a pre-up action. (See the Ubuntu page linked above for an example.)
  • FreeBSD
    • FreeBSD has no support for CGAs, other than a user-space implementation through the package “send”, which I had no success configuring.

So far, I haven’t been able to tell where macOS, Windows and NetworkManager persistently store their seeds for CGA generation. But the next time someone goes looking for an ff:fe address, I’ll know why it can’t be found.