Technical overview

Directories of the relay repository

The chatmail relay repository has four main directories.

scripts/

scripts offers two convenience tools for beginners:

  • initenv.sh installs a local virtualenv Python environment and installs necessary dependencies

  • scripts/cmdeploy script enables you to run the cmdeploy command line tool in the local Python virtual environment.

cmdeploy/

The cmdeploy directory contains the Python package and command line tool to setup a chatmail relay remotely via SSH:

  • cmdeploy init creates the chatmail.ini config file locally.

  • cmdeploy run under the hood uses pyinfra to automatically install or upgrade all chatmail components on a relay, according to the local chatmail.ini config.

The deployed system components of a chatmail relay are:

  • Postfix is the Mail Transport Agent (MTA) and accepts messages from, and sends messages to, the wider email MTA network

  • Dovecot is the Mail Delivery Agent (MDA) and stores messages for users until they download them

  • Nginx shows the web page with privacy policy and additional information

  • acmetool manages TLS certificates for Dovecot, Postfix, and Nginx

  • OpenDKIM for signing messages with DKIM and rejecting inbound messages without DKIM

  • mtail for collecting anonymized metrics in case you have monitoring

  • Iroh relay which helps client devices to establish Peer-to-Peer connections

  • TURN to enable relay users to start webRTC calls even if a p2p connection can’t be established

  • and the chatmaild services, explained in the next section:

chatmaild/

chatmaild is a Python package containing several small services which handle authentication, trigger push notifications on new messages, ensure that outbound mails are encrypted, delete inactive users, and some other minor things. chatmaild can also be installed as a stand-alone Python package.

chatmaild implements various systemd-controlled services that integrate with Dovecot and Postfix to achieve instant-onboarding and only relaying OpenPGP end-to-end messages encrypted messages. A short overview of chatmaild services:

  • doveauth implements create-on-login address semantics and is used by Dovecot during IMAP login and by Postfix during SMTP/SUBMISSION login which in turn uses Dovecot SASL to authenticate logins.

  • filtermail prevents unencrypted email from leaving or entering the chatmail service and is integrated into Postfix’s outbound and inbound mail pipelines.

  • chatmail-metadata is contacted by a Dovecot lua script to store user-specific relay-side config. On new messages, it passes the user’s push notification token to notifications.delta.chat so the push notifications on the user’s phone can be triggered by Apple/Google/Huawei.

  • chatmail-expire deletes users if they have not logged in for a longer while. The timeframe can be configured in chatmail.ini.

  • lastlogin is contacted by Dovecot when a user logs in and stores the date of the login.

  • echobot is a small bot for test purposes. It simply echoes back messages from users.

  • metrics collects some metrics and displays them at https://example.org/metrics.

www/

www contains the html, css, and markdown files which make up a chatmail relay’s web page. Edit them before deploying to make your chatmail relay stand out.

Component dependency diagram

         graph LR;
     cmdeploy --- sshd;
     letsencrypt --- |80|acmetool-redirector;
     acmetool-redirector --- |443|nginx-right(["`nginx
     (external)`"]);
     nginx-external --- |465|postfix;
     nginx-external(["`nginx
     (external)`"]) --- |8443|nginx-internal["`nginx
     (internal)`"];
     nginx-internal --- website["`Website
     /var/www/html`"];
     nginx-internal --- newemail.py;
     nginx-internal --- autoconfig.xml;
     certs-nginx[("`TLS certs
     /var/lib/acme`")] --> nginx-internal;
     cron --- chatmail-metrics;
     cron --- acmetool;
     chatmail-metrics --- website;
     acmetool --> certs[("`TLS certs
     /var/lib/acme`")];
     nginx-external --- |993|dovecot;
     autoconfig.xml --- postfix;
     autoconfig.xml --- dovecot;
     postfix --- echobot;
     postfix --- |10080,10081|filtermail;
     postfix --- users["`User data
     home/vmail/mail`"];
     postfix --- |doveauth.socket|doveauth;
     dovecot --- |doveauth.socket|doveauth;
     dovecot --- users;
     dovecot --- |metadata.socket|chatmail-metadata;
     doveauth --- users;
     chatmail-expire-daily --- users;
     chatmail-fsreport-daily --- users;
     chatmail-metadata --- iroh-relay;
     certs-nginx --> postfix;
     certs-nginx --> dovecot;
     style certs fill:#ff6;
     style certs-nginx fill:#ff6;
     style nginx-external fill:#fc9;
     style nginx-right fill:#fc9;

This diagram shows relay components and dependencies/communication paths.

Message between users on the same relay

        graph LR;
    sender --> |465|smtps/smtpd;
    sender --> |587|submission/smtpd;
    smtps/smtpd --> |10080|filtermail;
    submission/smtpd --> |10080|filtermail;
    filtermail --> |10025|smtpd_reinject;
    smtpd_reinject --> cleanup;
    cleanup --> qmgr;
    qmgr --> smtpd_accepts_message;
    qmgr --> |lmtp|dovecot;
    dovecot --> recipient;
    dovecot --> sender's_other_devices;

This diagram shows the path a non-federated message takes.

Operational details of a chatmail relay

Mailbox directory layout

Fresh chatmail addresses have a mailbox directory that contains:

  • a password file with the salted password required for authenticating whether a login may use the address to send/receive messages. If you modify the password file manually, you effectively block the user.

  • enforceE2EEincoming is a default-created file with each address. If present the file indicates that this chatmail address rejects incoming cleartext messages. If absent the address accepts incoming cleartext messages.

  • dovecot*, cur, new and tmp represent IMAP/mailbox state. If the address is only used by one device, the Maildir directories will typically be empty unless the user of that address hasn’t been online for a while.

Active ports

Postfix listens on ports

  • 25 (SMTP)

  • 587 (SUBMISSION) and

  • 465 (SUBMISSIONS)

Dovecot listens on ports

  • 143 (IMAP) and

  • 993 (IMAPS)

Nginx listens on port

  • 8443 (HTTPS-ALT) and

  • 443 (HTTPS) which multiplexes HTTPS, IMAP and SMTP using ALPN to redirect connections to ports 8443, 465 or 993.

acmetool listens on port:

  • 80 (HTTP).

chatmail-turn listens on port

  • 3478 UDP (STUN/TURN), and temporarily opens further UDP ports when users request them. UDP port range is not restricted, any free port may be allocated.

chatmail-core based apps will, however, discover all ports and configurations automatically by reading the autoconfig XML file from the chatmail relay server.

Email domain authentication (DKIM)

Chatmail relays enforce DKIM to authenticate incoming emails. Incoming emails must have a valid DKIM signature with Signing Domain Identifier (SDID, d= parameter in the DKIM-Signature header) equal to the From: header domain. This property is checked by OpenDKIM screen policy script before validating the signatures. This correpsonds to strict DMARC alignment (adkim=s). If there is no valid DKIM signature on the incoming email, the sender receives a “5.7.1 No valid DKIM signature found” error.

Note that chatmail relays

  • do not rely on DMARC and do not consult the sender policy published in DMARC records;

  • do not rely on legacy authentication mechanisms such as iprev and SPF. Any IP address is accepted if the DKIM signature was valid.

Outgoing emails must be sent over authenticated connection with envelope MAIL FROM (return path) corresponding to the login. This is ensured by Postfix which maps login username to MAIL FROM with smtpd_sender_login_maps and rejects incorrectly authenticated emails with reject_sender_login_mismatch policy. From: header must correspond to envelope MAIL FROM, this is ensured by filtermail proxy.

TLS requirements

Postfix is configured to require valid TLS by setting smtp_tls_security_level to verify. If emails don’t arrive at your chatmail relay server, the problem is likely that your relay does not have a valid TLS certificate.

You can test it by resolving MX records of your relay domain and then connecting to MX relays (e.g mx.example.org) with openssl s_client -connect mx.example.org:25 -verify_hostname mx.example.org -verify_return_error -starttls smtp from the host that has open port 25 to verify that certificate is valid.

When providing a TLS certificate to your chatmail relay server, make sure to provide the full certificate chain and not just the last certificate.

If you are running an Exim server and don’t see incoming connections from a chatmail relay server in the logs, make sure smtp_no_mail log item is enabled in the config with log_selector = +smtp_no_mail. By default Exim does not log sessions that are closed before sending the MAIL command. This happens if certificate is not recognized as valid by Postfix, so you might think that connection is not established while actually it is a problem with your TLS certificate.

Architecture of cmdeploy

cmdeploy is a Python program that uses the pyinfra library to deploy chatmail relays, with all the necessary software, configuration, and services. The deployment process performs three primary types of operation:

  1. Installation of software, universal across all deployments.

  2. Configuration of software, with deploy-specific variations.

  3. Activation of services.

The process is implemented through a family of “deployer” objects which all derive from a common Deployer base class, defined in cmdeploy/src/cmdeploy/deployer.py. Each object provides implementation methods for the three stages – install, configure, and activate. The top-level procedure in deploy_chatmail() calls these methods for all the deployer objects, via the Deployment.perform_stages() method, also defined in deployer.py. This first calls all the install methods, then the configure methods, then the activate methods.

The Deployment class also implements support for a CMDEPLOY_STAGES environment variable, which allows limiting the process to specific stages. Note that some deployers are stateful between the stages (this is one reason why they are implemented as objects), and that state will not get propagated between stages when run in separate invocations of cmdeploy. This environment variable is intended for use in future revisions to support building Docker images with software pre-installed, and configuration of containers at run time from environment variables.

The, install() methods for the deployer classes should use ‘self’ as little as possible, preferably not at all. In particular, install() methods should never depend on “config” data, such as the config dictionary in self.config or specific values like self.mail_domain. This ensures that these methods can be used to perform generic installation operations that are applicable across multiple relay deployments, and therefore can be called in the process of building a general-purpose container image.

Operations that start services for systemd-based deployments should only be called from the activate_impl() methods. These methods will not be called in non-systemd container environments.