Building a Complete Mail Server from Scratch

This post ties together the full mail server build documented across the previous nine posts. It covers the component architecture, installation order, dependencies between services, and the reasoning behind each choice. Use it as a map for the entire build or as a reference when troubleshooting interactions between components.

Architecture Overview

The mail server consists of seven components, each handling a specific function:

Supporting services:

Build Order

The installation order matters because later components depend on earlier ones. Build in this sequence:

1. Foundation

Before any mail components, the server needs:

• Base server setup — system packages, time sync, hostname
• SSH hardening — key-based authentication, disabled root login
• Unbound — local DNS resolver for fast lookups
• MySQL — database backend for virtual mailboxes
• Redis — storage backend for Rspamd
• Apache — web server for PostfixAdmin and Roundcube

2. DNS and Certificates

DNS records and TLS certificates must be in place before configuring any mail service:

• MX records pointing to mail.example.com
• A records for mail hostname and aliases (imap, smtp)
• PTR record (reverse DNS) matching the MX hostname
• TLS certificate covering all mail hostnames via SAN (Subject Alternative Names)

Without correct DNS, other servers cannot deliver mail to you. Without valid TLS, clients and servers refuse to connect.

3. Postfix

Postfix is installed first among the mail components because it is the core MTA:

• Accepts incoming mail on port 25
• Provides authenticated submission on port 587
• Queries MySQL for virtual domains, mailboxes, and aliases
• Hands mail to Dovecot via LMTP for delivery

At this stage, Postfix can receive mail but cannot deliver it locally (Dovecot is not yet running) and cannot authenticate users (SASL depends on Dovecot).

4. Dovecot

Dovecot provides three critical services:

• IMAP — client access to mailboxes on port 993
• LMTP — local delivery from Postfix via a Unix socket
• SASL — authentication for Postfix's submission port via a Unix socket

After Dovecot starts, restart Postfix to activate the LMTP and SASL sockets. The server can now receive, deliver, and authenticate mail.

Dovecot also runs Sieve filtering during LMTP delivery. The initial Sieve script files spam into the Junk folder based on headers added by Rspamd.

5. Email Authentication

DKIM, SPF, and DMARC prevent spoofing and improve deliverability:

• OpenDKIM generates a key pair per domain and signs outgoing messages
• SPF DNS records declare which IPs may send mail for the domain
• DMARC ties DKIM and SPF together with a reporting policy

Postfix connects to OpenDKIM via the milter protocol. DKIM signing happens after Postfix processes the message but before it leaves the server.

6. Rspamd

Rspamd processes incoming mail before delivery:

• Bayes classification learns from spam and ham patterns
• DNS blacklists check sender reputation
• Greylisting temporarily rejects unknown senders
• Rate limiting prevents flood attacks
• Content analysis catches phishing and scam patterns

Rspamd connects to Postfix as a milter (port 11332) and stores statistical data in Redis. Rspamd can optionally handle DKIM signing instead of OpenDKIM.

7. ClamAV

ClamAV scans attachments for malware:

• clamd runs as a daemon with the virus signature database loaded in memory
• freshclam updates signatures 12 times per day
• Rspamd connects to clamd via a Unix socket and rejects infected messages

ClamAV depends on Rspamd being configured — it integrates through Rspamd's antivirus module, not directly with Postfix.

8. PostfixAdmin

PostfixAdmin provides web-based management of the MySQL database:

• Add and remove domains
• Create and delete mailboxes with quota settings
• Configure aliases and domain aliases
• Manage admin accounts

PostfixAdmin writes to the same MySQL database that Postfix and Dovecot read from. It requires Apache with PHP-FPM.

9. Roundcube

Roundcube gives users browser-based email access:

• Connects to Dovecot for IMAP
• Connects to Postfix for SMTP
• Provides ManageSieve integration for server-side filters
• Supports PGP encryption via the Enigma plugin

Roundcube is the last component because it depends on every other service working correctly.

Data Flow

Understanding how a message flows through the system helps with troubleshooting:

Incoming Mail (external sender → local mailbox)

1. External server connects to Postfix on port 25
2. Postfix passes the message to Rspamd via milter protocol
3. Rspamd checks spam score, DNS blacklists, greylisting, and rate limits
4. Rspamd passes attachments to ClamAV for virus scanning
5. If clean, Rspamd returns the message to Postfix with headers (X-Spam, X-Spamd-Result)
6. Postfix queries MySQL to resolve the recipient's virtual mailbox
7. Postfix hands the message to Dovecot via LMTP
8. Dovecot runs Sieve filters (files spam to Junk, applies user rules)
9. Dovecot writes the message to the Maildir on disk

Outgoing Mail (authenticated user → external server)

1. User composes in Roundcube (or a desktop client)
2. Client connects to Postfix on port 587 with TLS
3. Postfix delegates authentication to Dovecot via SASL socket
4. Postfix passes the message to Rspamd (for DKIM signing if configured)
5. Postfix passes the message to OpenDKIM via milter (if using OpenDKIM for signing)
6. Postfix delivers the message to the recipient's MX server

Common Cross-Component Issues

LMTP Socket Missing

If Postfix logs delivery temporarily suspended: connect to /var/spool/postfix/private/dovecot-lmtp, Dovecot's LMTP service isn't running or the socket isn't in Postfix's chroot:

ls -la /var/spool/postfix/private/dovecot-lmtp

Restart Dovecot and verify the socket appears.

SASL Authentication Failing

If users can't send mail on port 587, check the auth socket:

ls -la /var/spool/postfix/private/auth

If missing, verify Dovecot's 10-master.conf has the auth service configured with the correct path and permissions.

Rspamd Not Scanning

If mail bypasses spam filtering, verify Postfix's milter configuration:

postconf | grep milter

Ensure smtpd_milters includes inet:127.0.0.1:11332 and Rspamd is listening on that port.

ClamAV Not Connected

If Rspamd logs ClamAV connection errors, verify the socket exists and permissions allow access:

ls -la /var/run/clamav/clamd.sock

Ensure LocalSocketMode 666 is set in clamd.conf or that the Rspamd user has group access to the socket.

DNS Authentication Failures

If DKIM/SPF/DMARC fail on received mail, verify DNS records are published and propagated:

dig +short TXT mail._domainkey.example.com
dig +short TXT example.com | grep spf
dig +short TXT _dmarc.example.com

DKIM key mismatches are common after key rotation — ensure the DNS record matches the current private key.

Verification

After the full build, run through the complete testing checklist to verify every component and their interactions.

Summary

A self-hosted mail server requires careful ordering of component installation:

1. Foundation services (DNS, MySQL, Redis, Apache)
2. Core mail (Postfix, Dovecot)
3. Authentication (DKIM, SPF, DMARC)
4. Filtering (Rspamd, ClamAV)
5. Management (PostfixAdmin, Roundcube)

Each layer depends on the previous one. The result is a fully functional mail server with spam filtering, antivirus, web management, and webmail — entirely under your control.