Mail Server Hardening Checklist: Secure Your SMTP Server

Why Harden Your Mail Server?

An unsecured mail server is a prime target for attackers. It can be exploited as an open relay to send spam, used for credential brute-forcing, or leveraged in phishing campaigns using your domain's reputation. Mail servers also handle sensitive data — every unencrypted connection is an eavesdropping opportunity.

Hardening reduces your attack surface, protects your domain reputation, and ensures compliance with security best practices. This guide covers every layer from OS to DNS.

OS-Level Security

Keep Software Updated

Enable automatic security updates for your OS and mail server software (Postfix, Exim, Dovecot). Most mail server vulnerabilities are patched quickly — running outdated versions is the #1 risk.

Firewall Configuration

Allow only required ports: 25 (inbound MTA), 587 (submission), 465 (SMTPS), 993 (IMAPS), 995 (POP3S). Block all others. Use iptables, nftables, or cloud security groups.

Minimal Installation

Run only mail-related services on the server. Remove unnecessary packages, disable unused daemons, and don't run a web server on the same host unless absolutely necessary.

SSH Hardening

Disable password authentication, use key-based auth only, change the default port, and restrict access to specific IPs with AllowUsers or firewall rules.

SMTP Configuration

Prevent Open Relay

The most critical check: ensure your server does NOT relay mail for unauthenticated users to external domains.

# Postfix main.cf
smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination

Limit Message Size

Set a reasonable maximum message size to prevent abuse:

message_size_limit = 26214400  # 25 MB

HELO/EHLO Restrictions

Reject clients that send invalid HELO hostnames:

smtpd_helo_required = yes
smtpd_helo_restrictions = reject_invalid_helo_hostname, reject_non_fqdn_helo_hostname

Disable VRFY and EXPN

These SMTP commands can be used to enumerate valid email addresses:

disable_vrfy_command = yes

TLS & Encryption

Enforce TLS 1.2+

Disable SSLv3, TLS 1.0, and TLS 1.1:

smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

Use Strong Ciphers

Prefer forward-secrecy ciphers (ECDHE):

smtpd_tls_mandatory_ciphers = high

Auto-Renew Certificates

Use Let's Encrypt with Certbot for free, auto-renewing certificates. Set up a post-renewal hook to reload your mail server.

Consider MTA-STS

Publish an MTA-STS policy to tell senders to always use TLS when delivering to your domain. This prevents TLS downgrade attacks.

Authentication Hardening

Require SASL Authentication

All submission (port 587/465) must require authentication. Never allow unauthenticated sending to external domains.

Rate Limit Auth Attempts

Use Fail2Ban to block IPs after repeated failed login attempts:

# /etc/fail2ban/jail.local
[postfix-sasl]
enabled = true
maxretry = 5
bantime = 3600

Strong Password Policies

Enforce minimum password length (12+ characters), complexity requirements, and consider supporting app-specific passwords for accounts with 2FA.

Anti-Abuse Measures

Connection Rate Limiting

Limit connections per IP to prevent abuse:

smtpd_client_connection_rate_limit = 30
smtpd_client_message_rate_limit = 60

RBL Checks

Reject mail from known spam sources using DNS-based blacklists:

smtpd_recipient_restrictions =
    reject_rbl_client zen.spamhaus.org,
    reject_rbl_client bl.spamcop.net

Content Filtering

Integrate SpamAssassin, Rspamd, or ClamAV for spam and malware scanning. Run the filter as a milter for best performance.

Outbound Rate Limiting

If a user account is compromised, limit the blast radius by restricting outbound message rates per authenticated user.

DNS Record Hardening

DNS records are essential for mail server security. Verify all of these:

MX Records — point to your mail server with correct priorities
PTR Record — reverse DNS must match your mail server hostname
SPF — publish with -all (hard fail) listing only authorized senders
DKIM — sign all outgoing mail with 2048-bit keys
DMARC — set p=reject after monitoring with p=none
CAA Records — restrict certificate issuance to your chosen CA
MTA-STS — enforce TLS for inbound mail delivery

Verify your DNS setup with the DNS Lookup Tool and RBL Blacklist Checker.

Logging & Monitoring

Centralize mail logs with syslog or a log aggregation service
Monitor queue size — sudden spikes indicate compromise or backscatter
Set up alerts for authentication failures (brute force detection)
Track outbound volume per user to detect compromised accounts
Review DMARC aggregate reports weekly for unauthorized sending
Monitor your IP reputation with RBL checks

Complete Checklist

☐ OS and mail server software up to date

☐ Firewall allows only required ports (25, 587, 465, 993)

☐ SSH hardened (key auth only, non-default port)

☐ Open relay test passed — unauthenticated relay blocked

☐ HELO restrictions enforced

☐ VRFY and EXPN commands disabled

☐ TLS 1.2+ enforced, weak ciphers disabled

☐ Valid SSL certificate with auto-renewal

☐ SASL authentication required on submission ports

☐ Fail2Ban configured for SASL failures

☐ Connection and message rate limits set

☐ RBL checks enabled for inbound mail

☐ Spam/malware filtering active (SpamAssassin, ClamAV)

☐ SPF record with -all published

☐ DKIM signing enabled (2048-bit key)

☐ DMARC policy set to p=reject

☐ PTR record matches mail server hostname

☐ CAA records restrict certificate issuance

☐ Log monitoring and alerting configured

☐ Outbound rate limiting per user enabled

FAQ

What is mail server hardening?

Mail server hardening secures an SMTP server by applying OS protections, enforcing TLS, configuring authentication, implementing anti-abuse measures, and publishing DNS records like SPF, DKIM, and DMARC.

Should I use port 25, 465, or 587?

Port 25 is for server-to-server delivery. Port 587 is for authenticated client submission with STARTTLS. Port 465 is for implicit TLS. Block port 25 from end users and require auth on 587/465.

How do I prevent an open relay?

Require authentication for sending to external domains. In Postfix, set smtpd_relay_restrictions to permit only authenticated users and local destinations.