SMTP Ports, Authentication Mechanisms & Protocol Extensions

SMTP Protocol

This post rounds out the SMTP deep dive series with the practical details you need to configure mail servers and clients: which port to use, how authentication works, what extensions are available, and what pitfalls can trip you up.


SMTP Ports

SMTP uses three main TCP ports, each with a distinct purpose. Choosing the wrong port is one of the most common configuration errors.

Port Use Encryption Typical Scenario
25 Relay (MTA ↔ MTA) None (plain) Server-to-server transfer. Often blocked by residential ISPs to prevent spam.
587 Submission (MUA → MSA) STARTTLS (opportunistic) Standard for client submission. Recommended for all email sending.
465 Submission (MUA → MSA) SSL/TLS (implicit) Legacy port. Deprecated by IETF but still widely supported by providers.

Which Port Should You Use?

  • For sending from an email client (Outlook, Thunderbird, Apple Mail): use port 587 with STARTTLS. This is the modern standard and the port most providers support.

  • For server-to-server relay: use port 25. This is the only port universally accepted by receiving MTAs. However, many residential ISPs and cloud providers block outbound port 25 to prevent spam — you may need to use a smarthost or a specialised email sending service.

  • For legacy systems: port 465 (implicit SSL/TLS) is still widely supported but is not standard. Use it only if your provider requires it or if your client does not support STARTTLS.

Key takeaway: Do not use port 25 for client submission. Do not use port 587 for server-to-server relay. Mixing them is a common configuration error that leads to mysterious delivery failures.


SMTP AUTH

Modern SMTP servers require authentication before accepting messages for relay. This prevents open relays — servers that accept mail from anyone — which were a major vector for spam in the early 2000s.

SASL Mechanisms

SMTP AUTH uses the SASL (Simple Authentication and Security Layer) framework. The EHLO response advertises which mechanisms the server supports:

EHLO client.example.tinkmail.me
250-AUTH LOGIN PLAIN XOAUTH2
Mechanism How It Works Security Notes
LOGIN Username and password sent in two separate base64-encoded challenges. Base64 is not encryption — anyone sniffing the connection can decode it. Requires TLS.
PLAIN Credentials sent as a single base64-encoded string (\0user\0pass). Same as LOGIN — requires TLS. Most widely supported.
CRAM-MD5 The server sends a challenge string; the client responds with an MD5 hash of the challenge combined with the password. Password is not sent in plain text, but the hash can be replayed. Weaker than TLS-based methods.
XOAUTH2 OAuth 2.0 token-based authentication. Used by Gmail, Outlook, and other large providers. More secure (short-lived tokens) and convenient for users (no app password).
GSSAPI Kerberos-based single sign-on. Common in enterprise Microsoft environments. No password exchange at all — uses tickets.

Security Best Practices

  1. Always use TLS-encrypted connections (ports 587 or 465) when authenticating. Sending credentials over port 25 exposes them in plain text.
  2. Prefer XOAUTH2 or GSSAPI over password-based mechanisms where available.
  3. Restrict AUTH mechanisms on your server — disable PLAIN and LOGIN if your clients support stronger alternatives.
  4. Use app-specific passwords for third-party clients that do not support OAuth 2.0.

SMTP Extensions

SMTP has been extended through the EHLO mechanism — servers advertise their capabilities in response to EHLO, and the client uses only the extensions it needs.

Extension Capability Impact
STARTTLS Upgrades a plain-text connection to an encrypted TLS session on the same port. Prevents eavesdropping and credential sniffing.
PIPELINING Allows multiple commands to be sent without waiting for individual responses. Reduces round-trips, improves throughput — especially on high-latency links.
SIZE The receiving server advertises its maximum message size during EHLO. Prevents wasted DATA transfers for oversized messages.
8BITMIME Enables sending of 8-bit characters without encoding to 7-bit. Supports international characters, emoji, and non-ASCII content natively.
DSN Delivery Status Notification — the sending server can request bounce/delay receipts. Covered in detail in the Email Delivery Journey.
AUTH Indicates which SASL authentication mechanisms the server supports. Required for client submission.
CHUNKING (RFC 3030) — allows sending the message in binary chunks with BDAT instead of DATA. More efficient for large messages and avoids the need for dot-stuffing.
BINARYMIME (RFC 3030) — enables transmission of binary MIME content without base64 encoding. Redises overhead for attachments. Typically used with CHUNKING.
VRFY Asks the server to verify whether an address exists. Often disabled for privacy reasons (prevents address enumeration).
ETRN Allows a server to request queued messages from another server (dial-in clients). Used in scenarios with intermittent connectivity.

Common Pitfalls

Open Relays

An SMTP server configured as an open relay accepts messages from any sender for any recipient, regardless of authentication. This is extremely dangerous:

  • Spammers discover open relays and use them to send millions of messages anonymously.
  • The relay’s IP address gets blacklisted, affecting legitimate email from the same server.
  • Modern MTA software disables open relays by default, but misconfigured custom setups remain a risk.

How to avoid it: Always require authentication for outgoing mail, and verify that your server does not accept mail for non-local domains from unauthenticated clients.

Greylisting

Some receiving servers use greylisting as an anti-spam technique: they temporarily reject messages from unknown senders with a 450 temporary failure code. A legitimate SMTP server will retry the delivery after a delay — spam bots often do not.

How to handle it: Ensure your MTA is configured to retry with appropriate backoff (typically 5–30 minutes). Greylisting delays are usually resolved within 30 minutes.

EHLO/HELO Mismatch

The domain presented in the EHLO greeting should ideally match the PTR record (reverse DNS) of the sending IP address. A mismatch between the EHLO hostname, the PTR record, and the domain in the MAIL FROM address can reduce deliverability or trigger spam filters.

How to avoid it:

  1. Ensure every sending IP has a PTR record that resolves to a meaningful hostname (e.g., mail.sender.tinkmail.me).
  2. Configure your MTA to use that same hostname in its EHLO greeting.
  3. Ensure the EHLO hostname resolves forward (A/AAAA record) as well, creating a valid FCrDNS (Forward-Confirmed reverse DNS) chain.

Overly Restrictive Rate Limiting

Aggressive rate limiting can cause legitimate bulk mail to be delayed or rejected. If you run a receiving MTA, configure rate limits with reasonable thresholds and return clear 4xx codes so senders understand they should retry.

Misconfigured Reverse DNS

A missing or mismatched PTR record is one of the most common reasons for rejected mail. Many receiving MTAs perform a reverse DNS lookup on the connecting IP and compare the result to the EHLO hostname. If they do not match, the message may be flagged as spam.

For a deeper look, see our dedicated guide on Reverse DNS (PTR) Records.


Summary

Topic Key Points
Ports 25 for relay, 587 for submission, 465 for legacy. Never use 25 for client submission.
AUTH Requires TLS. Prefer XOAUTH2 (OAuth) or strong password mechanisms.
Extensions STARTTLS, PIPELINING, SIZE, 8BITMIME, and DSN are the most important.
Pitfalls Avoid open relays, handle greylisting with proper retry logic, and ensure EHLO matches PTR.

This concludes the SMTP protocol deep dive series. Start with SMTP Protocol in Practice for the fundamentals, then Email Delivery Journey, Delivery Status Notifications, SMTP Bounces, and explore SMTP Response Codes as needed.