Forcing SSL/TLS encryption for all mail services on myVesta servers

[isscbta]

Goal
This guide explains how to enforce mandatory SSL/TLS encryption for all mail protocols (SMTP, IMAP, POP3) on MyVesta + Debian servers, and how to correctly link Exim and Dovecot authentication.
It also covers common issues such as “SMTP Error (535) Authentication failed” and “Sender verify failed”.

⸻

Backup everything before start.

1. Firewall configuration
To block plain-text connections, remove the following ports from the firewall:

Code: Select all

25, 110, 143

Only these encrypted ports should remain open:

• 465 – SMTPS (SSL/TLS)
• 587 – SMTP Submission (STARTTLS)
• 993 – IMAPS (SSL/TLS)
• 995 – POP3S (SSL/TLS)

Example nmap output after configuration:

Code: Select all

PORT    STATE    SERVICE
25/tcp  filtered smtp
110/tcp filtered pop3
143/tcp filtered imap
465/tcp open     smtps
587/tcp open     submission
993/tcp open     imaps
995/tcp open     pop3s

⸻

2. Dovecot configuration (IMAP/POP3)
Edit /etc/dovecot/conf.d/10-ssl.conf and set:

Code: Select all

ssl = required

This forces encrypted access for all IMAP and POP3 connections.
Restart Dovecot:

Code: Select all

systemctl restart dovecot

⸻

3. Exim4 configuration (SMTP + authentication)

Open /etc/exim4/exim4.conf.template and verify the following:

• TLS advertisement is enabled:

Code: Select all

tls_advertise_hosts = *

• TLS certificates are defined:

Code: Select all

tls_certificate = /usr/local/vesta/ssl/certificate.crt
tls_privatekey  = /usr/local/vesta/ssl/certificate.key

• Recommended cipher configuration:

Code: Select all

tls_require_ciphers = NORMAL:-VERS-SSL3.0:-VERS-TLS1.0

• The authenticator section must use the dovecot driver, but without the old TLS restriction line (server_condition = ${if def:tls_cipher {yes}{no}}).
Replace both blocks with this:

Code: Select all

begin authenticators

dovecot_plain:
driver = dovecot
public_name = PLAIN
server_socket = /var/run/dovecot/auth-client
server_set_id = $auth1

dovecot_login:
driver = dovecot
public_name = LOGIN
server_socket = /var/run/dovecot/auth-client
server_set_id = $auth1

Remove any existing “server_condition” lines from these sections, as they block AUTH on SMTPS (port 465).

Restart Exim:

Code: Select all

systemctl restart exim4

⸻

4. Permission fix for Exim ↔ Dovecot integration
If you see errors such as:

Code: Select all

535 Incorrect authentication data
or
failed to open /etc/exim4/domains/example.com/passwd for linear search: Permission denied (euid=101 egid=105)

add Exim to the “mail” group so it can read domain passwd files:

Code: Select all

usermod -aG mail Debian-exim
systemctl restart exim4

This step is safe and recommended for MyVesta/Hestia installations.

⸻

5. Roundcube configuration (optional check)
If you are using Roundcube, make sure it connects via SSL/TLS:

In /etc/roundcube/config.inc.php:

Code: Select all

$rcmail_config[‘default_host’] = ‘ssl://mail.example.com’;
$rcmail_config[‘smtp_server’]  = ‘ssl://mail.example.com’;
$rcmail_config[‘smtp_port’]    = 465;
$rcmail_config[‘smtp_user’]    = ‘%u’;
$rcmail_config[‘smtp_pass’]    = ‘%p’;
$rcmail_config[‘smtp_auth_type’] = ‘PLAIN’;

Restart Apache afterwards:

Code: Select all

systemctl restart apache2

⸻

6. Testing

Check external port status:

Code: Select all

nmap -Pn -sT -p 25,110,143,465,587,993,995 mail.example.com

Verify TLS functionality:

Code: Select all

openssl s_client -starttls smtp -crlf -connect mail.example.com:587
openssl s_client -connect mail.example.com:465
openssl s_client -connect mail.example.com:993
openssl s_client -connect mail.example.com:995

If everything is correct, authentication without TLS will be rejected:

Code: Select all

530 Must issue a STARTTLS command first

And authenticated delivery will appear in logs like:

Code: Select all

A=dovecot_login:[email protected] => [email protected] R=dnslookup T=remote_smtp Completed

⸻

7. Summary
✔ All IMAP, POP3, and SMTP connections require SSL/TLS
✔ Plain authentication is disabled
✔ Unencrypted ports (25, 110, 143) are blocked externally
✔ Exim and Dovecot authentication works through /var/run/dovecot/auth-client
✔ Debian-exim is part of the “mail” group and can access passwd files
✔ Compatible with Roundcube and external mail clients

⸻

8. Applying on other servers
To apply this setup elsewhere:

• Set

  Code: Select all

  ssl = required

  in Dovecot
• Remove any

  Code: Select all

  server_condition = ${if def:tls_cipher {yes}{no}}

  lines in Exim authenticators
• Add Exim to the mail group with

  Code: Select all

  usermod -aG mail Debian-exim

• Close ports 25, 110, and 143 in the firewall

Restart all services:

Code: Select all

systemctl restart dovecot
systemctl restart exim4
systemctl restart apache2