The simplest authentication mechanism is PLAIN. The client simply sends the password unencrypted to Dovecot. All clients support the PLAIN mechanism, but obviously there's the problem that anyone listening on the network can steal the password. For that reason (and some others) other mechanisms were implemented.
Today however many people use SSL/TLS, and there's no problem with sending unencrypted password inside SSL secured connections. So if you're using SSL, you probably don't need to bother worrying about anything else than the PLAIN mechanism.
Another plaintext mechanism is LOGIN. It's typically used only by SMTP servers to let Outlook clients perform SMTP authentication. Note that LOGIN mechanism is not the same as IMAP's LOGIN command. The LOGIN command is internally handled using PLAIN mechanism.
Non-plaintext mechanisms have been designed to be safe to use even without SSL/TLS encryption. Because of how they have been designed, they require access to the plaintext password or their own special hashed version of it. This means that it's impossible to use non-plaintext mechanisms with commonly used DES or MD5 password hashes.
If you want to use more than one non-plaintext mechanism, the passwords must be stored as plaintext so that Dovecot is able to generate the required special hashes for all the different mechanisms. If you want to use only one non-plaintext mechanism, you can store the passwords using the mechanism's own password scheme.
With success/failure password databases (e.g. PAM) it's not possible to use non-plaintext mechanisms at all, because they only support verifying a known plaintext password.
Dovecot supports the following non-plaintext mechanisms:
CRAM-MD5: Protects the password in transit against eavesdroppers. Somewhat good support in clients.
DIGEST-MD5: Somewhat stronger cryptographically than CRAM-MD5, but clients rarely support it.
- SCRAM-SHA-1: Salted Challenge Response Authentication Mechanism (SCRAM) SASL and GSS-API Mechanisms. Intended as DIGEST-MD5 replacement.
- APOP: This is a POP3-specific authentication. Similar to CRAM-MD5, but requires storing password in plaintext.
NTLM: Mechanism created by Microsoft and supported by their clients.
Optionally supported using Samba's winbind.
GSSAPI: Kerberos v5 support.
- RPA: Compuserve RPA authentication mechanism. Similar to DIGEST-MD5, but client support is rare.
- ANONYMOUS: Support for logging in anonymously. This may be useful if you're intending to provide publicly accessible IMAP archive.
- OTP and SKEY: One time password mechanisms.
- EXTERNAL: EXTERNAL SASL mechanism.
By default only PLAIN mechanism is enabled. To use more, edit your /etc/dovecot/conf.d/10-auth.conf and set:
auth_mechanisms = plain login cram-md5