This documentation is for Dovecot v2.x, see wiki1 for v1.x documentation.
Differences between revisions 13 and 14
Revision 13 as of 2006-12-05 22:23:11
Size: 5721
Editor: TimoSirainen
Comment:
Revision 14 as of 2006-12-05 23:24:18
Size: 6155
Editor: TimoSirainen
Comment:
Deletions are marked like this. Additions are marked like this.
Line 39: Line 39:
Although maildir was designed to be locking, Dovecot locks the maildir while doing modifications to it or while looking for new messages in it. This is required because otherwise mails could temporarily be lost, which would cause trouble. Although maildir was designed to be lockless, Dovecot locks the maildir while doing modifications to it or while looking for new messages in it. This is required because otherwise Dovecot might temporarily see mails incorrectly deleted, which would cause trouble. Basically the problem is that if one process modifies the maildir (eg. a rename() to change a message's flag), another process in the middle of listing files at the same time could skip a file. The skipping happens because readdir() system call doesn't guarantee that all the files are returned if the directory is modified between the calls to it. This problem exists with all the commonly used filesystems.

Maildir

This format debuted with the qmail server in the mid-1990s. Each mailbox folder is a directory and each message a file. This improves efficiency because individual emails can be modified, deleted and added without affecting the mailbox or other emails, and makes it safer to use on networked file systems such as NFS.

Filesystem directory Indexes

Linux

The main disadvantage is that searching can be slightly slower, and access to very large mailboxes (thousands of messages) can get slow with filesystems which don't have directory indexes.

Old versions of ext2 and ext3 on Linux don't support directory indexing (to speed up access), but newer versions of ext3 do, although you may have to manually enable it. Make sure that your kernel is configured with CONFIG_EXT3_INDEX=y. If this variable isn't available, you need a new kernel. You can check if the indexing is already enabled with tune2fs:

tune2fs -l /dev/hda3 | grep features

If you see dir_index, you're all set. If dir_index is missing, add it using:

umount /dev/hda3
tune2fs -O dir_index /dev/hda3
e2fsck -fD /dev/hda3
mount /dev/hda3

Directory Structure

Dovecot uses [http://www.inter7.com/courierimap/README.maildirquota.html Maildir++] directory layout for organizing mailbox directories. This means that all the folders are directly inside ~/Maildir directory:

  • ~/Maildir/new, ~/Maildir/cur and ~/Maildir/tmp directories contain the messages. The tmp directory is used during delivery, new messages arrive in new and read shall be moved to cur by the clients.

  • ~/Maildir/.folder/ is a mailbox folder

  • ~/Maildir/.folder.subfolder/ is a subfolder of a folder (ie. "folder/subfolder")

Most importantly this means that if your maildir folders exist in eg. ~/Maildir/folder and ~/Maildir/folder/subfolder, Dovecot won't see them unless you rename them to Maildir++ layout. Support for this may be added later.

Issues with the specification

Locking

Although maildir was designed to be lockless, Dovecot locks the maildir while doing modifications to it or while looking for new messages in it. This is required because otherwise Dovecot might temporarily see mails incorrectly deleted, which would cause trouble. Basically the problem is that if one process modifies the maildir (eg. a rename() to change a message's flag), another process in the middle of listing files at the same time could skip a file. The skipping happens because readdir() system call doesn't guarantee that all the files are returned if the directory is modified between the calls to it. This problem exists with all the commonly used filesystems.

Because Dovecot uses its own non-standard locking (dovecot-uidlist.lock dotlock file), other MUAs accessing the maildir don't support it. This means that if another MUA is updating messages' flags or expunging messages, Dovecot might temporarily lose some message. After the next sync when it finds it again, an error message may be written to log and the message will receive a new UID.

Delivering mails to new/ directory doesn't have any problems, so there's no need for LDAs to support any type of locking.

Mail delivery

Qmail's [http://www.qmail.org/man/man5/maildir.html how a message is delivered page] suggests to deliver the mail like this:

  1. Create a unique filename (only "time.pid.host" here, later Maildir spec has been updated to allow more uniqueness identifiers)
  2. Do stat(tmp/<filename>). If the stat() found a file, wait 2 seconds and go back to step 1.

  3. Create and write the message to the tmp/<filename>.

  4. link() it into new/ directory. Although not mentioned here, the link() could again fail if the mail existed in new/ dir. In that case you should probably go back to step 1.

All this trouble is rather pointless. Only the first step is what really guarantees that the mails won't get overwritten, the rest just sounds nice. Even though they might catch a problem once in a while, they give no guaranteed protection and will just as easily pass duplicate filenames through and overwrite existing mails.

Step 2 is pointless because there's a race condition between steps 2 and 3. PID/host combination by itself should already guarantee that it never finds such a file. If it does, something's broken and the stat() check won't help since another process might be doing the same thing at the same time, and you end up writing to the same file in tmp/, causing the mail to get corrupted.

In step 4 the link() would fail if an identical file already existed in the maildir, right? Wrong. The file may already have been moved to cur/ directory, and since it may contain any number of flags by then you can't check with a simple stat() anymore if it exists or not.

So really, all that's important in not getting mails overwritten in your maildir is the step 1: Always create filenames that are guaranteed to be unique. Forget about the 2 second waits and such that the Qmail's man page talks about.

Procmail Problems

Maildir format is somewhat compatible with MH format. This is sometimes a problem when people configure their procmail to deliver mails to Maildir/new. This makes procmail create the messages in MH format, which basically means that the file is called msg.inode_number. While this appears to work first, after expunging messages from the maildir the inodes are freed and will be reused later. This means that another file with the same name may come to the maildir, which makes Dovecot think that an expunged file reappeared into the mailbox and an error is logged.

The proper way to configure procmail to deliver to a Maildir is to use Maildir/ as the destination.

References

None: MailboxFormat/Maildir (last edited 2017-05-11 10:20:17 by 46)