This documentation is for Dovecot v2.x, see wiki1 for v1.x documentation.
Differences between revisions 1 and 24 (spanning 23 versions)
Revision 1 as of 2006-05-12 20:50:55
Size: 4678
Editor: TimoSirainen
Revision 24 as of 2019-09-12 08:22:35
Size: 75
Comment: Moved to new doc
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
= Mbox Mailbox Format =

Usually UNIX systems are configured by default to deliver mails to `/var/mail/username` or `/var/spool/mail/username` mboxes. In IMAP world these files are called INBOX mailboxes. IMAP protocol supports multiple mailboxes however, so there needs to be a place for them as well. Typically they're stored in `~/mail/` or `~/Mail/` directories.

The mbox file contains all the messages of a single mailbox. Because of this, the mbox format is typically thought of as a slow format. However with Dovecot's indexing this isn't true. Only expunging messages from the beginning of a large mbox file is slow with Dovecot, most other operations should be fast.

== Locking ==

Locking is a mess with mboxes. There are multiple different ways to lock a mbox, and software often uses incompatible locking. There are at least four different ways to lock a mbox:

 * '''dotlock''': `mailboxname.lock` file created by almost all software when writing to mboxes. This grants the writer an exclusive lock over the mbox, so it's usually not used while reading the mbox so that other processes can also read it at the same time. So while using a dotlock typically prevents actual mailbox corruption, it doesn't protect against read errors if mailbox is modified while a process is reading.
 * '''flock''': `flock()` system call is quite commonly used for both read and write locking. The read lock allows multiple processes to obtain a read lock for the mbox, so it works well for reading as well. The one downside to it is that it doesn't work if mailboxes are stored in NFS.
 * '''fcntl''': Very similar to '''flock''', also commonly used by software. In some systems this `fcntl()` system call is compatible with `flock()`, but in other systems it's not, so you shouldn't rely on it. '''fcntl''' works with NFS if you're using lockd daemon in both NFS server and client.
 * '''lockf''': POSIX `lockf()` locking. Because it allows creating only exclusive locks, it's somewhat useless so Dovecot doesn't support it. With Linux `lockf()` is internally compatible with `fcntl()` locks, but again you shouldn't rely on this.

=== Dotlock ===

Another problem with dotlocks is that if the mailboxes exist in `/var/mail/`, the user may not have write access to the directory, so the dotlock file can't be created. There are a couple of ways to work around this:

 * Give a mail group write access to the directory and then make sure that all software requiring access to the directory runs with the group's privileges. This may mean making the binary itself setgid-mail, or using a separate dotlock helper program which is setgid-mail. With Dovecot this can be done by setting `mail_extra_groups = mail`.
 * Set sticky bit to the directory (`chmod +t /var/mail`). This makes it somewhat safe to use, because users can't delete each others mailboxes, but they can still create new files (the dotlock files). The downside to this is that users can create whatever files they wish in there, such as a mbox for newly created user who hadn't yet received mail.

=== Deadlocks ===

If multiple lock methods are used, which is usually the case since dotlocks aren't typically used for read locking, the order in which the locking is done is important. Consider if two programs were running at the same time, both use dotlock and fcntl locking but in different order:

 * Program A: fcntl locks the mbox
 * Program B at the same time: dotlocks the mbox
 * Program A continues: tries to dotlock the mbox, but since it's already dotlocked by B, it starts waiting
 * Program B continues: tries to fcntl lock the, but since it's already fcntl locked by A, it starts waiting

Now both of them are waiting for each others locks. Finally after a couple of minutes they time out and fail the operation.

=== Locking used by different software ===

 * Debian: Debian's policy specifies that all software should use "fcntl and then dotlock" locking.
 * procmail: `procmail -v 2>&1|grep Lock`
 * mutt: `mutt -v|grep -i lock`
 * Dovecot: `mbox_read_locks` and `mbox_write_locks` settings.

== Dovecot's Metadata ==

Dovecot uses C-client (ie. UW-IMAP, Pine) compatible headers in mbox messages to store metadata. These headers are:

 * X-IMAPbase: Contains UIDVALIDITY, last used UID and list of used keywords
 * X-IMAP: Same as X-IMAPbase but also specifies that the message is a "pseudo message"
 * X-UID: Message's allocated UID
 * Status: R (\Seen) and O (non-\Recent) flags
 * X-Status: A (\Answered), F (\Flagged), T (\Draft) and D (\Deleted) flags
 * X-Keywords: Message's keywords
 * Content-Length: Length of the message body in bytes
Moved to

None: MailboxFormat/mbox (last edited 2019-09-12 08:22:35 by MichaelSlusarz)