This documentation is for Dovecot v2.x, see wiki1 for v1.x documentation.

dbox

dbox is Dovecot's own high-performance mailbox format. The original version was introduced in v1.0 alpha4, but since then it has been completely redesigned in v1.1 series and improved even further in v2.0.

dbox can be used in two ways:

  1. single-dbox (sdbox in mail location): One message per file, similar to Maildir. For backwards compatibility, dbox is an alias to sdbox in mail_location.

  2. multi-dbox (mdbox in mail location): Multiple messages per file, but unlike mbox multiple files per mailbox.

One of the main reasons for dbox's high performance is that it uses Dovecot's index files as the only storage for message flags and keywords, so the indexes don't have to be "synchronized". Dovecot trusts that they're always up-to-date (unless it sees that something is clearly broken). This also means that you must not lose the dbox index files, they can't be regenerated without data loss.

dbox has a feature for transparently moving message data to an alternate storage area. See Alternate storage below.

dbox storage is extensible. Single instance attachment storage was already implemented as such extension.

Layout

By default, the dbox filesystem layout will be as follows. Data which isn't the actual message content is stored in a layout common to both single-dbox and multi-dbox:

Note that with dbox the Index files actually contain significant data which is held nowhere else. Index files for both single-dbox and multi-dbox contain message flags and keywords. For multi-dbox, the index file also contains the map_uids which link (via the "map index") to the actual message data. This data cannot be automatically recreated, so it is important that Index files are treated with the same care as message data files.

Index files can be stored in a different location by using the INDEX parameter in the mail location specification. If the INDEX parameter is specified, it will make Dovecot look for the Index files as follows:

Actual message content is stored differently depending on whether it is single-dbox or multi-dbox.

Under single-dbox we have:

Under multi-dbox we have:

With Dovecot versions 2.0.4 and later, setting the INDEX parameter sets the location of the "map index" as well as the location of the mailbox indexes. So this would make the "map index" be stored as follows:

Multi-dbox

You can enable multi-dbox with:

mail_location = mdbox:~/mdbox

The directory layout (under ~/mdbox/) is:

The storage directory has files:

Each m.* file contains one or more messages. mdbox_rotate_size setting can be used to configure how large the files can grow.

The map index contains a record for each message:

Mailbox indexes refer to messages only using map_uids. This allows messages to be moved to different files by updating only the map index. Copying is done simply by appending a new record to mailbox index containing the existing map_uid and increasing its refcount. If refcount grows over 32768, currently Dovecot gives an error message. It's unlikely anyone really wants to copy the same message that many times.

Expunging a message only decreases the message's refcount. The space is later freed in "purge" step. This is typically done in a nightly cronjob when there's less disk I/O activity. The purging first finds all files that have refcount=0 mails. Then it goes through each file and copies the refcount>0 mails to other mdbox files (to the same files as where newly saved messages would also go), updates the map index and finally deletes the original file. So there is never any overwriting or file truncation.

The purging can be invoked explicitly running doveadm purge.

There are several safety features built into dbox to avoid losing messages or their state if map index or mailbox index gets corrupted:

Alternate storage

Unlike Maildir, with dbox the message file names don't change. This makes it possible to support storing files in multiple directories or mount points. dbox supports looking up files from "altpath" if they're not found from the primary path. This means that it's possible to move older mails that are rarely accessed to cheaper (slower) storage.

To enable this functionality, use the ALT parameter in the mail location. For example, specifying the mail location as:

mail_location = mdbox:/var/vmail/%d/%n:ALT=/altstorage/vmail/%d/%n

will make Dovecot look for message data first under /var/vmail/%d/%n ("primary storage"), and if it is not found there it will look under /altstorage/vmail/%d/%n ("alternate storage") instead. There's no problem having the same (identical) file in both storages.

Keep the unmounted /altstorage directory permissions such that Dovecot mail processes can't create directories under it (e.g. root:root 0755). This way if the alt storage isn't mounted for some reason, Dovecot won't think that all the messages in alt storage were deleted and lose their flags. With v2.1 this isn't strictly requires anymore, because it keeps track of missing mountpoints.

When messages are moved from primary storage to alternate storage, only the actual message data (stored in files u.* under single-dbox and m.* under multi-dbox) is moved to alternate storage; everything else remains in the primary storage.

Message data can be moved from primary storage to alternate storage using doveadm altmove. (In theory you could also do this with some combination of cp/mv, but better not to go there unless you really need to. The updates must be atomic in any case, so direct cp won't be work.)

The granularity at which data is moved to alternate storage is individual messages. This is true even for multi-dbox when multiple messages are stored in a single m.* storage file. If individual messages from an m.* storage file need to be moved to alternate storage, the message data is written out to a different m.* storage file (either new or existing) in the alternate storage area and the "map index" updated accordingly.

Alternate storage is completely transparent at the IMAP/POP level. Users accessing mail through IMAP or POP cannot normally tell if any given message is stored in primary storage or alternate storage. Conceivably users might be able to measure a performance difference; the point is that there is no IMAP/POP command which could be used to expose this information. It is entirely possible to have a mail folder which contains a mix of messages stored in primary storage and alternate storage.

dbox and mail header metadata

Unlike when using mbox as mailbox format, where mail headers (for example Status, X-UID, etc.) are used to determine and store meta-data, the mail headers within dbox files are (usually) not used for this purpose by dovecot; neither when mails are created/moved/etc. via IMAP nor when dboxes are placed (e.g. copied or moved in the filesystem) in a mail location (and then "imported" by dovecot). Therefore, it is (usually) not necessary, to strip any such mail headers at the MTA, MDA or LDA (as it is recommended with mbox).

There is one exception, though, namely when pop3_reuse_xuidl=yes (which is however rather deprecated): In this case X-UIDL is used for the POP3 UIDLs. Therefore, in this case, is recommended to strip the X-UIDL mail headers case-insensitively at the MTA, MDA or LDA.

Accessing expunged mails with mdbox

"mdbox_deleted" storage can be used to access mdbox's all mails that are completely deleted (reference count=0). The mdbox_deleted parameters should otherwise be exactly the same as mdbox's. Then you can use e.g. doveadm fetch or doveadm import commands to access the mails. For example:

# If you have mail_location=mdbox:~/mdbox:INDEX=/var/index/%u
doveadm import mdbox_deleted:~/mdbox:INDEX=/var/index/%u "" subject oops

This finds a deleted mail with subject "oops" and imports it into INBOX.

Mail delivery

Some MTA configurations have the MTA directly dropping mail into Maildirs or mboxes. Since most MTAs don't understand the dbox format, this option is not available. Instead, the MTA could be set up to use LMTP#mta or the Dovecot Local Delivery Agent.

MailboxFormat/dbox (last edited 2014-11-20 08:13:20 by TimoSirainen)