This documentation is for Dovecot v2.x, see wiki1 for v1.x documentation.
Differences between revisions 2 and 3
Revision 2 as of 2006-12-10 21:21:49
Size: 677
Comment:
Revision 3 as of 2007-03-24 19:42:31
Size: 2053
Editor: TimoSirainen
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
= Dovecot Index Files Design = = Dovecot's index files =
Line 3: Line 3:
Each dovecot folder has an index.
I
ndex files consist of three different files:
Dovecot's index files consist of three different files:
Line 6: Line 5:
 * Main index file (dovecot.index)
 * Transaction log (dovecot.index.log)
 * Cache file (dovecot.index.cache)
 * Main index file ({{{dovecot.index}}})
 * [:Design/Indexes/TransactionLog:Transaction log] ({{{dovecot.index.log}}} and {{{dovecot.index.log.2}}})
 * Cache file ({{{dovecot.index.cache}}})
Line 10: Line 9:
If index files are missing, Dovecot rebuilds them automatically when the folder is accessed the next time. Each mailbox has its own separate index files. If the index files are disabled, the same structure are still kept in the memory, except cache file is disabled completely (because the client probably won't fetch same data twice).
Line 12: Line 11:
The transaction log (dovecot.index.log) contains all changes made to the index. If index files are missing, Dovecot creates them automatically when the mailbox is opened. If at any point creating a file or growing a file gives "not enough disk space" error, the indexes are transparently moved to memory for the rest of the session.
Line 14: Line 13:
If you have trouble with reading large folders (10,000 mails), you can try removing the index files.
If you have upgraded from an old version of dovecot, the index files are called .imap.index et. al. You can safely remove them.
== Lockless integers ==
[[Anchor(locklessint)]]
Line 17: Line 16:
FIXME: continue. Dovecot uses several different techniques to allow reading files without locking them. One of them uses fields in a "lockless integer" format. Initially these fields have "unset" value. They can be set to a wanted value in range 0..2^28^ (with 32bit fields) once, but they cannot be changed. It would be possible to set them back to "unset", but setting them the second time isn't safe anymore, so Dovecot never does this.

The lockless integers work by allocating one bit from each byte of the value to "this value is set" flag. The reader then verifies that the flag is set for the value's all bytes. If all of them aren't set, the value is still "unset". Dovecot uses the highest bit for this flag. So for example:

 * 0x00000000: The value is unset
 * 0xFFFF7FFF: The value is unset, because one of the bytes didn't have the highest bit set
 * 0xFFFFFFFF: The value is 2^28^-1
 * 0x80808080: The value is 0
 * 0x80808180: The value is 0x80

Dovecot contains {{{mail_index_uint32_to_offset()}}} and {{{mail_index_offset_to_uint32()}}} functions to translate values between integers and lockless integers. The "unset" value is returned as 0, so it's not possible to differentiate between "unset" and "set" 0 values.

Dovecot's index files

Dovecot's index files consist of three different files:

  • Main index file (dovecot.index)

  • [:Design/Indexes/TransactionLog:Transaction log] (dovecot.index.log and dovecot.index.log.2)

  • Cache file (dovecot.index.cache)

Each mailbox has its own separate index files. If the index files are disabled, the same structure are still kept in the memory, except cache file is disabled completely (because the client probably won't fetch same data twice).

If index files are missing, Dovecot creates them automatically when the mailbox is opened. If at any point creating a file or growing a file gives "not enough disk space" error, the indexes are transparently moved to memory for the rest of the session.

Lockless integers

Anchor(locklessint)

Dovecot uses several different techniques to allow reading files without locking them. One of them uses fields in a "lockless integer" format. Initially these fields have "unset" value. They can be set to a wanted value in range 0..228 (with 32bit fields) once, but they cannot be changed. It would be possible to set them back to "unset", but setting them the second time isn't safe anymore, so Dovecot never does this.

The lockless integers work by allocating one bit from each byte of the value to "this value is set" flag. The reader then verifies that the flag is set for the value's all bytes. If all of them aren't set, the value is still "unset". Dovecot uses the highest bit for this flag. So for example:

  • 0x00000000: The value is unset
  • 0xFFFF7FFF: The value is unset, because one of the bytes didn't have the highest bit set
  • 0xFFFFFFFF: The value is 228-1

  • 0x80808080: The value is 0
  • 0x80808180: The value is 0x80

Dovecot contains mail_index_uint32_to_offset() and mail_index_offset_to_uint32() functions to translate values between integers and lockless integers. The "unset" value is returned as 0, so it's not possible to differentiate between "unset" and "set" 0 values.

None: Design/Indexes (last edited 2009-03-15 22:35:13 by localhost)