This documentation is for Dovecot v2.x, see wiki1 for v1.x documentation.

Input Streams

lib/istream.h describes Dovecot's input streams. Input streams can be stacked on top of each others as many times as wanted.

Input streams actually reading data:

Input stream filters:

Reading

i_stream_read() tries to read more data into the stream's buffer. It returns:

Reading from a stream doesn't actually go forward in the stream, that needs to be done manually with i_stream_skip(). This makes it easy to read full data records into the stream directly instead of creating separate buffers. For example when reading line-based input you can keep reading input into the stream until you find LF and then just access the string directly from the input buffer. There are actually helper functions for this: i_stream_next_line() attempts to return the next line if available, i_stream_read_next_line() does the same but does a read to try to get the data.

Because more and more data can be read into the buffer, the buffer size is typically limited, and once this limit is reached read returns -2. The buffer size is usually given as parameter to the i_stream_create_*(), filters use their parent stream's buffer size. The buffer size can be also changed with i_stream_set_max_buffer_size(). Figuring out what the buffer size should be depends on the situation. It should be large enough to contain all valid input, but small enough that users can't cause a DoS by sending a too large record and having Dovecot eat up all the memory.

Once read returns -1, the stream has reached EOF. stream->eof=TRUE is also set. In this situation it's important to remember that there may still be data available in the buffer. If i_stream_have_bytes_left() returns FALSE, there really isn't anything left to read.

Whenever i_stream_read() returns >0, all the existing pointers are potentially invalidated. v2.3+: When i_stream_read() returns <= 0, the data previously returned by i_stream_get_data() are still valid, preserved in "snapshots". (<v2.3 may or may not have invalidated them.)

Example:

/* read line-based data from file_fd, buffer size has no limits */
struct istream *input = i_stream_create_fd(file_fd, (size_t)-1, FALSE);
const char *line;

/* return the last line also even if it doesn't end with LF.
   this is generally a good idea when reading files (but not a good idea
   when reading commands from e.g. socket). */
i_stream_set_return_partial_line(input, TRUE);
while ((line = i_stream_read_next_line(input)) != NULL) {
  /* handle line */
}
i_stream_destroy(&input);

Internals

lib/istream-internal.h describes the internal API that input streams need to implement. The methods that need to be implemented are:

There are some variables available:

If your input stream needs a write buffer, you can use some of the common helper functions and variables:

The snapshots have made implementing slightly more complicated than earlier. There are a few different ways to implement istreams:

When Dovecot is configured with --enable-devel-checks, i_stream_read() will verify that the first and the last two bytes of the buffer didn't unexpectedly change due to a read(). While developing istream changes you should use this to make sure the istream is working properly. Running the istream unit test also via valgrind can also be used to verify that the buffer wasn't freed.

Design/InputStreams (last edited 2017-12-10 17:54:10 by TimoSirainen)