A combination of memory channel and file channel aka Spillable Memory Channel is currently in the development phase (not recommended for Production) that would store events in-memory and on the file system after max memory capacity is reached, thus giving the reliability of the file channel and the performance of in-memory queuing (memory channel). So, we have to choose very carefully based on our use-case, but typically, File channels are used in Production to balance performance and reliability. File channel has moderate performance and is moderately secure against failures and lastly, JDBC channel’s performance is slowest but it provides the highest security against failures. For example, memory channel buffered events have higher performance but are also highly volatile and easily susceptible to failures. The type of channel used is based upon a single factor, i.e. memory channel, file channel or JDBC channel (Kafka channel isn’t built-in). The buffer’s properties are based on the type of channel used, i.e. The “Channel” is a buffer between the source and the sink. Flume has built-in compatibility with several sources such as Avro, Thrift, Exec ( /bin/sh -c for example), Spooling directory, Taildir, JMS, Syslog and many more. Runners can be thought of as wrappers that control the source or sink. “Source Runners” and “Sink Runners” are the underlying invisible components of Flume that drive the source and sink, respectively. The extraction process can be event-driven or through frequent polling and is conducted by source-runners. The “Source” is a component which extracts the unstructured data aka “event/s” from one or more applications/clients. Together these three form a “Flume Agent”. There are 3 main components to Flume’s architecture, i.e. Apache Flume Architecture, Flume User Guide

Flume login