1. Background

Many companies' platforms generate a large number of logs (generally streaming data, such as PV of search engines, queries, etc.) every day. Processing these logs requires specific log systems. Generally speaking, these systems need to have the following characteristics:

(1) Build the bridge between application system and analysis system, and decouple the relationship between them;

(2) Support near real-time online analysis system and offline analysis system similar to Hadoop;

(3) High scalability. That is, when the amount of data increases, you can expand horizontally by adding nodes.

This paper compares the current open-source log systems in terms of design architecture, load balancing, scalability and fault tolerance, including Facebook's scribe, Apache's chukwa, LinkedIn's Kafka and cloudera's flume.

2. Scribe of Facebook

Scribe is Facebook's open source log collection system, which has been widely used in Facebook. It can collect logs from various log sources and store them on a central storage system (NFS, distributed file system, etc.) for centralized statistical analysis and processing. It provides a scalable and high error tolerance scheme for "distributed collection and unified processing" of logs.

Its most important feature is good fault tolerance. When the back-end storage system crashes, scribe writes the data to the local disk. When the storage system returns to normal, scribe reloads the logs to the storage system.

Framework:

The architecture of scribe is simple, including three parts: scribe agent, scribe and storage system.

(1) scribe agent

The scribe agent is actually a thrift client. The only way to send data to the scribe is to use the thrift client, and a thrift interface is defined in the scribe. The user uses the interface to send data to the server.

(2) scribe

Scribe receives the data sent by thrift client, and sends the data of different topics to different objects according to the configuration file. Scribe provides a variety of stores, such as file, HDFS, etc. scribe can load data into these stores.

(3) Storage system

The storage system is actually the store in the scribe. Currently, the scribe supports many stores, including file, buffer (double-layer storage, one primary storage, one secondary storage), network (another scribe server), bucket (including multiple stores, which store data to different stores through hash), null (ignore data), thriftfile (write to a thrift Tfiletransport file) and multi (store data in different stores at the same time).

3. Chukwa of Apache

Chukwa is a very new open source project. Because it belongs to Hadoop series products, it uses many Hadoop components (HDFS storage, MapReduce data processing). It provides many modules to support Hadoop cluster log analysis.

Demand:

(1) Flexible, dynamic and controllable data source

(2) High performance, highly scalable storage system

(3) Appropriate framework for analysis of collected large-scale data

Framework:

There are three main roles in chukwa: adapter, agent and collector.

(1) Adapter data source

It can encapsulate other data sources, such as file, UNIX command-line tools, etc

Currently available data sources are: Hadoop logs, application measurement data, system parameter data (such as Linux CPU usage flow rate).

(2) HDFS storage system

Chukwa uses HDFS as the storage system. HDFS is designed to support the application scenarios of large file storage and small concurrent high-speed writing, while the characteristics of the log system are just the opposite. It needs to support high concurrent low-speed writing and large amount of small file storage. Note that small files written directly to HDFS are not visible until they are closed. In addition, HDFS does not support file reopening.

(3) Collector and agent

In order to overcome the problems in (2), the agent and collector phases are added.

The function of agent is to provide various services for the adapter, including: starting and closing the adapter, passing the data to the collector through HTTP; recording the status of the adapter regularly for recovery after crash.

Function of collector: merge the data sent from multiple data sources and load it into HDFS; hide the details of HDFS implementation. For example, after the HDFS version is changed, just modify the collector.

(4) Demux and achieving

MapReduce is directly supported for data processing. It has two MapReduce jobs built in, which are used to get data and convert data into structured log. Store in data store (database or HDFS, etc.).

4. Kafka of LinkedIn

Kafka is an open source project in December 2010. It is written in Scala language and uses a variety of efficiency optimization mechanisms. The overall architecture is relatively novel (push / pull), which is more suitable for heterogeneous clusters.

Design objective:

(1) The cost of data access on disk is O (1)

(2) High throughput, processing hundreds of thousands of messages per second on ordinary servers

(3) Distributed architecture, able to partition messages

(4) Supports parallel loading of data into Hadoop

Framework:

Kafka is actually a news publishing and subscription system. Producer publishes a message to a topic, while consumer subscribes to a topic's message. Once there is a new message about a topic, broker will deliver it to all consumers who subscribe to it. In Kafka, messages are organized by topic, and each topic is divided into multiple partitions, which is convenient for data management and load balancing. At the same time, it also uses zookeeper for load balancing.

There are three main roles in Kafka: producer, broker and consumer.

(1) Producer

Producer's task is to send data to broker. Kafka provides two kinds of producer interfaces, one is low level interface, which will send data to a partition under a certain topic of a specific broker; the other is high level interface, which supports synchronous / asynchronous data transmission, broker automatic identification and load balancing based on zookeeper (based on partition).

Among them, the broker automatic recognition based on zookeeper is worth mentioning. Producer can get the list of available brokers through zookeeper, or register a listener in zookeeper, which will wake up in the following situations:

• a. Add a broker
• b. Delete a broker
• c. Register a new topic
• d. Broker registers existing topics

When the producer is informed of the above time, certain actions can be taken as required.

(2) Broker

Broker adopts a variety of strategies to improve data processing efficiency, including sendfile and zero copy technologies.

(3) Consumer

The purpose of consumer is to load the log information to the central storage system. Kafka provides two consumer interfaces, one is low level, which maintains a connection to a broker, and the connection is stateless, that is, each time you pull data from the broker, you need to tell the broker the offset of the data. The other is the high-level interface, which hides the details of the broker and allows consumers to push data from the broker without caring about the network topology. More importantly, for most of the log systems, the data information that the consumer has obtained is saved by the broker, while in Kafka, the data information is maintained by the consumer himself.

5. Cloudera's flume

Flume is cloudera's open source log system in July 2009. Its built-in components are so complete that users can use them without any additional development.

Design objective:

(1) Reliability

When a node fails, logs can be delivered to other nodes without loss. Flume provides three levels of reliability assurance, from strong to weak: end-to-end , store on failure (this is also the strategy used by scribe. When the data receiver crashes, write the data to the local area, and send it after recovery), and best effort (after the data is sent to the receiver, it will not be confirmed).

(2) Scalability

Flume adopts three layers of architecture, including agent, collector and storage. Each layer can be expanded horizontally. Among them, all agents and collectors are managed by the master, which makes the system easy to monitor and maintain, and the master allows multiple (using zookeeper for management and load balancing), which avoids the single point of failure.

(3) Manageability

All agents and collectors are managed by the master, which makes the system easy to maintain. Users can view the execution status of each data source or data flow on the master, and configure and dynamically load each data source. Flume provides two forms of data flow management: Web and shell script command.

(4) Functional scalability

Users can add their own agent, colletor or storage as required. In addition, flume comes with many components, including various agents (file, syslog, etc.), collector and storage (file, HDFS, etc.).

Framework:

As mentioned earlier, flume adopts a hierarchical architecture, which consists of three layers, namely agent, collector and storage. Agent and collector are composed of two parts: source and sink. Source is the data source and sink is the data destination.

(1) agent

The function of agent is to send data from data source to collector. Flume comes with many directly available data sources, such as:

• Text ("filename"): send file filename as data source by line
• Tail ("filename"): detect the newly generated data of filename and send it out by lines
• Fsyslogtcp (5140): listen to port 5140 of TCP and send the received data

At the same time, many sink are provided, such as:

• Console [("format")]: display data directly on the desktop
• Text ("txtfile"): writes data to the file txtfile
• DFS ("dfsfile"): writes data to the dfsfile file on HDFS
• Syslogtcp ("host", port): passing data to the host node through TCP

(2) collector

Collector is used to aggregate the data of multiple agents and load them into storage. Its source and sink are similar to agent.

In the following example, the agent listens for the data received by port 5140 of TCP and sends it to the collector, which loads the data to HDFS.

A more complex example is as follows:

There are 6 agents and 3 collectors, all of which import data into HDFS. Agent a, B send data to collector a, agent C, d send data to collector B, agent E, f send data to collector C. At the same time, add end-to-end reliability guarantee for each agent (three kinds of flume reliability guarantee are implemented by agente2echain, agentdfochain, and agentbechain respectively). For example, when collector a fails, agent a and agent B will send data to collector B and collector C respectively.

Here is a short profile snippet:

In addition, with autoe2echain, when a collector fails, flume will automatically detect an available collector and direct data to the new available collector.

(3) storage

Storage is a storage system, which can be a normal file, HDFS, hive, HBase, etc.

6. summary

According to the architecture of these four systems, it can be concluded that a typical log system needs three basic components: agent (encapsulating the data source, sending the data from the data source to the collector), collector (receiving the data from multiple agents, summarizing it and importing it into the back-end store), store (central storage system, which should be extensible and reliable, should be This supports HDFS, which is very popular at present.

The following table compares the four systems:

7. References

• Scribe homepage: https://github.com/facebook/scribe
• Chukwa home page: http://incubator.apache.org/chukwa/
• Kafka homepage: http://sna-projects.com/kafka/
• Flume homepage: https://github.com/cloudera/flume/

Original article, reprint please note: reprint from Dong's blog

Link address: open source log system comparison

WeChat official account: hadoop-123, focusing on big data technology sharing. Welcome to join us!