What problem is Kafka solving?

byMaytal Shamir
Updated Jul 13, 2018

Kafka can be used to solve a variety of problems.

Most of the systems that I know which use Kafka are distributed, and involve real-time data processing of a very large scale of messages (for example a live stream of posts from Facebook).

To better understand Kafka strengths, first ask yourself: “what will happen if I’ll solve my problem without Kafka?”(For example, as a solution, use a direct REST requests to transfer data between two services, or use a simple queue between them).

Now after you thought about a different solution, ask yourself: “How will this solution be affected if the scale grows by the factor of 100000?”

These questions always bring me to the conclusion that Kafka allows you to think big, without compromising on the stability, reliability and the high performance of your system.

At Alooma (where I work these days), we’re using Kafka for processing a very large number of data streams, which arrive from our customers databases. Kafka replication factor, leader-election logic, and many more features, enables us to make sure that the data is safe, reliable and will get to its destination in no time.

Like what you read? Share on

Published at Quora. See Original Question here

Apache Kafka

Further reading

What is Confluent?
Alooma Team • Updated Jul 24, 2018
What is the future of ETL tools?
Ofri Raviv • Updated Dec 14, 2018