What should businesses know before adopting Apache Kafka?

byMaytal Shamir
Updated Mar 5, 2018

I absolutely agree with the previous two answers which were given here.

When a business is starting to plan a project which depends on Apache Kafka technology, they should first define it’s use case. In some cases Kafka won’t be the best selection. However, if you decide that Kafka is your best option - start by defining the following:

  1. Data model
  2. Topics and partitions structure
  3. Deployment process

These are the basics of getting started with Kafka. Remember that in order to create a fault tolerant, resilient, scalable and secure system you should also think about:

  1. Cluster size and replication factor
  2. Monitoring (using Kafka JMX metrics for example)
  3. Data serialization and compaction
  4. Exactly once processing

It takes a lot of planning, trial and error, and experienced people in order to get to the optimal solution. Here at Alooma we are using Kafka for streaming to power our customers’ data from their sources (Kafka, MongoDB, MySQL etc.) to their target output (eg. Snowflake, Redshift, BigQuery). We’ve also built a way for you to visualize your Kafka topics.

good luck!

Like what you read? Share on

Published at Quora. See Original Question here

Apache KafkaData Pipelines

Further reading

What is Confluent?
Alooma Team • Updated Jul 24, 2018
What problem is Kafka solving?
Maytal Shamir • Updated Jul 13, 2018