What should businesses know before adopting Apache Kafka?
I absolutely agree with the previous two answers which were given here.
When a business is starting to plan a project which depends on Apache Kafka technology, they should first define it’s use case. In some cases Kafka won’t be the best selection. However, if you decide that Kafka is your best option - start by defining the following:
- Data model
- Topics and partitions structure
- Deployment process
These are the basics of getting started with Kafka. Remember that in order to create a fault tolerant, resilient, scalable and secure system you should also think about:
- Cluster size and replication factor
- Monitoring (using Kafka JMX metrics for example)
- Data serialization and compaction
- Exactly once processing
It takes a lot of planning, trial and error, and experienced people in order to get to the optimal solution. Here at Alooma we are using Kafka for streaming to power our customers’ data from their sources (Kafka, MongoDB, MySQL etc.) to their target output (eg. Snowflake, Redshift, BigQuery). We’ve also built a way for you to visualize your Kafka topics.
Published at Quora. See Original Question here