Extract, Transform, and Load (ETL) tools enable organizations to make their data accessible, meaningful, and usable across disparate data systems. When it comes to choosing the right ETL tool, there are many options to chose from. So, where should you start?
We've prepared a list that is simple to digest, organized into four categories to help you find the best solution for your needs.
Incumbent Batch ETL Tools
Until recently, most of the world’s ETL tools were on-prem and based on batch processing. Historically, most organizations used to utilize their free compute and database resources to perform nightly batches of ETL jobs and data consolidation during off-hours. This is why, for example, you used to see your bank account updated only a day after you made a financial transaction.
Cloud Native ETL Tools
With IT moving to the cloud, more and more cloud-based ETL services started to emerge. Some of them keep the same basic batch model of the legacy platforms, while others start to offer real-time support, intelligent schema detection, and more.
Open Source ETL Tools
Similarly to other areas of software infrastructure, ETL has had its own surge of open source tools and projects. Most of them were created as a modern management layer for scheduled workflows and batch processes. For example, Apache Airflow was developed by the engineering team at AirBnB, and Apache NiFi by the US National Security Agency (NSA).
Real-Time ETL Tools
Doing your ETL in batches makes sense only if you do not need your data in real-time. It might be good for salary reporting or tax calculations. However, most modern applications require a real-time access to data from different sources. When you upload a picture to your Facebook account, you want your friends to see it immediately, not a day later.
This shift to real-time generated a profound change in architecture: from a model based on batch processing to a model based on distributed message queues and stream processing. Apache Kafka has emerged as the leading distributed message queue for modern data applications, and companies like Alooma and others are building modern ETL solutions on top of it, either as a SaaS platform or an on-prem solution.
This post contains some representative examples for each ETL category to help you make the choice that meets your needs. For a complete, uncategorized list we recommend this post.