What is the future of ETL tools?
ETL tools will become simpler to setup and use, will move to the cloud, and they’ll put much more focus on real-time data processing.
Simplicity and the cloud
Many companies are choosing not to build and maintain their ETL in-house, but instead to use external service providers like Alooma and Fivetran (product). This helps them reduce the number of data engineers they need to hire, and lets them focus on their core business.
As more and more of your data resides in 3rd party SAAS websites, it doesn’t make a lot of sense to write the logic of extracting and cleaning data from these websites - many companies have already done that before you, so you’re probably better off just using their service to do it.
Additionaly, many companies today find that they need to scale up their ETL operation quite rapidly. Even if you use great infrastructure, scaling up ETL processing is a real challenge, and using a service takes care of that too.
Real-time data processing
Traditionally, ETL was done as nightly jobs. You can throw more computing power at it, and maybe even run it hourly, but more and more businesses are realizing that a lot of the value of their data lies in their ability to use it immediately to drive automatic business decisions, instead of at an hour or a day’s delay. ETL tools will adopt capabilities from the world of stream processing to handle these use-cases. We recently wrote a blog post on the transition of ETL tools to real-time, so you can read more there.