What is the difference between a data pipeline and an ETL pipeline?

byOfri Raviv
Updated Oct 29, 2017

They are two related, but different terms, and I guess some people use them interchangeably.

ETL pipeline refers to a set of processes extracting data from one system, transforming it, and loading into some database or data-warehouse.

Data pipeline is a slightly more generic term. It refers to any set of processing elements that move data from one system to another, possibly transforming the data along the way.

The term ETL pipeline usually implies that the pipeline works in batches - for example, the pipe is run once every 12 hours, while data pipeline can also be run as a streaming computation (meaning, every event is handled as it occurs).

Another type of a data pipeline that is an ETL pipeline, is an ELT pipeline: loading all of your data to the data warehouse, and transforming it only later.

Additionally, data pipeline doesn’t have to end in loading the data to a database or a data warehouse. It can, for example, trigger business processes by triggering webhooks on other systems

Further read on how to build a data pipeline.

Like what you read? Share on

Published at Quora. See Original Question here

Data PipelinesExtract, Transform, Load (ETL)

Further reading

What is CloverETL?
Alooma Team • Updated May 9, 2018
What is Oracle Data Integrator?
Alooma Team • Updated Apr 3, 2018
Take control of your data for free!
Sign up and get $500 worth of free credits to try Alooma.
Get started