Alooma: Solving your ETL problems, one question at a time

Keeping your data integrity is like cleaning your house. It’s a repetitive, iterative process of carefully examining each piece of data and making sure its is not skewed, missing, or duplicated. It’s a never ending process, especially in our era of abundant data sources generating millions of records daily.

Goal is to use python to build the ETL as well as Reporting applications.

Before we dive into the specific details of Alooma, we can briefly talk about trends in the ETL world that moved from batch-based to stream-oriented solution.

As a business you want to minimize the amount of effort you need to put into aspects which are not your main line of business, and unless you are Amazon, Google or Snowflake, for example, that is most likely not building or maintaining data warehouses.

Does Data Lake include raw data only or transformed data as well?

Spark is an open source processing engine built around speed, ease of use, and analytics. It’s good for large amounts of data that requires low latency.

This is actually one of the problems we are trying to address at Alooma - building a robust data pipeline that can take your inputs and reliably move the data into Redshift (in real time, without any data loss, and performing advanced transformation along the way).

There are a few ways to address this problem, and it mostly depends on what the requirements are and where the server is hosted.

They are two related, but different terms, and I guess some people use them interchangeably.

Amazon Aurora is a relational database engine. It’s designed to be compatible with MySQL 5.6, so that existing MySQL applications and tools can run without requiring modification. It’s good for production usage for lots of applications but not necessarily for complex data analytics.

There are two different approaches to analytics - custom and off-the-counter. I’m a personal fan of custom analytics, so for me the main ingredients of a stable, strong analytics stack are

Keeping your data pipeline working properly. It is an ever-growing, never-ending job.

A major factor here is that companies that provide ETL solutions do so as their core business focus, which means they will constantly work on improving their performance and stability while providing new features (sometimes ones you can’t foresee needing until you hit a certain roadblock on your own).