Maintaining its custom data pipeline had simply grown to be too burdensome, and the effort was taking resources away from doing more productive work that actually benefited the business. According to Bob Briski, Director of Data: “We have an onslaught of data coming constantly from both internal systems and our business partners that we store in an Amazon Redshift data warehouse, where we currently have over 2 billion records with over a million more being added every day.” Briski was also concerned about the accuracy and reliability of the custom scripts being used.
Because it assures the quality and fidelity of the data. “Anyone who has worked in data analytics knows about the challenges involved in moving data in a consistent way from one place to another. It’s not glamorous, but if it’s not done right, all the advanced data science and sophisticated machine learning techniques are essentially useless.”
Briski especially appreciates the way the Alooma solution enables him to validate the data, and to intervene to resolve any issues immediately: “The re-stream queue quickly became my favorite feature. I’m not aware of any other data pipeline that has this ability to capture all of the data that doesn’t fit the schema, correct it, and then re-stream it into the data warehouse. That’s a real game-changer.”
With the Alooma data pipeline completely replacing the company’s custom scripts, staff has been relieved of the drudgery of data movement, and can now focus entirely on data analytics. With its comprehensive set of integrations, Briski is now able to access data that was not previously being loaded into the Redshift data warehouse.
“We’re now able to move every bit of data, no matter how big or small, or whatever the source or format, and we’re doing it without errors and without having a team of people running around creating spreadsheets or writing scripts.”
Perhaps best of all is that Briski can now take the company’s data pipeline for granted. “It’s always running loading the data, and I never need to worry about the quality. I get weekly updates on what’s been streamed, and I’m happy every time because I see the inevitable issues are now being detected and resolved.”