Getting your Salesforce data into your BigQuery data warehouse is the first step in setting up a powerful analytical workflow and getting valuable insights from your data. Typically, having that data together with data from your various other sources in BigQuery delivers a compounding effect. The more data you have, the better your analysis, the better your insights.
At first, the idea of moving data from Salesforce into BigQuery may sound straight forward. You simply use the Data Loader export wizard and select the objects you want to export (and whether you want to export soft-deleted records). But what if you have huge Salesforce objects with gigabytes of data? You'll need a secure and robust way to pipe that data into BigQuery. That's likely to require a lot of time and specialized resources.
And usually around this point you realize that you might want to set up a way to repeat this task. Whether you want to perform the same migration into BigQuery periodically, or you want to add different objects (or even different data sources besides Salesforce), you'll need someone to build a method for scheduling, tracking, and logging the process. And it will need to be scalable. Oh, and you'll need to make sure you have a way of catching and handling any errors that occur along the way.
You may also notice an opportunity to scrub PII (personally identifiable information) or enrich the data with things like geolocation data or currency conversion before the data is uploaded.
It's not an easy task, and unless you already have a seasoned team in place, you will need to train or hire someone who has the expertise to do the work. In reality, by the time you factor in security concerns, headcount, training, and technical complexity, you realize that you are, in essence, building your own ETL platform, just to extract your Salesforce data.
The solution: Alooma
We recommend that you don't build a custom ETL tool and take on all of the technical challenge and resource costs. The better solution is to use a modern ETL platform designed to move data from Salesforce (and other sources) into BigQuery and make strategic transformations along the way.
Alooma is the enterprise data platform built for the cloud. With built-in support for Salesforce and BigQuery, and bolstered by enterprise security and scalability, it's the ideal solution.
Importing your Salesforce data into BigQuery
Getting your Salesforce data into BigQuery is incredibly simple with Alooma. Let's break down the process.
On the Plumbing page, click "Add new input" and select Salesforce from the list of integrations.
Enter your account information as required, name the input, and then select all the objects you want to import.
That's all there is to it. Once you save your input, assuming your credentials are correct, your Salesforce data will automatically begin importing into BigQuery. See our Salesforce documentation for more information.
Of course, there's a lot more you could do along the way:
- You could use the Code Engine to transform/enrich/cleanse data as it flows from Salesforce to BigQuery.
- You could change how the schema is mapped, via the Mapper; however, most of the time Alooma's powerful auto-mapping works just fine.
- You could click on the Live tab for your Salesforce input and monitor the data flow. Or click the Samples tab to see examples of the actual data being loaded.
Put Your Data to Work: Now that you have your Salesforce data in BigQuery you can, for example, merge client usage data with payment data to get insights on your business.
Bust Data Silos: Don't just work with data from Salesforce. Perform an information census and look for data silos within your company. Integrating multiple data sources into BigQuery is straightforward and simple, and each new source — whether it's a stream, a database, a file, etc. — potentially increases the usefulness and impact of your analysis.
Automate the Process: Using an enterprise data platform means you can automate data extraction and transformation from multiple sources without having to build out your infrastructure.
Enterprise scalability and performance: The Alooma platform provides horizontal scalability, handling as many events from as many data sources as you need.
Security at the core: The Alooma platform is built around a robust and flexible security architecture, providing full visibility and control over data. SOC 2 Type II, HIPAA and EU-US Privacy Shield, GDPR compliant, Alooma does not store any data permanently and encrypts all data in motion.
Guaranteed data integrity and reliability: The Restream Queue, Alooma's intelligent data integrity engine, is your safety net for ensuring zero data loss. The Restream Queue collects all the events that were not loaded to BigQuery, for whichever reason, making them easy to fix and enabling you to "restream" them into BigQuery later.
Flexible data enrichment: The Code Engine, a stateful, python-based processing engine, enables on the fly data enrichment for sophisticated use cases, such as real-time alerts, sessionization, anomaly detection, and more. Customize your data exactly how you want by writing real code to transform data on the stream.
Simple yet powerful data management: The Mapper automatically infers schemas, maps schema changes, or enables customization of mappings to your liking, ensuring you meet all your data governance requirements.
Cost effective: You won't need to hire or train staff to build the process, saving time and money. You won't need to buy more machines or processing power as your data grows, and adding new data sources to import into BigQuery is a breeze.
Ultimately, you want the process of getting insights from your data to be as simple as possible. The fewer steps, the lower the cost, the better. And if you can get data from other sources thrown in without requiring custom coding or processes, you're even further ahead of the game. Using the cloud to store and process that data is the natural next step.
Alooma was designed and built for the cloud. Alooma enables businesses to use all of their data to make better data-driven decisions, providing Data Scientists and Data Engineers the ability to integrate, cleanse, enrich, and bring together batch or streaming data from various data silos at any time to any destination.
Alooma makes the whole process of getting your Salesforce data into BigQuery simple and affordable.
Ready to get started? Alooma can help. Contact Alooma today to learn more about how a Salesforce and BigQuery integration solution can benefit your business.
Salesforce is the leading cloud based CRM tool. With Salesforce you can store all your customer information and all your customer interactions in one place.
Google BigQuery is a powerful Big Data analytics platform that enables super-fast SQL queries against append-only tables using the processing power of Google's infrastructure.