Documentation

Meta Tables

Abstract

SummaryInformation about meta tables, including what they are, why Alooma creates them, and how they're used.

Meta tables are tables that Alooma uses as part of our exactly once processing framework.

To guarantee that we don't write an event more than once, every table has an associated <table name>_meta in the same schema. Every time we load new data to your table, we write the following data to the meta table in the same transaction:

  • Timestamp of when was the transaction was executed

  • The number of rows we wrote to the original table in that transaction

  • An internal Transaction ID

  • A few more internal identifiers

This way if there is any internal issue in the system, we can verify if a transaction was already written to the table or not, instead of asking on a row by row basis for each row of data.

While it's intended for our internal processing use, you might find value in them as well. You can query the meta tables to know when Alooma wrote to your table, get an understanding of the distribution of writes over time, or to query row counts over time on a per table basis.

If you drop the meta table we will recreate it on the next transaction, but note that this may create dupes if the drop occurs at the same time as a recovery event.

If you have any questions about the meta table, or anything else for that matter, feel free to reach out!

Search results

    No results found