Documentation

Event Metadata

Abstract

SummaryA description of the metadata dictionary added to every event.

Every event includes a metadata dictionary with additional event information. Some of the fields can exist in every event (Input Source: All), while some fields are specific to the integration/input source from which they originate. The complete set of fields is the combination of both. For example, the metadata for a MySQL integration would include the applicable fields for the "All" and "MySQL" input sources.

This dictionary is accessed with event['_metadata'] and can be used in the Code Engine to make decisions about the event as well as mapped through the Mapper.

Note

Snowflake users, do not map _metadata to VARIANT.

The table below describes the metadata fields and the input sources with which they are associated.

Example Metadata dictionaries are included after the field name listing.

Field Name

Description

Input Source

input_label

The input name as shown on the Plumbing page.

e.g."Rest_Endpoint"

All

event_type

The name used to group different types of events in the Mapper. e.g."UserProfiles"

All

@timestamp

ISO 8601 timestamp of when the event was received by Alooma.

e.g."2015-10-16T15:26:57.027Z"

All

@uuid

Unique identifier for the event.

e.g. "af721753-370a-4cc8-9463-5f62c82988d2"

All

@parent_uuid

When splitting events in the code engine, the @uuid of the event that created the current event is included in the child event(s).

e.g. "af721753-370a-4cc8-9463-5f62c82988d2"

All

schema_url

The URL where the schema of the input is defined, such that if the input has a schema defined for it this is the location where that schema can be found.

All

input_type

The type of input, e.g. "MySQL"

All

input_id

The backend ID Alooma gave the input when it was created.

All

restream_count

The number of times the event has been through the restream queue - starts at 0.

All

deleted

Whether the row was deleted. e.g. true

All

snapshot_id

The ID that indicates which dump/load cycle this event is associated with.

All *

snapshot_size

The number of events that exist in the table during this dump/load cycle. e.g. 12834

All *

total_rows

Only for dump-load inputs. The amount of rows sent in the last batch. Used for consolidation.

All *

consolidation

In incremental inputs, this is used to store the replication index value as seen when the event was pulled from the source. Usually a timestamp/number (not to be confused with @timestamp, which reflects the time the event was received by Alooma).

All *

timestamp

The timestamp when the record changed on the source.

All *

log_file

The log file Alooma pulls events from in CDC replication.

All *

log_position

The position in the log file that the event represents.

All *

updated_at

The timestamp when the event was last updated.

Close.io

cio_id

The internal Close.io ID of this event.

Close.io

closeio_object

The name of the Close.io object this event is from.

Close.io

network_id

The DFP network_ID the report is from.

Doubleclick DFP

elasticsearchpull_timestamp

Timestamp where input pulled the data from Elasticsearch.

Elasticsearch

elasticsearch_id

The internal Elasticsearch ID of this event.

Elasticsearch

elasticsearch_index

The Elasticsearch index this event was pulled from.

Elasticsearch

elasticsearch_type

Elasticsearch

account_id

Facebook Ads

pull_time

Facebook Ads

date

Facebook Ads

customer_id

Google Adwords

pull_time

Google Adwords

date

Google Adwords

website_url

Google Analytics

hubspot_id

The internal Hubspot ID of this event.

Hubspot

pulled_at

Hubspot

topic

The Kafka topic this event was consumed from.

Kafka

partition

The Kafka partition this event was consumed from.

Kafka

offset

The offset of this event in the Kafka topic it was consumed from.

Kafka

pull_timestamp

The timestamp when the batch this event was pulled in was consumed from Kafka.

Kafka

produced_ts

The timestamp when the event was produced into the Kafka queue Alooma pulled it from, if available.

Kafka

mixpanel_fetch_date

Mixpanel

batch_ts

Mixpanel

db

Database name.

MongoDB

collection

Collection name.

MongoDB

timestamp

The timestamp when the event reached Alooma.

MongoDB

doc_id

MongoDB internal timestamp.

MongoDB

original_doc_id

MongoDB prior to update/delete.

MongoDB

type

The type of change the event represents (insert, update, delete).

MongoDB

deleted

Whether the row was deleted. e.g. true

MongoDB

ordinal

A secondary replication index used to tell which event happened first when more than 1 event happened at the same timestamp.

MongoDB

log_file

The log file Alooma pulls events from in CDC replication.

MySQL

log_position

The position in the log file that the event represents.

MySQL

timestamp

The server's log timestamp.

MySQL

table

The table associated with the event.

MySQL

schema

The schema associated with the table from the event.

MySQL

type

The type of change the event represents (insert, update, delete).

MySQL

netsuite_object

The name of the Netsuite object this event is from.

Netsuite

netsuite_pull_time

The timestamp when we started to pull this batch.

Netsuite

updated_at

The timestamp when the event was last updated.

Netsuite

netsuite_id

The internal Netsuite ID of this event.

Netsuite

table

The table associated with the event.

ODBC

database

The database associated with the event.

ODBC

schema

The schema associated with the table from the event.

ODBC

table

The table associated with the event.

Oracle

database

The database associated with the event.

Oracle

schema

The schema associated with the table from the event.

Oracle

type

The type of change the event represents (insert, update, delete).

Oracle

deleted

Whether the row was deleted. e.g. true

Oracle

row_id

The row ID for the event.

Oracle

scn

The log position in the Oracle transaction log.

Oracle

timestamp

The timestamp when the event reached Alooma.

Oracle

updated_at

The timestamp when the event was last updated.

Salesforce

salesforce_object

The name of the Salesforce object this event is from.

Salesforce

salesforce_pull_time

The timestamp when we started to pull this batch.

Salesforce

salesforce_id

The internal Salesforce ID of this event.

Salesforce

file_name

The name of the file the event originated from.

Storage

line_number

The line number of this event in the file the event originated from.

Storage

file_modified_ts

The timestamp at which the file this line originated from was last updated.

Storage

updated_at

The timestamp when the event was last updated.

Zendesk

zendesk_id

The internal Zendesk ID of this event.

Zendesk

zuora_object

The name of the Zuora object this event is from.

Zuora

zuora_pull_time

The timestamp when we started to pull this batch.

Zuora

updated_at

The timestamp when it was last updated.

Zuora

zuora_id

The internal Zuora ID of this event.

Zuora

* Note for "All" Input Sources

Some fields listed in the "All" section are not included for every event. Whether these fields are included in the metadata depends on the replication method:

Incremental replication can have consolidation or timestamp.

Only Full dump and load replication will have total_rows, snapshot_size, and snapshot_id.

Log replication can have log_file and log_position.

Example _metadata Dictionaries

Here's an example of the metadata dictionary from an Oracle LogMiner input source:

Copy
"_metadata": {
    "scn": 1234567890,
    "schema": "HR",
    "table": "LOCATIONS",
    "row_id": "AC2d+FAAAAAAARTAAV",
    "deleted": false,
    "type": "INSERT",
    "timestamp": "2018-01-12 16:05:06",
    "event_type": "LOCATIONS",
    "schema_url": "schema?id=31b454ba-b0bc-4520-bd50-236fa0b38c60&schema_object=HR.LOCATIONS",
    "@uuid": "c504f4bc-0ffc-4a1a-84df-6aba382fa651",
    "input_label": "oracle_log_miner",
    "@timestamp": "2018-01-12T16:05:12.948Z",
    "input_type": "ORACLE",
    "input_id": "19a454ba-b0bc-4520-bd50-236fa0b38c60",
    "restream_count": 0,
    "@parent_uuid": ""
  }

And from SQLServer:

Copy
"_metadata": {
    "schema": "dbo",
    "table": "dump_test",
    "event_type": "ssdump_test",
    "database": "testdb",
    "schema_url": "schema?id=730859c4-3f71-48be-9929-e9eea8410b6f&schema_object=dby_dump_test",
    "consolidation": 168001,
    "@uuid": "1035df15-4b5b-4c15-a416-e44118d7d39b",
    "input_label": "sql_server_demo",
    "@timestamp": "2017-12-23T00:27:43.710Z",
    "input_type": "odbc_mssql_incremental_load",
    "input_id": "630859c4-3f71-48be-9929-e9eea8410b6f",
    "restream_count": 3,
    "@parent_uuid": ""
  }

Search results

    No results found