Documentation

API Python Client Reference

Abstract

SummaryA listing of the various methods included in the alooma.py API Python client reference, including a description, any parameters required, and the return value if any.

Here's a listing of the methods included in the API Python client. Remember, you'll need to authenticate before using any of these.

Alter table
Copy
alter_table(table_name, columns)

Alter a table.

Parameters

  • table_name

  • columns

Columns example:

Copy
  columns = [
  {
      'columnName': 'price', 'distKey': False, 'primaryKey': False,
      'sortKeyIndex': -1,
      'columnType': {'type': 'FLOAT', 'nonNull': False}
  }, {
      'columnName': 'event', 'distKey': True, 'primaryKey': False,
      'sortKeyIndex': 0,
      'columnType': {
          'type': 'VARCHAR', 'length': 256, 'nonNull': False
      }
  }
  ]

Return

Returns a dict representation of the table.

Clean restream queue

Purges any events in the Restream Queue, regardless of origin. 

Copy
clean_restream_queue()

Return

None.

Clean system

WARNING: This method is destructive: it removes all inputs, event types, transforms, notifications, and S3 retention.

Copy
clean_system()

Note

This function does not purge the Incoming queue.

Return

None.

Create input
Copy
create_input(input_post_data, one_click=True, validate=True)

Create an input.

Parameters

  • input_post_data: 

  • one_click: (default is True)

  • validate: (default is True)

Return

The ID of the new input.

For more information about working with inputs, see Managing Inputs Programmatically.

Create Mixpanel input
Copy
create_mixpanel_input(mixpanel_api_key, mixpanel_api_secret, from_date,
                      name, transform_id=None, one_click=True)

Create a Mixpanel input using the supplied configuration.

Parameters

  • mixpanel_api_key: 

  • mixpanel_api_secret: 

  • from_date:

  • name:

  • transform_id: (default is none)

  • one_click: (default is True)

Return

The ID of the new input.

Create S3 Input
Copy
create_s3_input(name, key, secret, bucket, prefix='', load_files='all',
                file_format='json', delimiter=',', quote_char='',
                escape_char='', one_click=True)

Creates an S3 input using the supplied configurations.

Parameters

  • name: The designated input name

  • key: a valid AWS access key

  • secret: a valid AWS secret key

  • bucket: The bucket where the data resides

  • prefix: An optional file path prefix. If supplied, only files in paths matching the prefix will be retrieved

  • load_files: Can be either all or new. If new is selected, only files which are created/updated after the input was created will be retrieved. Default is all.

  • file_format: S3 file format. "json", "delimited", "other".

  • delimiter: When choosing file format delimited. Delimiter character (\t for tab)

  • quote_char: When choosing file format delimited. File quote character (optional)

  • escape_char: When choosing file format delimited. Escape character (optional)

Return

A requests.model.Response object representing the result of the create_input call.

Create table
Copy
create_table(table_name, columns)

Create a table with the supplied columns.

Parameters

  • table_name: self descriptive

  • columns: self descriptive

Columns example:

Copy
  columns = [
  {
      'columnName': 'price', 'distKey': False, 'primaryKey': False,
      'sortKeyIndex': -1,
      'columnType': {'type': 'FLOAT', 'nonNull': False}
  }, {
      'columnName': 'event', 'distKey': True, 'primaryKey': False,
      'sortKeyIndex': 0,
      'columnType': {
          'type': 'VARCHAR', 'length': 256, 'nonNull': False
      }
  }
  ]

Return

Returns a dict representation of the table.

For an example of creating a table from a mapping, see Working with Tables.

Delete all event types
Copy
delete_all_event_types()

Delete all of the event types. Scope is limited to the account you specify (if any) during authentication.

Return

None.

For more information about working with event types and mappings, see Managing Mappings Programmatically.

Delete Code Engine module
Copy
delete_code_engine_module(module_name)

Delete the specified module from the Code Engine. 

Parameters

  • module_name: name of the module to delete

Return

None.

For more information about working with Code Engine modules in the API, see Pushing to the Code Engine Programmatically

Delete event type
Copy
delete_event_type(event_type)

Delete a specific event type.

Parameters

  • event_type: event name found in the Mapper

Return

None.

For more information about working with event types and mappings, see Managing Mappings Programmatically.

Delete S3 retention

Deletes the current S3 retention.

Copy
delete_s3_retention()

Return

None.

Delete secret

Deletes the specified secret.

Copy
delete_secret(secret)

Parameters

  • secret: a key of a secret to delete

Example:

Copy
api.set_secrets({"my_user":"example", "my_password": "12345678"})

api.get_secrets() # returns ["my_user", "my_password"]
api.delete_secret("my_user") # deletes "my_user"

api.get_secrets() # returns ["my_password"]

Return

None.

Discard event type
Copy
discard_event_type(event_type)

Discard a specific event type.

Parameters

  • event_type: event name found in the Mapper

Return

None.

For more information about working with event types and mappings, see Managing Mappings Programmatically.

Discard field
Copy
discard_field(mapping, field_path)

Discard the specified field.

Parameters

  • mapping: this is the mapping JSON

  • field_path: use the field_path to find the exact field (see example below)

For example:

  1. mapping == {"a":1, b:{c:3}}

  2. the "c" field_path == a.b.c

Return

New mapping JSON that the last argument would be discarded. For example:

  1. mapping == {"a":1, b:{c:3}}

  2. field_path == "a.b.c"

  3. then the mapping would be the same as the old one except the "c" field would be discarded

For more information about working with event types and mappings, see Managing Mappings Programmatically.

Drop table
Copy
drop_table(schema, table_name, cascade=False)

Drop a table, and optionally cascade to drop the table's dependencies.

Parameters

  • schema: schema in which the table to be dropped is located.

  • table_name: name of the table to drop.

  • cascade: if true, drops all dependencies of the specified table, along with the table itself. Otherwise only drops the table.

Return

None.

Edit input
Copy
edit_input(input_post_data)

Edit an existing input using the supplied data.

Parameters

  • input_post_data: this is a set of parameters specific to the input type. 

Example input_post_data for Postgres with log replication:

Copy
{
  "configuration": {
    "auto_map": "true",
    "hostname": <hostname - string>
    "port": 5432,
    "database": <db-name>,
    "slot_name": <this is optional, the default is alooma_replication_slot>,
    "username": <username>,
    "password": <password>,
    "tables": "<table1> <table2>",
    "query_dump": "true" <can also be false>,
    "session_timeout": 300000 <optional parameter>

  },
  "name": <name of input we're editing>,
  "type": "POSTGRESQL"
}

Return

Returns a dict representation of the input.

For more information about working with inputs, see Managing Inputs Programmatically.

Get all transforms
Copy
get_all_transforms()

Get the list of modules in the Code Engine, and the code for each one.

Return 

A mapping from module name to module code.

Example usage:

Copy
atrans = api.get_all_transforms()

# print each module name and code
for module_name, code in atrans.items():
    print module_name
    print code
Get average event size
Copy
get_average_event_size(minutes)

Get the average size of events processed over the last number of minutes specified.

Parameters

  • minutes: the period of time over which to take the average

Return

Returns the average event size, in bytes, as a float.

Get BigQuery config
Copy
get_bigquery_config()

Returns the output configuration (connection details, etc) for the BigQuery output. The information is pulled from the main output node.

Return

Returns a Python dict with the BigQuery configuration settings. An example:

Copy
{
    "datasetName": "MyBQ",
    "oauth2": " ... ",
    "projectName": "examplecom-112211",
    "cron": "0 */8 * * *",
    "skipValidation": "true"
}
Get BigQuery node
Copy
get_bigquery_node()

Return

Get Code Engine code
Copy
get_code_engine_code()

Get a map of all of the modules and code from the Code Engine.

Return

Returns a Python dictionary containing a map for each module (including main) in the Code Engine, including the name of the module and its code.

Example:

Copy
cec = api.get_code_engine_code()

Returns something like:

Copy
{
    "main": <contents of the 'main' module>
    "submodule": <contents of the 'submodule' module>
    ( ... )
}

For more information about working with Code Engine modules in the API, see Pushing to the Code Engine Programmatically

Get Code Engine module
Copy
get_code_engine_module(module_name='main')

Get the code for the specified module.

Parameters

module_name: name of the module to return, default is 'main'

Return

Returns a string containing the module's code.

Copy
cem = api.get_code_engine_module("submodule")

Returns something like:

Copy
"transform_names = {'John': 'Jonathan',\n   'Bob': 'Robert',\n   'Rick': 'Richard'}\n\ndef transform_ft_to_mtr(x):\n    return x * 0.3048"

For more information about working with Code Engine modules in the API, see Pushing to the Code Engine Programmatically

Get company credits
Copy
get_company_credits()

Returns the credits status for the company:

  • Current period

  • Used credits for the current period

  • Total credits for the current period

Return

A dict representing the company credits status. An example:

Copy
{
    "company_name":"example.com",
    "payment_period":"Annually",
    "start_date":"2018-09-14",
    "subscription_type":"Prepaid",
    "total_credits":170000,
    "used_credits":11851.905976
}

For monthly payment:

Copy
{
    "company_name":"exampleOD.com",
    "payment_period":"Monthly",
    "start_date":"2018-10-17",
    "end_date":"2018-11-16",
    "subscription_type":"On Demand",
    "used_credits":8.217452
}
Get deployment information
Copy
get_deployment_info()

Get information about the current deployment.

Return

A dict containing the deployment information. An example:

Copy
{
    "gitBranch": "origin/master",
    "deploymentName": "<DEPLOYMENTNAME",
    "gitCommitId": "xxxxxxx",
    "gitAuthorEmail": "example@users.noreply.github.com",
    "buildTimestamp": "2018-06-13T20:31:54Z"
}
Get event type
Copy
get_event_type(event_type)

Returns a representation of the requested event type’s mapping and metadata if it exists.

Parameters

  • event_type: The name of the event type

Return

A dict representation of the event type’s data, An example:

Copy
{
    "autoMappingError": null,
    "stats": null,
    "name": "MyEventType",
    "fields": [],
    "origInputLabel": "MyEventTYpe",
    "consolidation": {
        "consolidatedSchema": null,
        "consolidatedTableName": null,
        "viewSchema": null,
        "consolidationKeys": null
    },
    "mapping": {
        "isDiscarded": true,
        "tableName": "",
        "readOnly": false,
        "schema": "public"
    },
    "state": "MAPPED",
    "usingDefaultMappingMode": true,
    "mappingMode": "STRICT",
    "schemaUrls": []
}

For more information about working with event types and mappings, see Managing Mappings Programmatically.

Get event types
Copy
get_event_types()

Returns all the event types in the system.

Return

A list of dicts of all the event types that exist in the system (basically, a list of mapping objects but without the fields section).

For more information about working with event types and mappings, see Managing Mappings Programmatically.

Get incoming events count
Copy
get_incoming_events_count(minutes)

Get the number of incoming events for the number of specified minutes.

Parameters

  • minutes: int, number of minutes

Return

Number of events. Example: 55960.0

Get incoming queue metric
Copy
get_incoming_queue_metric(minutes)

Gets the average number events in the incoming queue over the previous number of minutes specified. 

Parameters

  • minutes: 

Return

Returns the value of the "EVENTS_IN_PIPELINE" metric for the number of minutes specified, An example usage:

Copy
iqm = api.get_incoming_queue_metric(100) # 100 minutes
print iqm
Get input by id
Copy
get_input_by_id(input_id)

Gets an input based on the specified ID.

Parameters

  • input_id: the ID for the input. If you do not have the ID or the name for an input, you can use get_inputs() to get a list of all inputs in the instance, and get the ID or name from that.

Return

Returns an input object (a Python dictionary containing "job" and "task" information). For more information about input objects, see Managing Inputs Programmatically.

An example:

Copy
api = alooma.Client(api_key=apikey, account_name=account)

INPUT_ID = "aae13-bd1221-123123"
inputcfg = api.get_input_by_id(INPUT_ID)

state = inputcfg["tasks"].values()[0]

print state['type'] # shows the type of input (MYSQL, ORACLE, etc., etc.)
Get input by name
Copy
get_input_by_name(module_name)

Get an input based on the specified name.

Parameters

  • module_name: the name of the input. If you do not have the ID or the name for an input, you can use get_inputs() to get a list of all inputs in the instance, and get the ID or name from that.

Return

Returns an input object (a Python dictionary containing "job" and "task" dictionaries). For more information about input objects, see Managing Inputs Programmatically.

An example Salesforce input configuration (configurations are similar between different input types, with most differences limited to the input_state information):

Copy
{
    "job": {
        "validated": true,
        "name": "SFDC_test",
        "created_at": "2018-10-03T18:22:26.400436",
        "configuration": {
            "oauth2": "yJyY==",
            "custom_objects": "Account Contact Lead Opportunity ActionLinkGroupTemplate",
            "auto_map": "true",
            "is_sandbox": true,
            "input_default_schema": "PUBLIC",
            "daily_bulk_queries": 10000,
            "daily_api_calls": 200000
        },
        "type": "SALESFORCE",
        "id": "a3ef"
    },
    "tasks": {
        "42bcb": {
            "category": "inputs",
            "status": "active",
            "error_stack": "",
            "name": "SFDC_test",
            "deployment_name": "alooma-docs-1",
            "error_message": "",
            "next_processing_time": "2018-12-11T21:23:19.333634",
            "last_success_time": "2018-10-09T11:26:06.016097",
            "error_count": 1876,
            "state_id": "42bcb",
            "module_name": "inputs.pullers.salesforce.salesforce",
            "last_processing_time": "2018-12-11T20:23:19.397145",
            "task_state": {
                "batch_id": "56dee",
                "received_events": 0,
                "batch_offset": 0,
                "input_state": {
                    "org_creation_date": "2015-07-13T08:51:19.000+0000",
                    "trim_to_first_event": false,
                    "sobject": "ActionLinkGroupTemplate",
                    "last_existing_ts": "2011-11-30T00:00:00+00:00",
                    "api": null,
                    "batch_range": 8164800.0,
                    "last_deleted_ts": "1969-12-31T00:00:00+00:00"
                }
            },
            "type": "SALESFORCE",
            "id": "a3ef54"
        },
        ( ...)
    }
}

An example usage:

Copy
api = alooma.Client(api_key=apikey, account_name=account)

INPUT_LABEL = "My_Input"
inputcfg = api.get_input_by_name(INPUT_LABEL)

state = inputcfg["tasks"].values()[0]

print state['type'] # shows the type of input (MYSQL, ORACLE, etc., etc.)
Get input sleep time
Copy
get_input_sleep_time(input_id)

Gets the sleep time for the specified input. Sleep time is basically the interval between events injected into the incoming queue.

Parameters

  • input_id: ID of the input whose sleep time to return

Return

Sleep time of the input with ID input_id.

Get inputs
Copy
get_inputs(name=None, input_type=None, input_id=None)

Get a list of all the input nodes in the system.

Parameters

  • name: Filter by name (accepts Regex)

  • input_type: Filter by type (e.g. "mysql")

  • input_id: Filter by node ID

Return

A list of all the inputs in the system, along with metadata and configurations. An example of a single input:

Copy
{
    "category": "INPUT",
    "stats": {
        "latency": 0,
        "throughput": 0
    },
    "name": "Salesforce",
    "deleted": false,
    "state": {
        "autoMapping": {
            "inputLabel": "Salesforce",
            "state": null,
            "enabled": "false"
        }
    },
    "configuration": {
        "username": "<USERNAME>",
        "start_time": "2016-09-06T00:00:00.000Z",
        "auto_map": "false",
        "custom_objects": "Lead Opportunity Contact Account User Task Event Campaign OpportunityHistory",
        "daily_bulk_queries": 1000,
        "daily_api_calls": 7000
    },
    "type": "SALESFORCE",
    "id": "<ID>"
}

An example usage:

Copy
ainputs = api.get_inputs()

act = str(len(ainputs))
print "\nInputs : " + act
for inp in ainputs:
    print "Name : " + inp['name'] + " (" + inp['type'] + ")"

For more information about working with inputs, see Managing Inputs Programmatically.

Get mapping
Copy
get_mapping(event_type)

Returns a dict representation of the mapping of the event type requested, if it exists.

Parameters

  • event_type: The name of the event type

Return

The full mapping (dict) with all fields.

For more information about working with event types and mappings, see Managing Mappings Programmatically.

Get mapping mode
Copy
get_mapping_mode()

Returns the default mapping mode currently set in the system. The mapping mode should be one of the values in alooma.MAPPING_MODES.

Return

The default mapping mode currently set in the system.

Get maximum latency
Copy
get_max_latency(minutes)

Gets the maximum latency during the specified number of minutes.

Parameters

  • minutes: 

Return

Float, latency. Example: 309.576

Get metrics by names
Copy
get_metrics_by_names(metric_names, minutes)

Get the specified system metric(s).

Parameters

  • metric_names: One or more of the metrics listed in "METRICS_LIST" in alooma.py. The list includes:

    • EVENT_SIZE_AVG

    • EVENT_SIZE_TOTAL

    • EVENT_PROCESSING_RATE

    • INCOMING_EVENTS

    • RESTREAMED_EVENTS

    • UNMAPPED_EVENTS

    • IGNORED_EVENTS

    • ERROR_EVENTS

    • LOADED_EVENTS_RATE

    • LATENCY_AVG

    • LATENCY_PERCENTILE_50

    • LATENCY_PERCENTILE_95

    • LATENCY_MAX

    • EVENTS_IN_PIPELINE

    • EVENTS_IN_TRANSIT

  • minutes: the number of minutes

Return

Returns a dict containing the target (metric name) and data points (value, timestamp) for each metric specified. An example: 

Copy
[
    {
        "target": "EVENT_SIZE_AVG",
        "datapoints": [
            [
                22984.973548689137,
                1529073369
            ]
        ]
    }
]

Example usage:

Copy
from alooma import METRICS_LIST
for metric in METRICS_LIST:
    mbn = api.get_metrics_by_names(metric, 100)
    print json.dumps(mbn, indent=4)
Get notifications
Copy
get_notifications(epoch_time)

Get the notifications in the system based on the time provided and moving forward.

Parameters

  • epoch_time: the time (in seconds, from epoch) of the oldest notifications to retrieve

Return

Returns a nested dict object, with a single key: messages. This can be useful when, for example, troubleshooting problem events. Also, get_notifications() can return a much longer list of notifications than what is displayed in the UI. Other use cases might include using these errors to programmatically clear events stuck in the Restream Queue, or to trigger other tasks based on issues. Example usage:

Copy
data=api.get_notifications(time)

Inside data['messages'] is a list of dict objects, where each dict represents a single notification from the Notifications pane on the Alooma Dashboard. For example, a full response might look like:

Copy
{'messages':
 [
  {'description': None,
   'type': 'METRICS_OUTPUT_EVENTS_LOADED',
   'timestamp': 1531288180667,
   'params': ['365791', 'Redshift'],
   'state': 'NEW',
   'aggregatedCount': 1,
   'message': '365,791 events were loaded to Redshift',
   'typeDescription': 'Events Loaded to Output',
   'severity': 'info',
   'id': 'alooma-notifications+0+1339'},
  {'description': None,
   'type': 'METRICS_OUTPUT_EVENTS_LOADED',
   'timestamp': 1531284580669,
   'params': ['352047', 'Redshift'],
   'state': 'NEW',
   'aggregatedCount': 1,
   'message': '352,047 events were loaded to Redshift',
   'typeDescription': 'Events Loaded to Output',
   'severity': 'info',
   'id': 'alooma-notifications+0+1338'},
  {'description': None,
   'type': 'METRICS_OUTPUT_EVENTS_LOADED',
   'timestamp': 1531280980688,
   'params': ['358313', 'Redshift'],
   'state': 'NEW',
   'aggregatedCount': 1,
   'message': '358,313 events were loaded to Redshift',
   'typeDescription': 'Events Loaded to Output',
   'severity': 'info',
   'id': 'alooma-notifications+0+1337'}
]}
Get output config
Copy
get_output_config()

Gets the configuration of the current output. 

Return

Returns a dict representation of the output configuration.

Get output node
Copy
get_output_node()

Return

A dict containing the output node configuration. An example:

Copy
{
    "configuration": {
        "username": "<USERNAME>",
        "skipValidation": "false",
        "hostname": "<HOSTNAME>",
        "sinkType": "REDSHIFT",
        "databaseName": "<DATABASENAME>",
        "schemaName": "public",
        "password": "**********",
        "port": "1234"
    },
    "type": "REDSHIFT",
    "id": "<ID>",
    "name": "Redshift"
}

Example usage:

Copy
on = api.get_output_node()

print "Output name  : " + on['name']
print "Database name: " + on['configuration']['databaseName']
Get outputs metrics
Copy
get_outputs_metrics(minutes)

Returns the number of events erred / unmapped / discarded / loaded in the last X minutes.

Parameters

  • minutes: number of minutes to check

Return

A group of four numbers. An example:

Copy
(0, 20656.0, 15.0, 518333.0)
Get Redshift config
Copy
get_redshift_config()

Get the Redshift configuration.

Return

Returns a dict with the Redshift configuration settings/values. An example: 

Copy
{
    "username": "<USERNAME>",
    "skipValidation": "false",
    "hostname": "<HOSTNAME>",
    "databaseName": "<DATABASE>",
    "schemaName": "public",
    "password": "**********",
    "port": "1234"
}

Printing the username:

Copy
red = api.get_redshift_config()
print "Username : " + red['username']
Get Redshift node
Copy
get_redshift_node()

Get the Redshift node information.

Return

Returns a dict containing the node information. An example:

Copy
{
    "category": "OUTPUT",
    "stats": {
        "latency": 9052,
        "throughput": 69.9
    },
    "name": "Redshift",
    "deleted": false,
    "state": {
        "additionalInfo": null,
        "state": "CONNECTION_SUCCESSFUL",
        "message": "Connection Successful"
    },
    "configuration": {
        "username": "<USERNAME>",
        "skipValidation": "false",
        "hostname": "<HOSTNAME>",
        "databaseName": "<DATABASE>",
        "schemaName": "public",
        "password": "**********",
        "port": "1234"
    },
    "type": "REDSHIFT",
    "id": "<>"
}
Get restream queue metrics
Copy
get_restream_queue_metrics(minutes)

The number of events sent to the Restream Queue in the last specified number minutes (minutes parameter). This is the number shown in the Restream Queue graph in the Alooma Dashboard.

Parameters

  • minutes: 

Return

Returns a float.

Get restream queue size
Copy
get_restream_queue_size()

Returns the number of events currently held in the Restream Queue.

Return

An int representing the number of events in the queue.

Get restream stats
Copy
get_restream_stats()

Get statistics for the Restream Queue.

Return

Type dict with the following keys; number_of_events, size_used, max_size.

  • number_of_events: number of available events to restream

  • size_used: restream used size in bytes

  • max_size: maximum restream size in bytes

Get samples
Copy
get_samples(event_type=None, error_codes=None)

Get sample events.

Parameters

  • event_type: optional string containing an event type name

  • error_codes: optional list of strings containing event status codes. status codes may be any string returned by get_sample_status_codes()

Return

A list of 10 samples. if event_type is passed, only samples of that event type will be returned. if error_codes is given only samples of those status codes are returned.

Get samples stats
Copy
get_samples_stats()

Get statistics for sample events returned.

Return

A dictionary where the keys are names of event types, and each value is another dictionary which maps from status code to the amount of samples for that event type and status.

Get samples status codes
Copy
get_samples_status_codes()

Return

A list of status codes each event in Alooma may be tagged with in the form of a dictionary of status types and their human-readable descriptions. As Alooma supports more processing capabilities, status codes may be added. These status codes are used for sampling events according to the events' type and status.

Copy
 {
   "VALID": "Valid",
   "NO_MAPPING": "Valid",
   "NEW_TYPE": "Valid",
   "DROP": "Valid",
   "NO_MAPPED_FIELDS": "No mapped fields",
   ...
 }
Get scheduled queries
Copy
get_scheduled_queries()

Returns all scheduled queries.

Return

A dict of dicts representing all scheduled queries:

Copy
"1234": {
  "status": "active",
  "error_stack": "",
  "deployment_name": "deployment1",
  "error_message": "",
  "next_processing_time": "2018-06-08T21:57:40.158350",
  "last_success_time": "2018-06-08T13:57:40.158039",
  "error_count": 0,
  "state_id": "1234",
  "module_name": "module1",
  "consolidation_query": "<QUERY HERE>",
  "configuration": {
    "custom_query": "select 1;",
    "event_type": "<EVENT TYPE HERE>",
    "deployment_name": "alooma-1",
    "query_type": "custom",
    "run_at": null,
    "frequency": 8
  },
  "last_processing_time": "2018-06-08T13:57:40.158350"
"1235": {
  [ ... ]
  }

Example usage:

Copy
schqs = api.get_scheduled_queries() # returns dict of dicts

for query_id, data in schqs.items():
    if data['status'] == "active":
        print(query_id) # show IDs of all active queries

For more information about working with scheduled queries, see Managing Custom Scheduled Queries.

Get scheduled queries in error state
Copy
get_scheduled_queries_in_error_state()

Returns all scheduled queries that did not run successfully on the last attempt.

Return

A dict representing all failed scheduled queries.

Get schemas
Copy
get_schemas()

Get the schemas in the configured output.

Return

Returns a list of all the schemas of the configured output. An example:

Copy
[ 
   {
        "default": true,
        "schemaName": "PUBLIC"
    },
    {
        "default": false,
        "schemaName": "TESTSCHEMA"
    }
]
Get secrets
Copy
get_secrets()

Returns a list of existing secrets.

Example:

Copy
api.set_secrets({"my_user":"example", "my_password": "12345678"}

api.get_secrets() # returns ["my_user", "my_password"]
api.delete_secret("my_user") # deletes "my_user"

api.get_secrets() # returns ["my_password"]

Return

A Python list of secrets.

Get settings
Copy
get_settings()

Retrieves the general settings for different features in the UI. Currently it returns data about Digest Email and S3 Retention features.

Return

Returns a dict object. An example:

Copy
{
    "email-notifications": {
        "digestWarning": true,
        "digestSendHour": 20,
        "recipients": [
            "user@example.com"
        ],
        "digestInfo": true,
        "digestError": true,
        "digestFrequency": "DAILY",
        "recipientsChanged": true,
        "digestEmailsTimeZone": "PST",
        "notificationsInDigest": 20
    },
    "s3-retention": {
        "awsAccessKeyId": "AKIAJL2I123456789012",
        "filePrefix": "with_metadata",
        "awsSecretAccessKey": "0123456789Puel0J9uQjT4SLbyqQoK/abcdefghi",
        "awsBucketName": "alooma",
        "saveMetadata": true,
        "serverSideEncryption": true,
        "gzip": true
    }
}
Get Snowflake config
Copy
get_snowflake_config()

Get the Snowflake configuration.

Return

Returns a dict with the Snowflake configuration settings/values. An example:

Copy
{
    "username": "<USERNAME>",
    "warehouseName": "<WAREHOUSE NAME>",
    "skipValidation": "false",
    "accountName": "<ACCOUNT NAME>",
    "cron": "*/15 * * * *",
    "databaseName": "<DATABASE NAME>",
    "schemaName": "PUBLIC",
    "password": "**********"
}

Printing the username:

Copy
snow = api.get_snowflake_config()
print "Username : " + snow['username']
Get Snowflake node
Copy
get_snowflake_node()

Get the Snowflake node information.

Return

A dict containing the Snowflake node configuration. An example:

Copy
{
    "category": "OUTPUT",
    "stats": {
        "latency": 5091,
        "throughput": 0
    },
    "name": "Snowflake",
    "deleted": false,
    "state": {
        "additionalInfo": null,
        "state": "CONNECTION_SUCCESSFUL",
        "message": "Connection Successful"
    },
    "configuration": {
        "username": "<USERNAME>",
        "warehouseName": "<WAREHOUSE NAME>",
        "skipValidation": "false",
        "accountName": "<ACCOUNT NAME>",
        "databaseName": "<DATABASE NAME>",
        "schemaName": "PUBLIC",
        "password": "**********"
    },
    "type": "SNOWFLAKE",
    "id": "<ID>"
}
Get structure
Copy
get_structure()

Returns a representation of all the inputs, outputs, and on-stream processors currently configured in the system.

Return

A dict representing the structure of the system.

Get table dependencies
Copy
get_table_dependencies(schema, table_name)

Get a list of dependencies for a specific table.

Parameters

  • schema: the schema in which the table is located.

  • table_name: name of the table.

Return

Returns a list of strings.

Get table names
Copy
get_table_names(schema=None)

Parameters

  • schema: return tables from a specific schema, else use default

Return

Returns a list of dictionaries, each containing 4 fields: columns (always empty), retention, schema, and table name.

Copy
[ 
   {
        "retention": null,
        "tableName": "TABLE1",
        "columns": [],
        "schema": "PUBLIC"
    }
]
Get tables
Copy
get_tables(shallow=False, schema=None)

Parameters

  • shallow: only return schema and table names

  • schema: return tables from a specific schema, else use default

Return

A list object, containing entries for each table. An example of a single table with shallow=False:

Copy
    {
        "retention": null,
        "tableName": "user",
        "columns": [
            {
                "columnType": {
                    "length": 256,
                    "type": "VARCHAR",
                    "nonNull": false,
                    "truncate": false
                },
                "subColumnsMap": null,
                "columnName": "a",
                "distKey": false,
                "sortKeyIndex": -1,
                "subColumns": null,
                "primaryKey": false
            }
        ],
        "schema": "public"
    }

The same table, with shallow=True:

Copy
    {
        "retention": null,
        "tableName": "user",
        "columns": [],
        "schema": "public"
    }
Get throughput by name
Copy
get_throughput_by_name(name)

Parameters

  • name: the name of each node, i.e. Inputs, Code Engine, Restream, Mapper, and Output

Return

The throughput for the named node.

Example usage:

Copy
tp = api.get_throughput_by_name("Mapper")
print tp
Get transform node id
Copy
get_transform_node_id()

Return

The ID for the transform node.

Example usage:

Copy
ni = api.get_transform_node_id()
print ni
Get used credits
Copy
get_used_credits(from_day=None, to_day=None, all_accounts=False)

Get the credits consumption per day for the specified period for the whole company or for the login account. The current day used credits can change according to real use.

Parameters

  • from_day: 'YYYY-MM-DD' from date period. If None, no from limit is set.

  • to_day: 'YYYY-MM-DD' to date period. If None, use today as day.

  • all_accounts: if true, return the used credits for all accounts for the company.

Return

Returns a list of dicts. An example:

Copy
[
    {
        "date":"2018-10-24",
        "instance_name":"example.com",
        "value":3.0
    },
    {
        "date":"2018-10-25",
        "instance_name":"example.com",
        "value":10.4
    },
    {
        "date":"2018-10-26",
        "instance_name":"example.com",
        "value":0.0
    },
    {
        "date":"2018-10-27",
        "instance_name":"example.com",
        "value":0.0
    },
    {
        "date":"2018-10-28",
        "instance_name":"example.com",
        "value":4.0
    }
]
Get users
Copy
get_users()

Returns a list of all the users that have access to the account:

Copy
[{'email': 'user@example.com',
  'first_name': None,
  'last_name': None,
  'isPending': False},
 {'email': 'user2@example.com',
  'first_name': None,
  'last_name': None,
  'isPending': False}]

Return

List of all users with access to the account.

Example usage:

Copy
ausers = api.get_users()

for user in ausers:
    print "email : " + user['email']
Purge restream queue
Copy
purge_restream_queue()

Return

A standard HTTP response code on success or fail.

Remove all inputs
Copy
remove_all_inputs()

Return

A standard HTTP response code on success or fail.

Remove input
Copy
remove_input(self, input_id)

Parameters

  • input_id: the id for a given input

For example: This can be found in the url in the input settings.

Return

A standard HTTP response code on success or fail.

For more information about working with inputs, see Managing Inputs Programmatically.

Remove scheduled query
Copy
remove_scheduled_query(query_id)

Parameters

  • query_id: the id of the query to remove. See Get scheduled queries for an example of how to retrieve query ids.

Return

A standard HTTP response code on success or fail.

Schedule query
Copy
schedule_query(event_type, query, frequency=None, run_at=None)

Return requests response to create a query.

Parameters

  • event_type: Alooma Event Type Related to Query (exiting event type is required, even if just a placeholder)

  • query: Desired Query to Schedule

  • frequency: Desired hours between query runs

  • run_at: Cron like string to run query on

Note

You must select either frequency or run_at

For more information about working with scheduled queries, see Managing Custom Scheduled Queries.

Return

Return requests response to create a query.

Set BigQuery configuration
Copy
set_bigquery_config(project_name, dataset_name, skip_validation=False)

Set BigQuery configuration.

Parameters

  • BigQuery schema

  • database_name: BigQuery database name

  • skip_validation: :type bool: True skips configuration validation

Return

Python dict containing the response’s content.

Set Code Engine code
Copy
set_code_engine_code(modules)

Define modules in the Code Engine. The default is the "main" module.

Parameters

  • modules: a Python map of module name (string) to code (string)

Return

None.

Example:

Copy
# Define and submit a code module named 'submodule'.
#   this module contains a dict called 'transform_names'
#   and a method called 'transform_ft_to_mtr'

# Create the map
my_submodule = {"submodule" : """transform_names = {'John': 'Jonathan',
                   'Bob': 'Robert',
                   'Rick': 'Richard'}

def transform_ft_to_mtr(x):
    return x * 0.3048"""}

api.set_code_engine_code(my_submodule)

For more information about working with Code Engine modules in the API, see Pushing to the Code Engine Programmatically

Set input sleep time
Copy
set_input_sleep_time(input_id, sleep_time)

Parameters

  • input_id: ID of the input whose sleep time to change

  • sleep_time: new sleep time to set for input with ID input_id

Return

Result of the REST request.

Set mapping
Copy
set_mapping(mapping, event_type, timeout=300)

Parameters

  • mapping: 

  • event__type

  • timeout: (default is 300)

Return

For more information about working with event types and mappings, see Managing Mappings Programmatically.

Set mapping mode
Copy
set_mapping_mode(mode)

Sets the default mapping mode in the system. 

Parameters

  • mode: one of the values in alooma.MAPPING_MODES

Return

Set output
Copy
set_output(output_config, output_name=None)

Set Output configuration.

Parameters

  • output_config: :type dict. Output configuration. Should contain output-specific configurations (see examples below) and the following parameters:

  • skip_validation: (type bool) True skips output configuration validation

  • sink_type: Output type. REDSHIFT, BIGQUERY or SNOWFLAKE

Redshift example:

Copy
{"hostname":"redshift-host", "port":5439, "schemaName":"public",
"databaseName":"some_db", "username":"user", "password":"password",
"skipValidation":false, "sinkType":"REDSHIFT"}

Snowflake example:

Copy
{"username":"user", "password":"password", "accountName":"some-account",
 "warehouseName":"SOME_WH", "databaseName":"SOME_DB", "schemaName":"PUBLIC",
 "skipValidation":"false", "sinkType":"SNOWFLAKE"}

BigQuery example:

Copy
{"projectName":"some-project", "datasetName":"some-dataset",
               "skipValidation":"false", "sinkType":"BIGQUERY"}

  • output_name: UI display name

Return

Python dict containing the response’s content.

Set Redshift config
Copy
set_redshift_config(hostname, port, schema_name, database_name,
                    username, password, skip_validation=False, ssh_server=None,
                    ssh_port=None, ssh_username=None, ssh_password=None)

Set Redshift configuration.

Parameters

  • hostname: Redshift hostname

  • port: Redshift port

  • schema_name: Redshift schema

  • database_name: Redshift database name

  • username: Redshift username

  • password: Redshift password

  • skip_validation: :type bool: True skips configuration validation

  • ssh_server: (Optional) The IP or DNS of your SSH server as seen from the public internet

  • ssh_port: (Optional) The SSH port of the SSH server as seen from the public internet (default port is 22)

  • ssh_username: (Optional) The user name on the SSH server for the SSH connection (the standard is alooma)

  • ssh_password: (Optional) The password that matches the user name on the SSH server

Return

Python dict containing the response’s content.

Set S3 retention
Copy
set_s3_retention(aws_bucket_name, aws_access_key, aws_secret_key,
                 file_prefix=None, save_metadata=True, gzip=True,
                 server_side_encryption=True)

Configure s3 retention which stores the raw events received by Alooma to a custom S3 bucket. Retention files are stored in folders by input name and timestamp.

Parameters

  • aws_bucket_name: where retention files will be created

  • aws_access_key: with permissions to list and write files

  • aws_secret_key: to go with the aws_access_key

  • file_prefix: (optional) files will upload to s3://bucket/prefix/

  • save_metadata: save raw events with their metadata (default=True)

  • gzip: create gzipped files (default=True)

  • server_side_encryption: use S3 encryption (default=True)

Return

A standard HTTP response code on success or fail.

Set secrets
Copy
set_secrets(secrets)

Create or update secrets.

Parameters

  • secrets: dictionary of secrets to create/update

Example:

Copy
api.set_secrets({"my_user":"example", "my_password": "12345678"}

api.get_secrets() # returns ["my_user", "my_password"]
api.delete_secret("my_user") # deletes "my_user"

api.get_secrets() # returns ["my_password"]

Return

None.

Set settings email notifications
Copy
set_settings_email_notifications(email_settings_json)

Parameters

  • email_settings_json: here are the default email settings:

    Copy
    DEFAULT_SETTINGS_EMAIL_NOTIFICATIONS = {
    "digestInfo": True,
    "digestWarning": True,
    "digestError": True,
    "digestFrequency": "DAILY",
    "recipientsChanged": False,
    "recipients": []
}

Return

A standard HTTP response code on success or fail.

Set Snowflake config
Copy
set_snowflake_config(account_name, warehouse_name, schema_name,
                     database_name, username, password, skip_validation=False)

Set Snowflake configuration.

Parameters

  • account_name: Snowflake account name

  • warehouse_name: Snowflake warehouse name

  • schema_name: Snowflake schema

  • database_name: Snowflake database name

  • username: Snowflake username

  • password: Snowflake password

  • skip_validation: :type bool: True skips configuration validation

Return Type dict. Response’s content.

Set transform to default
Copy
set_transform_to_default()

Sets the Code Engine Python code to the default, which is a blank transform that makes no changes to any events.

Return

A standard HTTP response code on success or fail.

Start restream
Copy
start_restream()

Starts a Restream, streaming data from the Restream Queue to the pipeline for processing.

Return

A standard HTTP response code on success or fail.

Test transform
Copy
test_transform(sample, temp_transform=None)

Runs the current deployed transform — or one that is provided to the method — against the specified sample event.

Parameters

  • sample: a json string or a dict, representing a sample event

  • temp_transform: optional string containing transform code. if not provided, the currently deployed transform will be used.

Return

The results of a test run of the temp_transform on the given sample. This returns a dictionary with the following keys:

  • output - strings printed by the transform function

  • result - the resulting event

  • runtime - milliseconds it took the function to run

Test transform all samples
Copy
test_transform_all_samples(event_type=None, status_code=None)

Test many samples on the current transform at once.

Parameters

  • event_type: optional string containing event type name

  • status_code: optional status code string

Return

A list of samples (filtered by the event type and status code if provided), for each sample, a result key is added which includes the result of the current transform function after it was run with the sample.

Unmap field
Copy
unmap_field(mapping, field_path)

Parameters

  • mapping: this is the mapping json

  • field_path: use the field_path to find the exact field

For example:

  1. mapping == {"a":1, b:{c:3}}

  2. the "c" field_path == a.b.c

Return

New mapping JSON from which the last argument would be removed.

For example:

  1. mapping == {"a":1, b:{c:3}}

  2. field_path == "a.b.c"

  3. then the mapping would be the same as the old mapping, but the "c" field would be removed -> {"a":1, b:{}}

In this section:

Search results

    No results found