API Python Client Reference
SummaryA listing of the various methods included in the alooma.py API Python client reference, including a description, any parameters required, and the return value if any.
Here's a listing of the methods included in the API Python client. Remember, you'll need to authenticate before using any of these.
Alter table
alter_table(table_name, columns)
Alter a table.
Parameters
table_name
columns
Columns example:
columns = [ { 'columnName': 'price', 'distKey': False, 'primaryKey': False, 'sortKeyIndex': -1, 'columnType': {'type': 'FLOAT', 'nonNull': False} }, { 'columnName': 'event', 'distKey': True, 'primaryKey': False, 'sortKeyIndex': 0, 'columnType': { 'type': 'VARCHAR', 'length': 256, 'nonNull': False } } ]
Return
Returns a dict representation of the table.
Clean restream queue
Purges any events in the Restream Queue, regardless of origin.
clean_restream_queue()
Return
None.
Clean system
WARNING: This method is destructive: it removes all inputs, event types, transforms, notifications, and S3 retention.
clean_system()
Note
This function does not purge the Incoming queue.
Return
None.
Create input
create_input(input_post_data, one_click=True, validate=True)
Create an input.
Parameters
input_post_data:
one_click: (default is True)
validate: (default is True)
Return
The ID of the new input.
For more information about working with inputs, see Managing Inputs Programmatically.
Create Mixpanel input
create_mixpanel_input(mixpanel_api_key, mixpanel_api_secret, from_date, name, transform_id=None, one_click=True)
Create a Mixpanel input using the supplied configuration.
Parameters
mixpanel_api_key:
mixpanel_api_secret:
from_date:
name:
transform_id: (default is none)
one_click: (default is True)
Return
The ID of the new input.
Create S3 Input
create_s3_input(name, key, secret, bucket, prefix='', load_files='all', file_format='json', delimiter=',', quote_char='', escape_char='', one_click=True)
Creates an S3 input using the supplied configurations.
Parameters
name: The designated input name
key: a valid AWS access key
secret: a valid AWS secret key
bucket: The bucket where the data resides
prefix: An optional file path prefix. If supplied, only files in paths matching the prefix will be retrieved
load_files: Can be either all or new. If new is selected, only files which are created/updated after the input was created will be retrieved. Default is all.
file_format: S3 file format. "json", "delimited", "other".
delimiter: When choosing file format delimited. Delimiter character (\t for tab)
quote_char: When choosing file format delimited. File quote character (optional)
escape_char: When choosing file format delimited. Escape character (optional)
Return
A requests.model.Response object representing the result of the create_input
call.
Create table
create_table(table_name, columns)
Create a table with the supplied columns.
Parameters
table_name: self descriptive
columns: self descriptive
Columns example:
columns = [ { 'columnName': 'price', 'distKey': False, 'primaryKey': False, 'sortKeyIndex': -1, 'columnType': {'type': 'FLOAT', 'nonNull': False} }, { 'columnName': 'event', 'distKey': True, 'primaryKey': False, 'sortKeyIndex': 0, 'columnType': { 'type': 'VARCHAR', 'length': 256, 'nonNull': False } } ]
Return
Returns a dict representation of the table.
For an example of creating a table from a mapping, see Working with Tables.
Delete all event types
delete_all_event_types()
Delete all of the event types. Scope is limited to the account you specify (if any) during authentication.
Return
None.
For more information about working with event types and mappings, see Managing Mappings Programmatically.
Delete Code Engine module
delete_code_engine_module(module_name)
Delete the specified module from the Code Engine.
Parameters
module_name: name of the module to delete
Return
None.
For more information about working with Code Engine modules in the API, see Pushing to the Code Engine Programmatically
Delete event type
delete_event_type(event_type)
Delete a specific event type.
Parameters
event_type: event name found in the Mapper
Return
None.
For more information about working with event types and mappings, see Managing Mappings Programmatically.
Delete S3 retention
Deletes the current S3 retention.
delete_s3_retention()
Return
None.
Delete secret
Deletes the specified secret.
delete_secret(secret)
Parameters
secret: a key of a secret to delete
Example:
api.set_secrets({"my_user":"example", "my_password": "12345678"}) api.get_secrets() # returns ["my_user", "my_password"] api.delete_secret("my_user") # deletes "my_user" api.get_secrets() # returns ["my_password"]
Return
None.
Discard event type
discard_event_type(event_type)
Discard a specific event type.
Parameters
event_type: event name found in the Mapper
Return
None.
For more information about working with event types and mappings, see Managing Mappings Programmatically.
Discard field
discard_field(mapping, field_path)
Discard the specified field.
Parameters
mapping: this is the mapping JSON
field_path: use the field_path to find the exact field (see example below)
For example:
mapping == {"a":1, b:{c:3}}
the "c" field_path == a.b.c
Return
New mapping JSON that the last argument would be discarded. For example:
mapping == {"a":1, b:{c:3}}
field_path == "a.b.c"
then the mapping would be the same as the old one except the "c" field would be discarded
For more information about working with event types and mappings, see Managing Mappings Programmatically.
Drop table
drop_table(schema, table_name, cascade=False)
Drop a table, and optionally cascade to drop the table's dependencies.
Parameters
schema: schema in which the table to be dropped is located.
table_name: name of the table to drop.
cascade: if true, drops all dependencies of the specified table, along with the table itself. Otherwise only drops the table.
Return
None.
Edit input
edit_input(input_post_data)
Edit an existing input using the supplied data.
Parameters
input_post_data: this is a set of parameters specific to the input type.
Example input_post_data for Postgres with log replication:
{ "configuration": { "auto_map": "true", "hostname": <hostname - string> "port": 5432, "database": <db-name>, "slot_name": <this is optional, the default is alooma_replication_slot>, "username": <username>, "password": <password>, "tables": "<table1> <table2>", "query_dump": "true" <can also be false>, "session_timeout": 300000 <optional parameter> }, "name": <name of input we're editing>, "type": "POSTGRESQL" }
Return
Returns a dict representation of the input.
For more information about working with inputs, see Managing Inputs Programmatically.
Get all transforms
get_all_transforms()
Get the list of modules in the Code Engine, and the code for each one.
Return
A mapping from module name to module code.
Example usage:
atrans = api.get_all_transforms() # print each module name and code for module_name, code in atrans.items(): print module_name print code
Get average event size
get_average_event_size(minutes)
Get the average size of events processed over the last number of minutes specified.
Parameters
minutes: the period of time over which to take the average
Return
Returns the average event size, in bytes, as a float.
Get BigQuery config
get_bigquery_config()
Returns the output configuration (connection details, etc) for the BigQuery output. The information is pulled from the main output node.
Return
Returns a Python dict with the BigQuery configuration settings. An example:
{ "datasetName": "MyBQ", "oauth2": " ... ", "projectName": "examplecom-112211", "cron": "0 */8 * * *", "skipValidation": "true" }
Get BigQuery node
get_bigquery_node()
Return
Get Code Engine code
get_code_engine_code()
Get a map of all of the modules and code from the Code Engine.
Return
Returns a Python dictionary containing a map for each module (including main) in the Code Engine, including the name of the module and its code.
Example:
cec = api.get_code_engine_code()
Returns something like:
{ "main": <contents of the 'main' module> "submodule": <contents of the 'submodule' module> ( ... ) }
For more information about working with Code Engine modules in the API, see Pushing to the Code Engine Programmatically
Get Code Engine module
get_code_engine_module(module_name='main')
Get the code for the specified module.
Parameters
module_name: name of the module to return, default is 'main'
Return
Returns a string containing the module's code.
cem = api.get_code_engine_module("submodule")
Returns something like:
"transform_names = {'John': 'Jonathan',\n 'Bob': 'Robert',\n 'Rick': 'Richard'}\n\ndef transform_ft_to_mtr(x):\n return x * 0.3048"
For more information about working with Code Engine modules in the API, see Pushing to the Code Engine Programmatically
Get company credits
get_company_credits()
Returns the credits status for the company:
Current period
Used credits for the current period
Total credits for the current period
Return
A dict representing the company credits status. An example:
{ "company_name":"example.com", "payment_period":"Annually", "start_date":"2018-09-14", "subscription_type":"Prepaid", "total_credits":170000, "used_credits":11851.905976 }
For monthly payment:
{ "company_name":"exampleOD.com", "payment_period":"Monthly", "start_date":"2018-10-17", "end_date":"2018-11-16", "subscription_type":"On Demand", "used_credits":8.217452 }
Get deployment information
get_deployment_info()
Get information about the current deployment.
Return
A dict containing the deployment information. An example:
{ "gitBranch": "origin/master", "deploymentName": "<DEPLOYMENTNAME", "gitCommitId": "xxxxxxx", "gitAuthorEmail": "example@users.noreply.github.com", "buildTimestamp": "2018-06-13T20:31:54Z" }
Get event type
get_event_type(event_type)
Returns a representation of the requested event type’s mapping and metadata if it exists.
Parameters
event_type: The name of the event type
Return
A dict representation of the event type’s data, An example:
{ "autoMappingError": null, "stats": null, "name": "MyEventType", "fields": [], "origInputLabel": "MyEventTYpe", "consolidation": { "consolidatedSchema": null, "consolidatedTableName": null, "viewSchema": null, "consolidationKeys": null }, "mapping": { "isDiscarded": true, "tableName": "", "readOnly": false, "schema": "public" }, "state": "MAPPED", "usingDefaultMappingMode": true, "mappingMode": "STRICT", "schemaUrls": [] }
For more information about working with event types and mappings, see Managing Mappings Programmatically.
Get event types
get_event_types()
Returns all the event types in the system.
Return
A list of dicts of all the event types that exist in the system (basically, a list of mapping objects but without the fields section).
For more information about working with event types and mappings, see Managing Mappings Programmatically.
Get incoming events count
get_incoming_events_count(minutes)
Get the number of incoming events for the number of specified minutes.
Parameters
minutes: int, number of minutes
Return
Number of events. Example: 55960.0
Get incoming queue metric
get_incoming_queue_metric(minutes)
Gets the average number events in the incoming queue over the previous number of minutes specified.
Parameters
minutes:
Return
Returns the value of the "EVENTS_IN_PIPELINE" metric for the number of minutes specified, An example usage:
iqm = api.get_incoming_queue_metric(100) # 100 minutes print iqm
Get input by id
get_input_by_id(input_id)
Gets an input based on the specified ID.
Parameters
input_id: the ID for the input. If you do not have the ID or the name for an input, you can use
get_inputs()
to get a list of all inputs in the instance, and get the ID or name from that.
Return
Returns an input object (a Python dictionary containing "job" and "task" information). For more information about input objects, see Managing Inputs Programmatically.
An example:
api = alooma.Client(api_key=apikey, account_name=account) INPUT_ID = "aae13-bd1221-123123" inputcfg = api.get_input_by_id(INPUT_ID) state = inputcfg["tasks"].values()[0] print state['type'] # shows the type of input (MYSQL, ORACLE, etc., etc.)
Get input by name
get_input_by_name(module_name)
Get an input based on the specified name.
Parameters
module_name: the name of the input. If you do not have the ID or the name for an input, you can use
get_inputs()
to get a list of all inputs in the instance, and get the ID or name from that.
Return
Returns an input object (a Python dictionary containing "job" and "task" dictionaries). For more information about input objects, see Managing Inputs Programmatically.
An example Salesforce input configuration (configurations are similar between different input types, with most differences limited to the input_state
information):
{ "job": { "validated": true, "name": "SFDC_test", "created_at": "2018-10-03T18:22:26.400436", "configuration": { "oauth2": "yJyY==", "custom_objects": "Account Contact Lead Opportunity ActionLinkGroupTemplate", "auto_map": "true", "is_sandbox": true, "input_default_schema": "PUBLIC", "daily_bulk_queries": 10000, "daily_api_calls": 200000 }, "type": "SALESFORCE", "id": "a3ef" }, "tasks": { "42bcb": { "category": "inputs", "status": "active", "error_stack": "", "name": "SFDC_test", "deployment_name": "alooma-docs-1", "error_message": "", "next_processing_time": "2018-12-11T21:23:19.333634", "last_success_time": "2018-10-09T11:26:06.016097", "error_count": 1876, "state_id": "42bcb", "module_name": "inputs.pullers.salesforce.salesforce", "last_processing_time": "2018-12-11T20:23:19.397145", "task_state": { "batch_id": "56dee", "received_events": 0, "batch_offset": 0, "input_state": { "org_creation_date": "2015-07-13T08:51:19.000+0000", "trim_to_first_event": false, "sobject": "ActionLinkGroupTemplate", "last_existing_ts": "2011-11-30T00:00:00+00:00", "api": null, "batch_range": 8164800.0, "last_deleted_ts": "1969-12-31T00:00:00+00:00" } }, "type": "SALESFORCE", "id": "a3ef54" }, ( ...) } }
An example usage:
api = alooma.Client(api_key=apikey, account_name=account) INPUT_LABEL = "My_Input" inputcfg = api.get_input_by_name(INPUT_LABEL) state = inputcfg["tasks"].values()[0] print state['type'] # shows the type of input (MYSQL, ORACLE, etc., etc.)
Get input sleep time
get_input_sleep_time(input_id)
Gets the sleep time for the specified input. Sleep time is basically the interval between events injected into the incoming queue.
Parameters
input_id: ID of the input whose sleep time to return
Return
Sleep time of the input with ID input_id
.
Get inputs
get_inputs(name=None, input_type=None, input_id=None)
Get a list of all the input nodes in the system.
Parameters
name: Filter by name (accepts Regex)
input_type: Filter by type (e.g. "mysql")
input_id: Filter by node ID
Return
A list of all the inputs in the system, along with metadata and configurations. An example of a single input:
{ "category": "INPUT", "stats": { "latency": 0, "throughput": 0 }, "name": "Salesforce", "deleted": false, "state": { "autoMapping": { "inputLabel": "Salesforce", "state": null, "enabled": "false" } }, "configuration": { "username": "<USERNAME>", "start_time": "2016-09-06T00:00:00.000Z", "auto_map": "false", "custom_objects": "Lead Opportunity Contact Account User Task Event Campaign OpportunityHistory", "daily_bulk_queries": 1000, "daily_api_calls": 7000 }, "type": "SALESFORCE", "id": "<ID>" }
An example usage:
ainputs = api.get_inputs() act = str(len(ainputs)) print "\nInputs : " + act for inp in ainputs: print "Name : " + inp['name'] + " (" + inp['type'] + ")"
For more information about working with inputs, see Managing Inputs Programmatically.
Get mapping
get_mapping(event_type)
Returns a dict representation of the mapping of the event type requested, if it exists.
Parameters
event_type: The name of the event type
Return
The full mapping (dict) with all fields.
For more information about working with event types and mappings, see Managing Mappings Programmatically.
Get mapping mode
get_mapping_mode()
Returns the default mapping mode currently set in the system. The mapping mode should be one of the values in alooma.MAPPING_MODES.
Return
The default mapping mode currently set in the system.
Get maximum latency
get_max_latency(minutes)
Gets the maximum latency during the specified number of minutes.
Parameters
minutes:
Return
Float, latency. Example: 309.576
Get metrics by names
get_metrics_by_names(metric_names, minutes)
Get the specified system metric(s).
Parameters
metric_names: One or more of the metrics listed in "METRICS_LIST" in alooma.py. The list includes:
EVENT_SIZE_AVG
EVENT_SIZE_TOTAL
EVENT_PROCESSING_RATE
INCOMING_EVENTS
RESTREAMED_EVENTS
UNMAPPED_EVENTS
IGNORED_EVENTS
ERROR_EVENTS
LOADED_EVENTS_RATE
LATENCY_AVG
LATENCY_PERCENTILE_50
LATENCY_PERCENTILE_95
LATENCY_MAX
EVENTS_IN_PIPELINE
EVENTS_IN_TRANSIT
minutes: the number of minutes
Return
Returns a dict containing the target (metric name) and data points (value, timestamp) for each metric specified. An example:
[ { "target": "EVENT_SIZE_AVG", "datapoints": [ [ 22984.973548689137, 1529073369 ] ] } ]
Example usage:
from alooma import METRICS_LIST for metric in METRICS_LIST: mbn = api.get_metrics_by_names(metric, 100) print json.dumps(mbn, indent=4)
Get notifications
get_notifications(epoch_time)
Get the notifications in the system based on the time provided and moving forward.
Parameters
epoch_time: the time (in seconds, from epoch) of the oldest notifications to retrieve
Return
Returns a nested dict object, with a single key: messages
. This can be useful when, for example, troubleshooting problem events. Also, get_notifications()
can return a much longer list of notifications than what is displayed in the UI. Other use cases might include using these errors to programmatically clear events stuck in the Restream Queue, or to trigger other tasks based on issues. Example usage:
data=api.get_notifications(time)
Inside data['messages']
is a list of dict objects, where each dict represents a single notification from the Notifications pane on the Alooma Dashboard. For example, a full response might look like:
{'messages': [ {'description': None, 'type': 'METRICS_OUTPUT_EVENTS_LOADED', 'timestamp': 1531288180667, 'params': ['365791', 'Redshift'], 'state': 'NEW', 'aggregatedCount': 1, 'message': '365,791 events were loaded to Redshift', 'typeDescription': 'Events Loaded to Output', 'severity': 'info', 'id': 'alooma-notifications+0+1339'}, {'description': None, 'type': 'METRICS_OUTPUT_EVENTS_LOADED', 'timestamp': 1531284580669, 'params': ['352047', 'Redshift'], 'state': 'NEW', 'aggregatedCount': 1, 'message': '352,047 events were loaded to Redshift', 'typeDescription': 'Events Loaded to Output', 'severity': 'info', 'id': 'alooma-notifications+0+1338'}, {'description': None, 'type': 'METRICS_OUTPUT_EVENTS_LOADED', 'timestamp': 1531280980688, 'params': ['358313', 'Redshift'], 'state': 'NEW', 'aggregatedCount': 1, 'message': '358,313 events were loaded to Redshift', 'typeDescription': 'Events Loaded to Output', 'severity': 'info', 'id': 'alooma-notifications+0+1337'} ]}
Get output config
get_output_config()
Gets the configuration of the current output.
Return
Returns a dict representation of the output configuration.
Get output node
get_output_node()
Return
A dict containing the output node configuration. An example:
{ "configuration": { "username": "<USERNAME>", "skipValidation": "false", "hostname": "<HOSTNAME>", "sinkType": "REDSHIFT", "databaseName": "<DATABASENAME>", "schemaName": "public", "password": "**********", "port": "1234" }, "type": "REDSHIFT", "id": "<ID>", "name": "Redshift" }
Example usage:
on = api.get_output_node() print "Output name : " + on['name'] print "Database name: " + on['configuration']['databaseName']
Get outputs metrics
get_outputs_metrics(minutes)
Returns the number of events erred / unmapped / discarded / loaded in the last X minutes.
Parameters
minutes: number of minutes to check
Return
A group of four numbers. An example:
(0, 20656.0, 15.0, 518333.0)
Get Redshift config
get_redshift_config()
Get the Redshift configuration.
Return
Returns a dict with the Redshift configuration settings/values. An example:
{ "username": "<USERNAME>", "skipValidation": "false", "hostname": "<HOSTNAME>", "databaseName": "<DATABASE>", "schemaName": "public", "password": "**********", "port": "1234" }
Printing the username
:
red = api.get_redshift_config() print "Username : " + red['username']
Get Redshift node
get_redshift_node()
Get the Redshift node information.
Return
Returns a dict containing the node information. An example:
{ "category": "OUTPUT", "stats": { "latency": 9052, "throughput": 69.9 }, "name": "Redshift", "deleted": false, "state": { "additionalInfo": null, "state": "CONNECTION_SUCCESSFUL", "message": "Connection Successful" }, "configuration": { "username": "<USERNAME>", "skipValidation": "false", "hostname": "<HOSTNAME>", "databaseName": "<DATABASE>", "schemaName": "public", "password": "**********", "port": "1234" }, "type": "REDSHIFT", "id": "<>" }
Get restream queue metrics
get_restream_queue_metrics(minutes)
The number of events sent to the Restream Queue in the last specified number minutes (minutes parameter). This is the number shown in the Restream Queue graph in the Alooma Dashboard.
Parameters
minutes:
Return
Returns a float.
Get restream queue size
get_restream_queue_size()
Returns the number of events currently held in the Restream Queue.
Return
An int representing the number of events in the queue.
Get restream stats
get_restream_stats()
Get statistics for the Restream Queue.
Return
Type dict with the following keys; number_of_events, size_used, max_size.
number_of_events: number of available events to restream
size_used: restream used size in bytes
max_size: maximum restream size in bytes
Get samples
get_samples(event_type=None, error_codes=None)
Get sample events.
Parameters
event_type: optional string containing an event type name
error_codes: optional list of strings containing event status codes. status codes may be any string returned by
get_sample_status_codes()
Return
A list of 10 samples. if event_type is passed, only samples of that event type will be returned. if error_codes is given only samples of those status codes are returned.
Get samples stats
get_samples_stats()
Get statistics for sample events returned.
Return
A dictionary where the keys are names of event types, and each value is another dictionary which maps from status code to the amount of samples for that event type and status.
Get samples status codes
get_samples_status_codes()
Return
A list of status codes each event in Alooma may be tagged with in the form of a dictionary of status types and their human-readable descriptions. As Alooma supports more processing capabilities, status codes may be added. These status codes are used for sampling events according to the events' type and status.
{ "VALID": "Valid", "NO_MAPPING": "Valid", "NEW_TYPE": "Valid", "DROP": "Valid", "NO_MAPPED_FIELDS": "No mapped fields", ... }
Get scheduled queries
get_scheduled_queries()
Returns all scheduled queries.
Return
A dict of dicts representing all scheduled queries:
"1234": { "status": "active", "error_stack": "", "deployment_name": "deployment1", "error_message": "", "next_processing_time": "2018-06-08T21:57:40.158350", "last_success_time": "2018-06-08T13:57:40.158039", "error_count": 0, "state_id": "1234", "module_name": "module1", "consolidation_query": "<QUERY HERE>", "configuration": { "custom_query": "select 1;", "event_type": "<EVENT TYPE HERE>", "deployment_name": "alooma-1", "query_type": "custom", "run_at": null, "frequency": 8 }, "last_processing_time": "2018-06-08T13:57:40.158350" "1235": { [ ... ] }
Example usage:
schqs = api.get_scheduled_queries() # returns dict of dicts for query_id, data in schqs.items(): if data['status'] == "active": print(query_id) # show IDs of all active queries
For more information about working with scheduled queries, see Managing Custom Scheduled Queries.
Get scheduled queries in error state
get_scheduled_queries_in_error_state()
Returns all scheduled queries that did not run successfully on the last attempt.
Return
A dict representing all failed scheduled queries.
Get schemas
get_schemas()
Get the schemas in the configured output.
Return
Returns a list of all the schemas of the configured output. An example:
[ { "default": true, "schemaName": "PUBLIC" }, { "default": false, "schemaName": "TESTSCHEMA" } ]
Get secrets
get_secrets()
Returns a list of existing secrets.
Example:
api.set_secrets({"my_user":"example", "my_password": "12345678"} api.get_secrets() # returns ["my_user", "my_password"] api.delete_secret("my_user") # deletes "my_user" api.get_secrets() # returns ["my_password"]
Return
A Python list of secrets.
Get settings
get_settings()
Retrieves the general settings for different features in the UI. Currently it returns data about Digest Email and S3 Retention features.
Return
Returns a dict object. An example:
{ "email-notifications": { "digestWarning": true, "digestSendHour": 20, "recipients": [ "user@example.com" ], "digestInfo": true, "digestError": true, "digestFrequency": "DAILY", "recipientsChanged": true, "digestEmailsTimeZone": "PST", "notificationsInDigest": 20 }, "s3-retention": { "awsAccessKeyId": "AKIAJL2I123456789012", "filePrefix": "with_metadata", "awsSecretAccessKey": "0123456789Puel0J9uQjT4SLbyqQoK/abcdefghi", "awsBucketName": "alooma", "saveMetadata": true, "serverSideEncryption": true, "gzip": true } }
Get Snowflake config
get_snowflake_config()
Get the Snowflake configuration.
Return
Returns a dict with the Snowflake configuration settings/values. An example:
{ "username": "<USERNAME>", "warehouseName": "<WAREHOUSE NAME>", "skipValidation": "false", "accountName": "<ACCOUNT NAME>", "cron": "*/15 * * * *", "databaseName": "<DATABASE NAME>", "schemaName": "PUBLIC", "password": "**********" }
Printing the username
:
snow = api.get_snowflake_config() print "Username : " + snow['username']
Get Snowflake node
get_snowflake_node()
Get the Snowflake node information.
Return
A dict containing the Snowflake node configuration. An example:
{ "category": "OUTPUT", "stats": { "latency": 5091, "throughput": 0 }, "name": "Snowflake", "deleted": false, "state": { "additionalInfo": null, "state": "CONNECTION_SUCCESSFUL", "message": "Connection Successful" }, "configuration": { "username": "<USERNAME>", "warehouseName": "<WAREHOUSE NAME>", "skipValidation": "false", "accountName": "<ACCOUNT NAME>", "databaseName": "<DATABASE NAME>", "schemaName": "PUBLIC", "password": "**********" }, "type": "SNOWFLAKE", "id": "<ID>" }
Get structure
get_structure()
Returns a representation of all the inputs, outputs, and on-stream processors currently configured in the system.
Return
A dict representing the structure of the system.
Get table dependencies
get_table_dependencies(schema, table_name)
Get a list of dependencies for a specific table.
Parameters
schema: the schema in which the table is located.
table_name: name of the table.
Return
Returns a list of strings.
Get table names
get_table_names(schema=None)
Parameters
schema: return tables from a specific schema, else use default
Return
Returns a list of dictionaries, each containing 4 fields: columns (always empty), retention, schema, and table name.
[ { "retention": null, "tableName": "TABLE1", "columns": [], "schema": "PUBLIC" } ]
Get tables
get_tables(shallow=False, schema=None)
Parameters
shallow: only return schema and table names
schema: return tables from a specific schema, else use default
Return
A list object, containing entries for each table. An example of a single table with shallow=False
:
{ "retention": null, "tableName": "user", "columns": [ { "columnType": { "length": 256, "type": "VARCHAR", "nonNull": false, "truncate": false }, "subColumnsMap": null, "columnName": "a", "distKey": false, "sortKeyIndex": -1, "subColumns": null, "primaryKey": false } ], "schema": "public" }
The same table, with shallow=True
:
{ "retention": null, "tableName": "user", "columns": [], "schema": "public" }
Get throughput by name
get_throughput_by_name(name)
Parameters
name: the name of each node, i.e. Inputs, Code Engine, Restream, Mapper, and Output
Return
The throughput for the named node.
Example usage:
tp = api.get_throughput_by_name("Mapper") print tp
Get transform node id
get_transform_node_id()
Return
The ID for the transform node.
Example usage:
ni = api.get_transform_node_id() print ni
Get used credits
get_used_credits(from_day=None, to_day=None, all_accounts=False)
Get the credits consumption per day for the specified period for the whole company or for the login account. The current day used credits can change according to real use.
Parameters
from_day: 'YYYY-MM-DD' from date period. If None, no from limit is set.
to_day: 'YYYY-MM-DD' to date period. If None, use today as day.
all_accounts: if true, return the used credits for all accounts for the company.
Return
Returns a list of dicts. An example:
[ { "date":"2018-10-24", "instance_name":"example.com", "value":3.0 }, { "date":"2018-10-25", "instance_name":"example.com", "value":10.4 }, { "date":"2018-10-26", "instance_name":"example.com", "value":0.0 }, { "date":"2018-10-27", "instance_name":"example.com", "value":0.0 }, { "date":"2018-10-28", "instance_name":"example.com", "value":4.0 } ]
Get users
get_users()
Returns a list of all the users that have access to the account:
[{'email': 'user@example.com', 'first_name': None, 'last_name': None, 'isPending': False}, {'email': 'user2@example.com', 'first_name': None, 'last_name': None, 'isPending': False}]
Return
List of all users with access to the account.
Example usage:
ausers = api.get_users() for user in ausers: print "email : " + user['email']
Purge restream queue
purge_restream_queue()
Return
A standard HTTP response code on success or fail.
Remove all inputs
remove_all_inputs()
Return
A standard HTTP response code on success or fail.
Remove input
remove_input(self, input_id)
Parameters
input_id: the id for a given input
For example: This can be found in the url in the input settings.
Return
A standard HTTP response code on success or fail.
For more information about working with inputs, see Managing Inputs Programmatically.
Remove scheduled query
remove_scheduled_query(query_id)
Parameters
query_id: the id of the query to remove. See Get scheduled queries for an example of how to retrieve query ids.
Return
A standard HTTP response code on success or fail.
Schedule query
schedule_query(event_type, query, frequency=None, run_at=None)
Return requests response to create a query.
Parameters
event_type: Alooma Event Type Related to Query (exiting event type is required, even if just a placeholder)
query: Desired Query to Schedule
frequency: Desired hours between query runs
run_at: Cron like string to run query on
Note
You must select either frequency
or run_at
For more information about working with scheduled queries, see Managing Custom Scheduled Queries.
Return
Return requests response to create a query.
Set BigQuery configuration
set_bigquery_config(project_name, dataset_name, skip_validation=False)
Set BigQuery configuration.
Parameters
BigQuery schema
database_name: BigQuery database name
skip_validation: :type bool: True skips configuration validation
Return
Python dict containing the response’s content.
Set Code Engine code
set_code_engine_code(modules)
Define modules in the Code Engine. The default is the "main" module.
Parameters
modules: a Python map of module name (string) to code (string)
Return
None.
Example:
# Define and submit a code module named 'submodule'. # this module contains a dict called 'transform_names' # and a method called 'transform_ft_to_mtr' # Create the map my_submodule = {"submodule" : """transform_names = {'John': 'Jonathan', 'Bob': 'Robert', 'Rick': 'Richard'} def transform_ft_to_mtr(x): return x * 0.3048"""} api.set_code_engine_code(my_submodule)
For more information about working with Code Engine modules in the API, see Pushing to the Code Engine Programmatically
Set input sleep time
set_input_sleep_time(input_id, sleep_time)
Parameters
input_id: ID of the input whose sleep time to change
sleep_time: new sleep time to set for input with ID input_id
Return
Result of the REST request.
Set mapping
set_mapping(mapping, event_type, timeout=300)
Parameters
mapping:
event__type
timeout: (default is 300)
Return
For more information about working with event types and mappings, see Managing Mappings Programmatically.
Set mapping mode
set_mapping_mode(mode)
Sets the default mapping mode in the system.
Parameters
mode: one of the values in alooma.MAPPING_MODES
Return
Set output
set_output(output_config, output_name=None)
Set Output configuration.
Parameters
output_config: :type dict. Output configuration. Should contain output-specific configurations (see examples below) and the following parameters:
skip_validation: (type bool) True skips output configuration validation
sink_type: Output type. REDSHIFT, BIGQUERY or SNOWFLAKE
Redshift example:
{"hostname":"redshift-host", "port":5439, "schemaName":"public", "databaseName":"some_db", "username":"user", "password":"password", "skipValidation":false, "sinkType":"REDSHIFT"}
Snowflake example:
{"username":"user", "password":"password", "accountName":"some-account", "warehouseName":"SOME_WH", "databaseName":"SOME_DB", "schemaName":"PUBLIC", "skipValidation":"false", "sinkType":"SNOWFLAKE"}
BigQuery example:
{"projectName":"some-project", "datasetName":"some-dataset", "skipValidation":"false", "sinkType":"BIGQUERY"}
output_name: UI display name
Return
Python dict containing the response’s content.
Set Redshift config
set_redshift_config(hostname, port, schema_name, database_name, username, password, skip_validation=False, ssh_server=None, ssh_port=None, ssh_username=None, ssh_password=None)
Set Redshift configuration.
Parameters
hostname: Redshift hostname
port: Redshift port
schema_name: Redshift schema
database_name: Redshift database name
username: Redshift username
password: Redshift password
skip_validation: :type bool: True skips configuration validation
ssh_server: (Optional) The IP or DNS of your SSH server as seen from the public internet
ssh_port: (Optional) The SSH port of the SSH server as seen from the public internet (default port is 22)
ssh_username: (Optional) The user name on the SSH server for the SSH connection (the standard is alooma)
ssh_password: (Optional) The password that matches the user name on the SSH server
Return
Python dict containing the response’s content.
Set S3 retention
set_s3_retention(aws_bucket_name, aws_access_key, aws_secret_key, file_prefix=None, save_metadata=True, gzip=True, server_side_encryption=True)
Configure s3 retention which stores the raw events received by Alooma to a custom S3 bucket. Retention files are stored in folders by input name and timestamp.
Parameters
aws_bucket_name: where retention files will be created
aws_access_key: with permissions to list and write files
aws_secret_key: to go with the aws_access_key
file_prefix: (optional) files will upload to s3://bucket/prefix/
save_metadata: save raw events with their metadata (default=True)
gzip: create gzipped files (default=True)
server_side_encryption: use S3 encryption (default=True)
Return
A standard HTTP response code on success or fail.
Set secrets
set_secrets(secrets)
Create or update secrets.
Parameters
secrets: dictionary of secrets to create/update
Example:
api.set_secrets({"my_user":"example", "my_password": "12345678"} api.get_secrets() # returns ["my_user", "my_password"] api.delete_secret("my_user") # deletes "my_user" api.get_secrets() # returns ["my_password"]
Return
None.
Set settings email notifications
set_settings_email_notifications(email_settings_json)
Parameters
email_settings_json: here are the default email settings:
CopyDEFAULT_SETTINGS_EMAIL_NOTIFICATIONS = { "digestInfo": True, "digestWarning": True, "digestError": True, "digestFrequency": "DAILY", "recipientsChanged": False, "recipients": [] }
Return
A standard HTTP response code on success or fail.
Set Snowflake config
Copyset_snowflake_config(account_name, warehouse_name, schema_name, database_name, username, password, skip_validation=False)
Set Snowflake configuration.
Parameters
account_name: Snowflake account name
warehouse_name: Snowflake warehouse name
schema_name: Snowflake schema
database_name: Snowflake database name
username: Snowflake username
password: Snowflake password
skip_validation: :type bool: True skips configuration validation
Return Type dict. Response’s content.
Set transform to default
Copyset_transform_to_default()
Sets the Code Engine Python code to the default, which is a blank transform that makes no changes to any events.
Return
A standard HTTP response code on success or fail.
Start restream
Copystart_restream()
Starts a Restream, streaming data from the Restream Queue to the pipeline for processing.
Return
A standard HTTP response code on success or fail.
Test transform
Copytest_transform(sample, temp_transform=None)
Runs the current deployed transform — or one that is provided to the method — against the specified sample event.
Parameters
sample: a json string or a dict, representing a sample event
temp_transform: optional string containing transform code. if not provided, the currently deployed transform will be used.
Return
The results of a test run of the temp_transform on the given sample. This returns a dictionary with the following keys:
output - strings printed by the transform function
result - the resulting event
runtime - milliseconds it took the function to run
Test transform all samples
Copytest_transform_all_samples(event_type=None, status_code=None)
Test many samples on the current transform at once.
Parameters
event_type: optional string containing event type name
status_code: optional status code string
Return
A list of samples (filtered by the event type and status code if provided), for each sample, a result key is added which includes the result of the current transform function after it was run with the sample.
Unmap field
Copyunmap_field(mapping, field_path)
Parameters
mapping: this is the mapping json
field_path: use the field_path to find the exact field
For example:
mapping == {"a":1, b:{c:3}}
the "c" field_path == a.b.c
Return
New mapping JSON from which the last argument would be removed.
For example:
mapping == {"a":1, b:{c:3}}
field_path == "a.b.c"
then the mapping would be the same as the old mapping, but the "c" field would be removed -> {"a":1, b:{}}
Search results
No results found
Was this helpful?