Documentation

Managing Inputs Programmatically

Abstract

SummaryAn overview of inputs, including how to work with them using various input functions: create, get, edit, remove.

An input is a way of referencing your source data. For each data source, you need an input in order for Alooma to be able to import that source's data into the data destination. You can manage your inputs programmatically instead of via the UI.

Each input has the following keys: 'name', 'id', 'type', and 'configuration' ('configuration' is actually a python dictionary and is unique per input 'type').

Input functions

Create Input: Create an input based on the specified data and parameters

Copy
api.create_input(input_post_data, one_click=True, validate=True)

Get Inputs: Return a list of inputs and their configurations

Copy
api.get_inputs()

Get an Input: Return a specific input

Copy
api.get_input_by_name(name)

# or by ID:

api.get_input_by_id(input_id)

Edit Input: Change the configuration and add new components (objects, tables, etc.)

Copy
api.edit_input(input_data)

Remove Input: Remove the specified input

Copy
api.remove_input(input_data['id'])

You can also pause and resume inputs.

See the examples below for usage.

How to get a list of current inputs

You can use get_inputs() to retrieve a list of input objects.

Copy
inputlist = api.get_inputs()

for inp in inputlist:
    print inp['name']
How to get an input

Input objects have the following keys: job and tasks. The job key is basically the configuration for the selected input, including things like the name of the input and the objects being retrieved. The tasks key contains state information for the input, including the current status and last processing time (and other information useful for replication/consolidation). You will generally have a state for each table or object being replicated, or a single "miner" state to track the most recent replication timestamp.

Copy
# Authenticate
api = alooma.Client(API_KEY, ACCOUNT_NAME)

INPUT_NAME = 'TestInput'
input_data = api.get_input_by_name(INPUT_NAME)[0]
How to remove an input
Copy
# Authenticate
api = alooma.Client(API_KEY, ACCOUNT_NAME)

INPUT_NAME = 'TestInput'
input_data = api.get_input_by_name(INPUT_NAME)[0]

#
# Remove an Input
#
api.remove_input(input_data['id'])

How to edit an input (set an input to automap)
Copy
INPUT_LABEL = 'Input Label'

api = alooma.Client(API_KEY, ACCOUNT_NAME)

input_data = api.get_input_by_name(INPUT_LABEL)[0]
input_data['configuration']['auto_map'] = 'true' # use 'false' to disable automap

api.edit_input(input_data)
How to edit an input (add a table)

Suppose you have an input named "Production" replicating incrementally via ODBC and you want to add the "tags" table to the replication:

Copy
INPUT_LABEL = 'Production'
TABLE_MAP = {
"tags": "updated_at"
}
api = alooma.Client(API_KEY, ACCOUNT_NAME)
input_data = api.get_input_by_name(INPUT_LABEL)[0]

#
# List of Current Tables Replicating
#
tables = json.loads(input_data['configuration']['tables'])

#
# Add New Tables
#
for table in TABLE_MAP.keys():
    if table in tables:
        print 'ALERT: %s already replicating' % table
        continue
    tables[table] = TABLE_MAP[table]
input_data['configuration']['tables'] = json.dumps(tables)
api.edit_input(input_data)
How to edit an input (toggle autodump for tables)

For Log Replication inputs (like MySQL) you can toggle the autodump feature on/off using the following logic with the query_dump key. Enabling autodump means that when you add a table, Alooma not only adds it to the ongoing replication, but starts a full table dump. It's needed when you first add a table.

Copy
import alooma
INPUT_NAME = 'Input Name'
api = alooma.Client(API_KEY, ACCOUNT_NAME)
inp = api.get_input_by_name(INPUT_NAME)[0]

inp['configuration']['query_dump'] = True  # Set to False to disable auto-dump
api.edit_input(inp)
How to create an input (Kafka)

Each different type of input requires a different set of parameters. For this example, we'll create a Kafka input. You'll need to replace the values in <>s (such as <HOSTNAME> and <INPUT_NAME>), and "topics" which is a space delimited list of topics to replicate.

Copy
import alooma
import json
import os

# authenticate
api = alooma.Client(api_key=os.environ['ALOOMA_API_KEY'])


# sasl is optional/based on your environment
sasl_config = {
                'sasl_mechanism': 'PLAIN',
                'sasl_plain_password': '<SASL_PLAIN_PW>',
                'sasl_plain_username': '<SASL_PLAIN_USERNAME>',
                'security_protocol': 'SASL_SSL'
              }

input_config = {
                'auto_map': 'true',
                'hostname': '<HOSTNAME>',
                'input_default_schema': 'PUBLIC',
                'port': 9092,
                'sasl_auth': json.dump(sasl_config),    # sasl is optional
                'topics': 'topic1 topic2 topic3 topic4' # define topics to replicate
               }

new_input ={
             'configuration': input_config,
             'name': '<INPUT_NAME>',
             'type': 'KAFKA_INPUT'
           }

input_id = api.create_input(input_post_data=new_input)
print input_id
How to pause and resume an input

Once you've initialized the API you can pause and resume an input via:

Copy
import requests
import alooma

def pause_input(api, input_label):
    inp_cfg = api.get_inputs(input_label)[0]

    params = {"pause": 'true'}
    url = api.rest_url + "inputs/{input_id}/pause".format(input_id=inp_cfg['id'])

    return api._Client__send_request(requests.put, url, params=params)

def resume_input(api, input_label):
    inp_cfg = api.get_inputs(input_label)[0]

    params = {"pause": 'false'}
    url = api.rest_url + "inputs/{input_id}/pause".format(input_id=inp_cfg['id'])

    return api._Client__send_request(requests.put, url, params=params)

Search results

    No results found