Documentation

S3 Integration

Alooma can read and replicate all of the data in files from an AWS S3 (Simple Storage Service) bucket. This allows you to, for example, merge arbitrary data from S3 with client usage data in your data destination.

Connecting to S3

Files in your S3 bucket can be packaged and compressed using Tar or GZip, and can contain a variety of different file formats.

  1. Add your S3 input from Alooma's plumbing screen.

  2. Give your input a name. The name will appear in the plumbing screen, and will name the events that the input emits.

  3. Provide the following access info:

    • AWS Access Key ID (20 uppercase characters). Found under Users in your dashboard.

    • AWS Secret Access Key (40 base64 characters). Where is my access key?

    • Bucket Name (between 3 and 63 lowercase characters)

    • File prefix The prefix of the files you'd like to upload, for instance mydir/sys_log. Files not referenced by the prefix are ignored.

  4. Define which files to import:

    • All files will pull all the data from your bucket (for a given prefix, if defined above), and will continue to do so forever.

    • A date span pulls files written between the given dates, inclusive. If you select the date span option and don't specify dates, the input will only pull files written after the input was created.

    Note

    Because the input reads a single type of file, one approach is to have the bucket contain only files of the same file type (consider storing different file types in different buckets). Alternatively, you can "prefix" the files and have several file types in the same S3 bucket so that the input only reads the ones that have your specified prefix and are, for example, .csv files.

  5. Define the format of the files in your S3 from the list of our available file formats.

Keep the mapping mode to the default of OneClick if you'd like Alooma to automatically map the S3 file events directly to your data destination. Otherwise, they'll have to be mapped manually from the Mapper screen.

Required Permissions

The AWS account must have the following permissions:

Copy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:Get*",
        "s3:List*"
      ],
      "Resource": "arn:aws:s3:::BUCKET_NAME"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:Get*",
        "s3:List*"
      ],
      "Resource": "arn:aws:s3:::BUCKET_NAME/*"
    }
  ]
}

If any file in S3 is updated, then that entire file is sent again (not only its updates). Alooma creates an event type in the Mapper screen for each S3 input you create. You can use the Code Engine to change the event type.

Note

If there are any null bytes in rows in your data, those rows will be discarded and will not appear in the Restream Queue.

That's it! You're ready to integrate S3 data into Alooma!

Search results

    No results found