Zoomdata Version

Connecting to Amazon Kinesis

Amazon Kinesis is a managed service that scales elastically for real-time processing of streaming big data (1). Amazon Kinesis takes in large streams of data records that can then be consumed in real time by multiple data-processing applications that can be run on Amazon Elastic Compute Cloud (Amazon EC2) instances. The data-processing applications use the Amazon Kinesis Client Library and are called "Amazon Kinesis applications (2)."
(1), (2): Excerpted from Amazon Kinesis Developer Guide. http://docs.aws.amazon.com/kinesis/latest/dev/introduction.html .

In order for Zoomdata to connect to Kinesis, you will first need to set up a Kinesis stream from your AWS account. Once that has been established, then you can use Zoomdata to connect to that Kinesis stream and visualize your desired datasets.

Then to allow users access to the Kinesis stream, access rules need to be configured using AWS IAM (Identity and Access Management). IAM enables you to securely control access to AWS services and resources for your users(3). You can create and manage AWS users and groups and use permissions to allow and deny their access to AWS resources. Therefore, users will require an IAM account. For additional information on Amazon IAM and the set up process, visit the AWS IAM page .
(3): Excerpted from AWS Identity and Access Management webpage. http://aws.amazon.com/iam/ .

If you need to set up the Kinesis stream and set up access permissions for users, perform the steps outlined below for “Setting Up A Kinesis Stream.” If you have already set up the Kinesis stream, then you can proceed to the “Connecting Zoomdata to Amazon Kinesis.” section.

SETTING UP A KINESIS STREAM

Before starting the setup process, you may want to take time to address the following action items:

  • define the size of your Kinesis stream by calculating the number of shards needed for the stream.
  • define the users that require access to the Kinesis stream. These users will need access permissions enabled via the Amazon IAM security console. IAM allows you to configure the specific rules for your users that must be adhered to for accessing your Kinesis stream.

To set up a Kinesis Stream, log into your AWS account and take the following steps:

  1. Access Amazon Kinesis.

Figure 1

  1. Click Create Stream .

Figure 2

  1. Specify a name for the Kinesis stream.

Figure 3

  1. Enter the number of shards needed for this stream.
  2. Click Create . It may take several minutes to create the shards for your stream.

Figure 4

  1. Once you have successfully created the Kinesis Stream, the next step is to allow users access to the stream. User permissions must be enabled on Amazon IAM.

SETTING UP USERS ON THE KINESIS STREAM

  1. Click your account name and select Security Credentials .
  2. Click Create New Users .

Figure 5

  1. On the Create User page add up to five new users at once. Enter the names of the new users to provide access to the Kinesis stream.

Figure 6

  1. Verify that the checkbox for Generate an access key for each User is selected. Click Create .
    You will receive a confirmation message indicating that the User(s) have been successfully created. Next, you will want to view the security credentials that was created for each user and record them. The security credentials include: (1) the Access Key ID and (2) the Secret Access Key.
You will be able to view the secret access key only once.

It is advisable to have a record-keeping method to log or track the credentials with the Users. You need to provide these credentials to the users. Users need to enter their personal security credentials into Zoomdata to connect to the Kinesis connector.

Next, permissions need to be set for each new User that was created. IAM offers different policy options for users - from read-only, limited access to more open, administrative options. When selecting the policy, ensure that you are selecting the Kinesis types as different types will be listed.

Zoomdata only needs Read Only Access to be able to visualize the data. However, if a user wants to add data to the stream, then they would need Full Access .
  1. Assign a policy for each new user that was created.

Figure 7

  1. Check the Policy Name and edit as needed to ensure a unique name for the policy.

Figure 8

  1. Click Apply Policy . The Policy setting can be accessed from the Permissions tab.

Figure 9

  1. Once the policy has been set for each user, you can contact them to provide their security credentials. Users will need to know their credential to connect Zoomdata to the Kinesis stream.

CONNECTING ZOOMDATA TO AMAZON KINESIS

To configure the connector, perform the following steps:

  1. Log into Zoomdata.
  2. Click the Sources menu item.

Figure 10

  1. Click the Kinesis connector icon.
  2. Specify the name of your source and add a description (if desired).

Figure 11

  1. Click Next .

Figure 12

  1. On the Streams page, select a stream.

Figure 13

  1. When you create a data source, only the _ts (time stamp) field is added. The data from the stream will be retrieved when you save your data source.
    Save your data source. After this, edit the created data source and complete steps 8 and 9.
  2. On the Fields page, create unique label names, as needed, for each Label field. If necessary, change the Type and Default options, select the checkboxes in the Distinct Count column. You can also add calculations if needed.
    Click Next .

Figure 14

  1. On the Charts page, you can:
  • Edit Global Default Settings
  • Select the Standard and, if available, Custom chart styles to be used with the data source
  • Set default parameters  (group-bys, colors, sorting, etc.) for each chart style
Learn more about how to customize a chart .


Figure 15

  1. Click Finish to save your changes.