Zoomdata Version

Managing Streaming Sources (Twitter, Kinesis, Upload API)

Zoomdata supports the following live streaming sources: (1) Twitter, (2) Kinesis and (3) Upload API. However, keep in mind that these sources require a storage medium for the incoming data flow. Zoomdata utilizes MongoDB to handle the caching of the data.

A known issue with real-time streaming sources ( Twitter and Kinesis ) exists. If such sources are left enabled for an extended period of time, 'Out of Memory' errors may occur in the Zoomdata Server. To avoid this problem, disable such sources when not in use.

Prerequisites

If you upgraded from Zoomdata v2.2 or earlier, then MongoDB already exists in your Zoomdata environment and no additional setup is necessary. If Zoomdata v2.3 is a new installation and you do not already have MongoDB in your network environment, then review this article to install and setup MongoDB.

The Zoomdata Administrator will need administrator level access to the Linux environment in order to access and set up MongoDB along with the necessary parameters in the Zoomdata property file.

If you do not have MongoDB in your network environment, then you will first need to download and install MongoDB to your server. Refer to the MongoDB Install documentation .

  • If you upgraded your Zoomdata Server from v2.2 or earlier, then MongoDB is already available in your Zoomdata environment and you will be able to access it. For more information about upgrades, refer to the article README: Preparations to Upgrade Zoomdata to v2.3 .
  • For the MongoDB version that is supported by Zoomdata, refer to the Data Sources Quick Reference Sheet .

Setup Process

To set up and enable Zoomdata to support your streaming source, you will first need to add the MongoDB parameters to the zoomdata.properties file. To do this, the Zoomdata Administrator will need to stop the Zoomdata Client and access the Linux prompt.

Step 1. Log out of the Zoomdata Client and close the browser window.

Step 2. Access the Linux prompt and log into your Zoomdata Server (via Secure Shell or SSH).

Step 3. Stop the Zoomdata Server.

sudo service zoomdata stop

Step 4. Use the following command to access and open the properties file:

vi /etc/zoomdata/zoomdata.properties
  • If you are not logged in as a root user, then you will need to enter sudo vi /etc/zoomdata/zoomdata.properties in order to create the file.
  • If the file does not exist, this command will create it.
  • Make sure the file is readable by the zoomdata service account

Step 5. Add the following parameters into the file:

# MongoDb parameters
zoomdata.db.host= yourHostIP : PortAddress
zoomdata.db.name= MongoDB_Name
zoomdata.db.username= MongoDB_Username
zoomdata.db.password= MongoDB_User's_Password
  • yourHostIP and PortAddress : enter your MongoDB's connection information
  • MongoDB_Name : enter the space in MongoDB where data for streaming sources will be stored
  • MongoDB_Username : if needed, otherwise there is no need to include this parameter
  • MongoDB_User's_Password : if needed, otherwise there is no need to include this parameter

Step 6. Save and exit the configuration file.

Step 7. Restart the Zoomdata Server.

sudo service zoomdata restart

Disable the Streaming Data Source

To disable the streaming source, do the following:

  • Log in using the Supervisor credential

  • Select the ' Connectors ' tab

  • Go to the 'Connection Types' section and locate the RTS source under 'Connection Type Name'

  • Uncheck the box in the 'Enabled' column (as shown in Figure 4) for the desired connector type


Figure 4

Known Issue

Real-time streaming sources, if left enabled for an extended period of time, may cause 'Out of Memory' errors in Zoomdata. To avoid this problem, disable such sources when not in use.