Managing Elasticsearch Connectors
The following versions of Elasticsearch are supported:
- Elasticsearch 1.4.1 - 1.7.5
- Elasticsearch 2.0.0 - 2.4.3
- Elasticsearch 5.0 - 5.4
- Elasticsearch 6.0 - 6.1
The table below lists information on the features that are supported by Elasticsearch.
Supports Distinct Count? | Yes |
Supports Group-by Time? | Yes |
Supports Multi Group-by Charts? | Yes |
Supports Histogram? | Yes |
Supports Box Plot? | Yes |
Supports Derived Fields? | Yes |
Custom SQL Capable? | No |
Live Mode & Playback | Yes |
Supports Last Value? | No |
Managing your elasticsearch Connectors
When establishing a connection to Elasticsearch, you need to do the following:
- Specify the Connection String: you may use HTTP/HTTPs or Transport (TCP)/Transports protocols to connect to your data source.
- For HTTP/HTTPs protocol specify the base URL, whereas for Transport/Transports, specify the list of nodes. Keep in mind, that you must specify the nodes within one cluster.
- Provide the connection string in the corresponding format:
Protocol Connection String Format Example http/https <schema>://<host1>:<port1>,...,<hostN>:<postN>/<prefix>
http://ip-10-2-2-241.ec2.internal:80/es transport/transports <schema>://<node>,<node>,<node>
transports://10.2.2.2:9010,10.2.2.3:9010 <schema>
- stands for the protocol that you want to use:- HTTP or HTTPs (with SSL support);
- Transport/Transports (with SSL support)
<node>
- an address of a node within a cluster in the following format: host:port- If required, specify your Elasticsearch User Name and Password .
- Select Validate to confirm your connection.
You select the indices and types to be queried, and select the fields to be handled. You can do this in three steps:
-
- Select indices and aliases to be queried.
-
You can select indices
Manually
or
Automatically
.
- If you want to get the data only from specific indices, select the
Manually
option and choose the corresponding indices from the list below.
- The
Automatically
option is more flexible. It lets you set the pattern by which the indices will be selected automatically.
-
Keep in mind, that in such case the fields for the indexes will not be refreshed. That is, if the new fields are added to your data source, they are added to Zoomdata only after you click the
Refresh Fields
button on the
Fields
page. If there are some changes in the existing fields (for example, if a field has been removed) they won't be applied.
Note that in case no indices match pattern while querying, your charts are returned empty. - For this option, you can select one of the pattern types:
-
Native
- specify the pattern for index names. Use asterisk (*) to replace one character or a set of characters.
For example, you want to get all the indices whose name starts with log and ends with 16 . In this case, specify the following pattern: log*16 -
Time-Based
- set the time pattern to get the matching indices.
Check the supported date and time patterns
.
For example, the time pattern YYYY-MM will return all the indices, whose name will match this pattern (as shown in the Figure 5 example). Note that if the Index Name include text with the time and date pattern, you need to enclose the text portion in brackets [ ]: -
Optionally, configure filtering by type If you need to filter by the type, select
Enable Filter By Type
and click
Filter
. When you click
Edit
, the list of types available in the selected indices is displayed.
If this checkbox is cleared, all the types that refer to the selected indices are selected.
If some fields have different data types in types, you are not able to use them for grouping, filters, and so on. However, the option is still available for raw export. - Configure the fields settings if needed. If your data set contains multi-field types, they are recognized and listed under the select fields section.
Index name | Pattern |
---|---|
2016-01 | YYYY-MM |
2016-3 | YYYY-Q |
10:23:11 | HH:MM:SS |
logstash-2016-06-14 | [logstash-]YYYY-MM-DD |
Their sub-fields are detected according to mapping. The fields of the token_count type cannot be used in raw export and are not shown in details and the text-search results.
Chart Settings for Elasticsearch
On the Charts page, you can:
- Edit Global Default Setting.
- Select the Standard and, if available, Custom chart styles to be used with the data source.
- Set default parameters (group, sub-group, colors, sorting, and so on) for each chart style.
Select Finish to save your changes. Once your data connection has been established, it is listed under My Data Sources.
SERVICE COLUMNS
When you connect to your Elasticsearch data source, the additional service column _type is added.
The _type column contains all selected Elasticsearch types that you can visualize as attributes on your charts.
WORKING WITH ELASTICSEARCH
Distinct Counts and Percentiles
Distinct count and percentiles metrics return approximate values in Elasticsearch. The precision of the result returned by distinct count metric depends on precision threshold setting (default value is 1000).
You can change the value of precision threshold by setting the
elasticsearch.query.cardinality.precision.threshold
property in the
zoomdata.properties
file.
See Elasticsearch's documentation on the following for more information:
- For Elasticsearch version 1.4-1.7, see the following for percentiles and distinct count .
- For Elasticsearch version 2.0-2.4, see the following for percentiles and distinct count .
The table below lists all available properties that you can modify to work with Elasticsearch.
Property | Default Value | Use the property to | Notes |
---|---|---|---|
elasticsearch.query.cardinality.precision.threshold | 1000 | control the level of accuracy of the distinct counts | The maximum supported value is 40000. However, Zoomdata does not recommend to set such value as it may result in performance issues and the data source itself may return errors. For more info, refer to the Precision Control section by Elasticsearch. |
elasticsearch.query.limit.nongrouped | 10000 | set the limit for the number of non-grouped records (per shard) to execute on. | |
elasticsearch.query.limit.grouped | 10000 | set the limit for the number of grouped records (per shard) to execute on. |
If you need to change the default settings, you can add the corresponding properties (listed above) to the
zoomdata.properties
file and assign the required values. For more details about working with the
zoomdata.properties
file, refer to the article
Managing Configurations in Zoomdata
.
Tokenization
Keep in mind that Elasticsearch, by default, tokenizes or analyzes 'Fields' that are of type 'string' (or attribute). As a result, strings consisting of two or more words may become separate fields when connected to Zoomdata (for example, city names like Las Vegas ). To disable this process and ensure that a string field is not tokenized, enter the following code for that field:
index: "not analyzed"
Example:
City: {
type: "string"
index: "not_analyzed"
}
Was this topic helpful?