Zoomdata Version

Managing the Zoomdata Query Engine

The query engine is a stand-alone service within the Zoomdata environment that processes visualization queries. The query engine is installed as a Zoomdata service. If you install or upgrade Zoomdata using the supplied installation script, the query engine service is started automatically. If you install or upgrade Zoomdata manually, you must manually enable and start the query engine service.

The query engine also hosts the embedded Spark driver that Zoomdata uses for processing.

Configuring the Query Engine

In your environment, the query engine is called zoomdata-query-engine. To configure and manage the services that the query engine executes and runs, you need to make changes to the query-engine.properties file. It contains similar information to that of zoomdata.properties, but contains properties that are specific to how the query engine operates within your environment.

The query engine also has a query-engine.env file and a query-engine.jvm file. You can edit these files to configure the following:

  • To define the environment variables that are visible for service, edit the query-engine.env file.
  • To configure the JVM options that are used to startup Zoomdata services, edit the query-engine.jvm file.

By default, the memory configurations gives the query engine up to 6GB of heap memory. During service startup, query engine consumers at least 1GB of memory. You can alter this using the following parameters in the query-engine.jvm file:

-Xms1g
-Xmx6g

Query Engine Properties

Most of the query engine components can be edited in the query-engine.properties file. You can manage the query engine using the following properties.

Property Default Value Description
Service Discovery
discovery.enabled true Whether Consul is used for service discovery
discovery.registry.host localhost Consul service host
discovery.registry.port 8500 Consul service port
Logging Properties
syslog.log.level OFF Sets the syslog log level for messages logged to the Syslog server. The following options are available for this property: TRACE, DEBUG, INFO, WARN, ERROR. and OFF.
syslog.host 127.0.0.1 Sets the host IP address for Syslog server message logging.
syslog. port 1514 Sets the port for Syslog server message logging.
logging.level.com.zoomdata INFO Sets the log level for log messages in the query engine service log file. The following options are available for this property: TRACE, DEBUG, INFO, WARN, and ERROR.
logging.unified.level OFF Sets the log level for messages logged to the Fluentd server. The following options are available for this property: TRACE, DEBUG, INFO, WARN, ERROR, and OFF. If set to OFF, Fluentd unified logging is disabled. For more information, see Setting Up Fluentd Unified Logging.
logging.unified.tag query-engine Sets the service tag name for messages logged to the Fluentd server. This is important because the tag identifies the service to which the log messages apply. Valid values are query-engine, job-scheduler, zoomdata-server, stream-writer, upload-service, and edc-<connector-name> (where <connector-name> is one of the names listed in Connector Properties). For more information, see Setting Up Fluentd Unified Logging.
logging.unified.label Query Engine FluentD
logging.unified.host 127.0.0.1 Sets the host IP address for Fluentd server message logging. For more information, see Setting Up Fluentd Unified Logging.
logging.unified.port 24224

Sets the port for Fluentd server message logging. For more information, see Setting Up Fluentd Unified Logging.

qe.log.file.size 20 General log file size (in MB)
qe.error.log.file.size 5 Error log file size
qe.websocket.log.file.size 10 Websocket log file size
qe.access.log.file.size 10 Access log file size
qe.spark.log.file.size 10 Spark log file size
trace.requests false Enables detailed tracing of HTTP request
qe.activity.log.file.max.index 1 Number of files with activity logs
qe.activity.log.file.size 10 Activity log file size
General Service Configuration for Jetty Web Server
server.port 5580 REST/WebSocket service port
server.ssl.enabled false Defines whether REST/WS API should use SSL
server.ssl.key-store Path to file with keystore
server.ssl.key-store.password Password for keystore
jetty.threadPool.maxThreads 200 Number of threads to serve HTTP & WebSocket clients
qe.server.ws.idle.timeout 86400000 Idle time that allow WebSocket to be still valid
qe.server.ws.input.buffer.size 16384 Buffer size for WebSocket message
qe.server.ws.max.message.size 1048576 Max size of a message that could be sent over QE WebSocket
Topology Configuration    
topology.pool.count 25 Maximum size of the thread pool for the Spark topology for all operations except reading from connector
topology.pool.count.read 100 Maximum size of the thread pool for the Spark topology for readings from connector
qe.zengine.edc.rows.limit 1000000 The maximum number of records that can be fused from a single data source.
sharpening.timegroup.batch.size 10 A minimal number of time groups on visualization to enable sharpening
sharpening.samples.skip.first 1 Defines how many first samples are skipped before visualizing the result of sharpening
zoomdata.validation.histomgram.max.size 1000 Histogram max bucket count
Spark Driver Configuration    
spark.<property-name> Any property from QE configuration file that starts with prefix spark. will be propagated to Spark context config.
spark.sql.<property-name> Any property from QE configuration file that starts with prefix spark.sql. will be propagated to Spark SQL config
spark.ui.port 4040 Port where Spark UI is exposed

Was this topic helpful?