Zoomdata Version

Managing your Query Engine for Zoomdata

Query engine is a stand-alone service within the Zoomdata environment which runs functionality related to visualization query processing. Previously, the query engine was bundled with many other components within the Zoomdata environment. Query engine is installed and runs just like another other service in Zoomdata. After an installation or upgrade, the query engine is automatically started.

Query engine also hosts the embedded Spark driver that Zoomdata uses for processing.

Configuring Query Engine

In your environment, query engine is called zoomdata-query-engine. To configure and manage the services that query engine executes and runs, you need to make changes to the query-engine.properties file. It contains similar information to that of zoomdata.properties, but contains properties that are specific to how the query engine operates within your environment.

Query Engine also has a query-engine.env file and a query-engine.jvm file. You can edit these files to configure the following:

  • To define the environment variables that are visible for service, edit the query-engine.env file.
  • To configure the JVM options that are used for the startup of Zoomdata services, edit the query-engine.jvm file.

By default, the memory configurations gives the query engine up to 6GB of heap memory. During the service startup, query engine consumers at least 1GB of memory. You can alter this via the following parameters in the query-engine.jvm file:

-Xms1g
-Xmx6g

 

Query Engine Properties

Most of the query engine components can be edited in the query-egine.properties file. You can manage the query engine by utilizing the following information.

Property Default Value Description
Service Discovery
discovery.enabled true Whether Consul is used for service discovery
discovery.registry.host localhost Consul service host
discovery.registry.port 8500 Consul service port
Service Logging
syslog.log.level OFF SysLog logging, disabled by default
syslog.host 127.0.0.1 Host name for syslog logging
syslog. port 1514 Port number for syslog logging
logging.unified.level OFF Unified logging; enabling this directs logs to a FluentD server
logging.unified.tag query-engine Query engine service tag for FluentD
logging.unified.label Query Engine FluentD
logging.unified.host 127.0.0.1 FluentD server host
logging.unified.port 24224 FluentD server port
qe.log.file.size 20 General log file size (in MB)
qe.error.log.file.size 5 Error log file size
qe.websoscket.log.file.size 10 Websocket log file size
qe.access.log.file.size 10 Access log file size
qe.spark.log.file.size 10 Spark log file size
trace.requests false Enables detailed tracing of HTTP request
qe.activity.log.file.max.index 1 Number of files with activity logs
qe.activity.log.file.size 10 Activity log file size
General Service Configuration for Jetty web-server
server.port 5580 REST/WebSocket service port
server.ssl.enabled false Defines whether REST/WS API should use SSL
server.ssl.key-store   Path to file with keystore
server.ssl.key-store.password   Password for keystore
jetty.threadPool.maxThreads 200 Number of threads to serve HTTP & WebSocket clients
qe.server.ws.idle.timeout 86400000 Idle time that allow WebSocket to be still valid
qe.server.ws.input.buffer.size 16384 Buffer size for WebSocket message
qe.server.ws.max.message.size 1048576 Max size of a message that could be sent over QE WebSocket
Topology Configuration
topology.pool.count 25 Maximum size of the thread pool for the Spark topology for all operations except reading from connector
topology.pool.count.read 100 Maximum size of the thread pool for the Spark topology for readings from connector
sharpening.timegroup.batch.size 10 A minimal number of time groups on visualization to enable sharpening
sharpening.samples.skip.first 1 Defines how many first samples are skipped before visualizing the result of sharpening
zoomdata.validation.histomgram.max.size 1000 Histogram max bucket count
Spark Driver Configuration
spark.<property-name>   Any property from QE configuration file that starts with prefix spark. will be propagated to Spark context config.
spark.sql.<property-name>   Any property from QE configuration file that starts with prefix spark.sql. will be propagated to Spark SQL config
spark.ui.port 4040 Port where Spark UI is exposed

Was this topic helpful?