Zoomdata Version

Managing the Zoomdata Query Engine

The query engine is a stand-alone microservice within the Zoomdata environment that processes visualization queries. The query engine is installed as a Zoomdata microservice. If you install or upgrade Zoomdata using the supplied installation script, the query engine microservice is started automatically. If you install or upgrade Zoomdata manually, you must manually enable and start the query engine microservice.

The query engine also hosts the embedded Spark driver that Zoomdata uses for processing.

Configuring the Query Engine

In your environment, the query engine is called zoomdata-query-engine. To configure and manage the microservices that the query engine executes and runs, you need to make changes to the query-engine.properties file. It contains similar information to that of zoomdata.properties, but contains properties that are specific to how the query engine operates within your environment.

The query engine also has a query-engine.env file and a query-engine.jvm file. You can edit these files to configure the following:

  • To define the environment variables that are visible for the microservice, edit the query-engine.env file.
  • To configure the JVM options that are used to startup Zoomdata microservices, edit the query-engine.jvm file.

By default, the memory configurations gives the query engine up to 6GB of heap memory. During microservice startup, query engine consumers at least 1GB of memory. You can alter this using the following parameters in the query-engine.jvm file:

-Xms1g
-Xmx6g

Query Engine Properties

Most of the query engine components can be edited in the query-engine.properties file. You can manage the query engine using the following properties.

Property Default Value Description
Service Discovery
discovery.enabled true Whether Consul is used for microservice discovery
discovery.registry.host localhost Consul microservice host
discovery.registry.port 8500 Consul microservice port
Service Logging
logging.level.com.zoomdata INFO Sets the log level for log messages in the query engine microservice log file. The following options are available for this property: TRACE, DEBUG, INFO, WARN, and ERROR.
logging.unified.host 127.0.0.1 Sets the host IP address for Fluentd server message logging. For more information, see Setting Up Fluentd Unified Logging.
logging.unified.label Query Engine FluentD
logging.unified.level OFF Sets the log level for messages logged to the Fluentd server. The following options are available for this property: TRACE, DEBUG, INFO, WARN, ERROR, and OFF. If set to OFF, Fluentd unified logging is disabled. For more information, see Setting Up Fluentd Unified Logging.
logging.unified.port 24224

Sets the port for Fluentd server message logging. For more information, see Setting Up Fluentd Unified Logging.

logging.unified.tag query-engine Sets the microservice tag name for messages logged to the Fluentd server. This is important because the tag identifies the microservice to which the log messages apply. Valid values are query-engine, job-scheduler, zoomdata-server, stream-writer, upload-service, and edc-<connector-name> (where <connector-name> is one of the names listed in Connector Properties). For more information, see Setting Up Fluentd Unified Logging.
qe.access.log.file.size 10 Access log file size
qe.activity.log.file.max.index 1 Number of files with activity logs
qe.activity.log.file.size 20 Activity log file size
qe.error.log.file.size 5 Error log file size
qe.log.file.size 20 General log file size (in MB)
qe.spark.log.file.size 10 Spark log file size
qe.websocket.log.file.size 10 Websocket log file size
syslog.host 127.0.0.1 Sets the host IP address for Syslog server message logging.
syslog.log.level OFF Sets the syslog log level for messages logged to the Syslog server. The following options are available for this property: TRACE, DEBUG, INFO, WARN, ERROR. and OFF.
syslog. port 1514 Sets the port for Syslog server message logging.
trace.requests false Enables detailed tracing of HTTP request
tracing.sampler.probability 1 The distributed tracing request rate percentage. Valid values are between 0 to 1.0, where 0 disables tracing and 1.0 indicates tracing 100% of requests.
General Microservice Configuration for Jetty Web Server
jetty.threadPool.maxThreads 200 Number of threads to serve the HTTP & WebSocket clients
qe.server.ws.idle.timeout 86400000 Idle time that allows the WebSocket to be still valid
qe.server.ws.input.buffer.size 16384 Buffer size for the WebSocket message
qe.server.ws.max.message.size 1048576 Max size of a message that could be sent over the query engine WebSocket
server.port 5580 REST/WebSocket microservice port
server.ssl.enabled false Defines whether the REST/WS API should use SSL
server.ssl.key-store Path to the file with the keystore
server.ssl.key-store.password Password for the keystore
Graceful Shutdown    
application.graceful.shutdown.enabled true Indicates whether or not graceful shutdown processing should occur. Valid values are true (perform graceful shutdown processing) or false (do not perform graceful shutdown processing).
application.graceful.shutdown.event-propagation-timeout-sec 5 Specifies how long (in seconds) a query engine instance will wait to allow clients to receive the information that it is out of service.
application.graceful.shutdown.force-kill-timeout-sec 30 The maximum number of seconds that a query engine instance will wait for the number of its active tasks to reach zero. When this time has elapsed, all remaining active WebSockets serving in-flight queries are closed and then the query engine instance will stop.
Topology Configuration    
qe.fuzzycache.timeout 60 The time, in minutes, for which internal query engine fuzzy cache will store cached data (RDD references).
qe.max.allowed.time.groups 10000 The maximum amount of time in which groups can be generated by the Include Blanks function.
qe.zengine.edc.rows.limit 1000000 The maximum number of records that can be fused from a single data source.
sharpening.samples.max.count 1000 The maximum number of sharpening requests that can be generated by the sharpening process.
sharpening.samples.skip.first 1 Defines how many first samples are skipped before visualizing the result of sharpening
sharpening.timegroup.batch.size 10 A minimal number of time groups on visualization to enable sharpening
topology.pool.count 25 Maximum size of the thread pool for the Spark topology for all operations except reading from connector
topology.pool.count.read 100 Maximum size of the thread pool for the Spark topology for readings from connector
zoomdata.validation.histogram.max.size 1000 Histogram max bucket count
Spark Driver Configuration    
spark.<property-name> Any property from QE configuration file that starts with prefix spark. will be propagated to Spark context config.
spark.sql.<property-name> Any property from QE configuration file that starts with prefix spark.sql. will be propagated to Spark SQL config
spark.ui.port 4040 Port where Spark UI is exposed

Was this topic helpful?