Setting up Unified Logging
In Zoomdata, you have the ability to use FluentD as a logging layer to which you can direct the logs for various components of Zoomdata. This enables you to customize the output of logging to meet the needs of your environment.
Zoomdata leverages FluentD’s unified logging layer to collect logs via a central API. FluentD can be configured to aggregate logs to various data sources or outputs. For example, if you are directing all log files from your
zoomdata-errors.log to a FluentD server, you can add one of FluentD’s plugins to write the log files to Elasticsearch to analyze web client errors for your environment.
Unified logging does not replace the default logging architecture. It only augments that experience with an option for those wanting additional logging control. By default, unified logging is disabled and needs to be enabled for each component you want captured in FluentD
Before you begin
Zoomdata’s unified logging configuration relies on FluentD. To write logs from FluentD to a particular database or source, FluentD should be configured appropriately.. To see a list of compatible FluentD plugins, refer to FluentD's website and documentation. If written to a data source compatible with Zoomdata’s connectors, Zoomdata can then be used to analyze and visualize your logs.
- A server with FluentD installed and configured. For information and steps, refer to FluentD's documentation. The Zoomdata installer also provides a simple, pre-configured FluentD service.
- If FluentD is not on the same server as Zoomdata, yo need the host and port numbers for the server.
- To output logs to another data source, that data source should be available and configured in conjunction with FluentD to receive logs
- To visualize logs within Zoomdata, the data source referenced above should also be supported by Zoomdata. For more information, see the Data Source Quick Reference.
To set up unified logging for your Zoomdata server
At a high level, the steps to set up FluentD are as follows:
- Decide what services (such as zoomdata, query-engine, etc.) you want to log to FluentD
- Enable unified logging for that particular service.
- Configure the host and port information for your FluentD server to the properties file.
- On your server, access the
.propertiesfile of the service you want to run unified logging on. For example,
etc/zoomdata/query-engine.properties. You can enable unified logging for these services:
- Zoomdata scheduler
- Web services (zoomdata.properties)
- Query engine
- Data source connectors
- Upload service
- Streamwriter framework
logging.unified.level = INFO
logging.unified.tag = query-engine
logging.unified.host = yourserverIPaddress
logging.unified.port = yourserverport
On the FluentD server, you can then direct your logs to a data source of your choice. For information and steps, see FluentD's documentation on Output.
The log level is configured with standard logging levels from: ERROR, WARN, INFO, DEBUG, TRACE, or OFF. If set to OFF, unified logging is disabled.
Was this topic helpful?