Reference docs

• 1: Command line options
• 2: Config file format
• 3: Available Metrics
• 3.1: Available metrics: InfluxDB
• 3.2: Available Metrics: New Relic
• 3.3: Available Metrics: Prometheus

1 - Command line options

Command line options

When starting Butler SOS, you can pass command line options to customize its behavior.
Looks like this:

Usage: butler-sos [options]

Butler SenseOps Stats ("Butler-SOS") is a microservice publishing operational Qlik Sense metrics to InfluxDB, Prometheus and New Relic.
User events and log events can be forwarded from Sense to Butler SOS and then acted upon there. Events can be stored in InfluxDB and sent to New Relic.
Add Grafana for great looking dashboards and you get real-time monitoring of what happens inside a Qlik Sense environment.

Options:
  -V, --version                        output the version number
  -c, --configfile <file>              path to config file
  -l, --loglevel <level>               log level (choices: "error", "warn", "info", "verbose", "debug", "silly")
  --new-relic-account-name  <name...>  New Relic account name. Used within Butler SOS to differentiate between different target New Relic accounts
  --new-relic-api-key <key...>         insert API key to use with New Relic
  --new-relic-account-id <id...>       New Relic account ID
  --skip-config-verification           Disable config file verification (default: false)
  -h, --help                           display help for command

-V, –version

Output the version number of Butler SOS.

-c, –configfile

Specifies the configuration file to use.

Valid values: A path to a configuration file.

Default: Whatever is specified in the NODE_ENV environment variable, with a .yaml extension added. Butler SOS will look for that file in the ./config directory.

Example:

• -c or --configfile are not specified. NODE_ENV is set to production. Butler SOS will try to read settings from ./config/production.yaml.

-l, –loglevel

Specifies the log level to use.
When set, this overrides the log level specified in the configuration file.

Valid values: ’error’, ‘warn’, ‘info’, ‘verbose’, ‘debug’, ‘silly’

Default: ‘info’

New Relic related options

When using New Relic as backend for storing metrics, you can specify New Relic credentials in the config file - but that is not ideal from a security perspective.

To avoid that, you can specify the New Relic credentials on the command line using the following options.

–new-relic-account-name

List of New Relic account names. Used within Butler SOS to differentiate between different target New Relic accounts to which data can be sent. This name has nothing to do with the account name used in New Relic - it’s purely for Butler SOS’ internal use.
Specifically, it’s at multiple places in the config file where you can specificy to which New Relic account to send data.

Enclose account names in quotes if they contain spaces.
Separate multiple account names with a space.

Example: --new-relic-account-name "Account 1" "Account 2"

–new-relic-api-key

List of New Relic API keys. Used to authenticate with New Relic.

Enclose API keys in quotes if they contain spaces.
Separate multiple API keys with a space. Note that the order of the API keys must match the order of the account names, i.e. the first API key corresponds to the first account name, the second API key corresponds to the second account name, and so on.

Example: --new-relic-api-key "API key 1" "API key 2"

–new-relic-account-id

List of New Relic account IDs. Used to identify the New Relic account to which data should be sent.

Enclose account IDs in quotes if they contain spaces.
Separate multiple account IDs with a space. Note that the order of the account IDs must match the order of the account names, i.e. the first account ID corresponds to the first account name, the second account ID corresponds to the second account name, and so on.

–skip-config-verification

Disable config file verification.

By default, Butler SOS verifies the config file when it starts. If the config file is invalid, Butler SOS will log an error and exit.
Use this option to disable config file verification.

-h, –help

Display help for command.

2 - Config file format

The config file is the heart of Butler SOS.
All setting must be defined in the config file - run time errors are likely to occur otherwise.

The sample config file looks like this:

A few things to keep in mind:

• Topic names (e.g. “Butler-SOS.logdb”) are case sensitive.
• First time Butler SOS is started, a new check is done if the specified InfluxDB database already exists. If it doesn’t exist it will be created together with a default InfluxDB retention policy. The retention policy is based on the time period set in the config file.

Top level

Butler-SOS.heartbeat

Heartbeats can be used to send “I’m alive” messages to some other tool, e.g. an infrastructure monitoring tool.
The concept is simple: The remoteURL will be called at the specified frequency. The receiving tool will then know that Butler SOS is alive.

Butler-SOS.dockerHealthCheck

Docker health checks are used when running Butler SOS as a Docker container.

The Docker engine will call the container’s health check REST endpoint with a set interval to determine whether the container is alive/well or not.
If you are not running Butler SOS in Docker you can disable this feature.

Butler-SOS.uptimeMonitor

Butler-SOS.thirdPartyToolsCredentials

Butler-SOS.userEvents

Track individual users opening/closing apps and starting/stopping sessions.
Requires log appender XML file(s) to be added to Sense server(s).

Butler-SOS.logEvents

Log events are used to capture Sense warnings, errors and fatals in real time. Requires log appender XML file(s) to be added to Sense server(s).

Note that log events can be enabled/disabled per source (repository, proxy, scheduler etc).

Butler-SOS.logdb

As of August 2021 log db has been deprecated in Qlik Sense.
It is no longer installed when doing fresh QSEoW installs.

To support older QSEoW clusters out there Butler SOS will for now keep log db support intact.

Butler-SOS.cert

Certificates to use when connecting to Sense. Get these from the Certificate Export in QMC.

Butler-SOS.mqttConfig

MQTT config parameters. These must be correctly defined for any other MQTT features in Butler SOS to work.

Butler-SOS.newRelic

If enabled, select Butler SOS metrics and events will be sent to New Relic.

Note that New Relic destination accounts for events are defined in the Butler-SOS.userEvent and Butler-SOS.logEvent sections, whereas destination accounts for metrics are defined in this section (Butler-SOS.newRelic).

Butler-SOS.prometheus

If enabled, select Butler SOS metrics will be exposed on a Prometheus compatible URL from where they can be scraped by Prometheus.

Butler-SOS.influxdbConfig

InfluxDB config parameters. These must be correctly defined for any other InfluxDB features in Butler SOS to work.

Butler-SOS.appNames

Butler-SOS.userSessions

Extract user session data per virtual proxy.

Butler-SOS.serversToMonitor

The Butler-SOS.serversToMonitor.servers.logDbHost property can be tricky to get right. Easiest way to get the correct value is to look in the Nodes section in the QMC. In the Host name column you find the host names of the various nodes. logDbHost should be set to the first part of each host name:

3 - Available Metrics

3.1 - Available metrics: InfluxDB

InfluxDB

Metrics retrieved from the Sense servers can be stored in an InfluxDB database. You don’t have to be an InfluxDB expert to use Butler SOS, but understanding some basic concepts are helpful.

Storing metrics in InfluxDB is not mandatory, but some kind of metrics storage - either in InfluxDB, New Relic or Prometheus - is needed to take full benefit of Butler SOS’ features.

• InfluxDB is a time series database. This means it is super good at storing values that have a timestamp associated with them - and pretty bad at everything else. In many respects time series databases are the opposite of traditional SQL databases (who are usually pretty bad at handling time series data).

• Because of it’s focus on time series data, InfluxDB has its own query language, InfluxQL. It is somewhat similar to SQL, but also has many unique commands and features.

• Browsing through the key concepts of InfluxDB is a good idea. There you will learn about things such as measurements, series and tags - which are all key to using data stored in InfluxDB.

Metrics structure

The metrics are grouped based on what kind of Qlik Sense data they represent. InfluxDB is a very capable database, so we will only touch on the basics here.

Overview

Measurements are just what it sounds like: snapshots of some value(s), taken at a specific point in time. A measurement can contain several field keys, which for practical purposes can be viewed as the individual metrics.

For example, the list of measurements look like this (using the InfluxDB command line client to explore the database structure):

> use senseops
Using database senseops
> show measurements
name: measurements
name
----
apps
butlersos_memory_usage
cache
cpu
log_event
log_event_logdb
mem
saturated
sense_server
session
user_events
user_session_details
user_session_list
user_session_summary
users
>

Let’s take a look at what field keys the apps measurement contains:

> show field keys from apps
name: apps
fieldKey                     fieldType
--------                     ---------
active_docs                  string
active_docs_count            integer
active_docs_names            string
active_session_docs_names    string
calls                        integer
in_memory_docs               string
in_memory_docs_count         integer
in_memory_docs_names         string
in_memory_session_docs_names string
loaded_docs                  string
loaded_docs_count            integer
loaded_docs_names            string
loaded_session_docs_names    string
selections                   integer
>

Ok, so the field keys are the actual metrics for which we gather data. Collectively those metrics (again: field keys in InfluxDB lingo) above are grouped into a measurement called apps.

There is one more concept you need to understand: tag keys

It’s pretty simple: Tag keys are used to categorise (or simply “tag”) measurements.
Let’s say you use Butler SOS to collect data from ten Sense servers. That’s great, but how will you later distinguish between server 3 and server 8? You need some way of telling your Grafana dashboard to show the data for server 3 (if that’s what you want).

Tags solve this. In the Butler SOS YAML config file you can define any number of tags that will be used to tag data coming in from Qlik Sense.

The beauty of tags is that they play very nicely with Grafana - without them the Grafana dashboards would not be nearly as flexible as they are.

To see what tag keys a certain measurement has you use a query similar to the one above/for fields:

> show tag keys from apps
name: apps
tagKey
------
host
serverBrand
serverLocation
server_description
server_group
server_name
server_type

Note that this list of tags consits of

1. Tags always present. These are inserted by Butler SOS and are present for all measurements. These are host, server_description and server_name.
2. Tags configured in Butler SOS’ config fil. In the example above these are serverBrand, serverLocation, server_group andserver_type.

Measurements and fields

The measurements are grouped based on what part of Sense they are retrieved from. The groups are

1. General health metrics.
2. Messages from the log database.
3. Detailed metrics about what users are connected to (i.e. have sessions open with) which virtual proxies.
4. Messages from the log database.
5. Log events: Warning, error and fatal messages from QSEoW logs.
6. User events: Session and connection related messages from QSEoW logs.
7. Metric relating to Butler SOS itself (i.e. not retrieved from Sense).

General health metrics

A shared set of tag keys are available for all general health metrics:

In addition to the above, all tags defined in the YAML config file for the servers will be included as tag keys.

Measurement: apps

Source: Health check API

Measurement: cache

Source: Health check API

Measurement: cpu

Source: Health check API

Measurement: mem

Source: Health check API

Measurement: saturated

Source: Health check API

Measurement: sense_server

Source: Health check API

Measurement: session

Source: Health check API

Measurement: users

Source: Health check API

User session details

User session metrics have slightly different tag keys depending on the granularity level of the metric - those metrics are therefore listed under each heading below.

Measurement: user_session_summary

Source: Session module API

Tag keys:

Measurement: user_session_list

Source: Session module API

Tag keys:

Measurement: user_session_details

Source: Session module API

Tag keys:

User events

User events capture real-time events in Qlik Sense as they happen.
They originate from Sense’s log4net logging framework and are forwarded from Sense to Butler SOS by means of XML log appenders in Sense.
These events are also forwarded as MQTT messages, allowing other systems to act when warnings/errors/fatals occur in Qlik Sense.

Setup instructions here.

The following user events are handled by Butler SOS:

• Session start
• Session stop
• Connection open
• Connection close.

Measurement: user_events

Tag keys present for all user_events records:

If the user event includes browser user agent information, the following tags will be present:

In addition to the above tags defined in the Butler SOS config file will be added.
More info here.

Fields:

Log events

Log events are used to capture warning, error and fatal messages in Sense. Once in Butler SOS these events are stored in InfluxDB (enabling Grafana dashboards).
These events are also forwarded as MQTT messages, allowing other systems to act when warnings/errors/fatals occur in Qlik Sense.

Setup instructions here.

Source: Proxy service

Events such as failed login attempts will be sent from the proxy service.

Proxy log events have these tags:

Fields in proxy log events:

The raw_event is the actual log event message sent from QSEoW to Butler SOS.
It has the following components:

Source: Scheduler service

Events such as failed reload tasks will be sent from the scheduler service.

Scheduler log events have these tags:

Fields in scheduler log events:

The raw_event is the actual log event message sent from QSEoW to Butler SOS.
It has the following components:

Source: Repository service

The repository service is the hub around which the rest of Qlik Sense revolves.
As such it emit events in many different situations. One example can be when a Sense node is offline (thais example is used in the field description below).

Repository log events have these tags:

Fields in scheduler log events:

The raw_event is the actual log event message sent from QSEoW to Butler SOS.
It has the following components:

Messages from the log database

All log data written to InfluxDB share a common set of tag keys:

Measurement: log_event_logdb

Source: More or less log db. A query is done to the log db in Postgres, the results are stored in InfluxDB. There is thus no Qlik API call per se.

Butler SOS metrics

Measurement: butlersos_memory_usage

These metrics tell you how much memory Butler SOS itself uses.
More info on these metrics and what they mean is available here.

3.2 - Available Metrics: New Relic

New Relic

New Relic offers a complete SaaS observablity stack, ranging from high-volume ingestion of events/metrics/logs/traces to advanced dashboards that can be created ad-hoc using a web UI or from files and templates, for more of an infrastructure-as-code approach.

Storing metrics in New Relic is not mandatory, but some kind of metrics storage - either in New Relic, InfluxDB or Prometheus - is needed to take full benefit of Butler SOS’ features.

In order to view data in New Relic you first have to send data to them.

Butler SOS does this for you.

Furthermore, you can to a large degree control which Qlik Sense metrics, logs and events are sent to New Relic.

Data volumes and pricing

At the time of this writing New Relic offers a generous free plan.
It will be a great starting point for everyone, if there’s a need for more dashboard users etc the account can be upgraded as needed.

In most cases Butler SOS will not generate a lot of data and you can stay within New Relic’s free tier.
The amount of data generated by Sense health metrics and Butler SOS uptime metrics is very small indeed, but if your Qlik Sense environment for some reason generate a lot of log events that can cause the data volumes to increase rapidly.

For example, if a user connects to Sense and gets a https certificate warning in the browser, this will also cause a number of warnings and errors in the proxy logs. Multiple this by X users and there can suddenly be thousands of errors and warnings per hour in the Sense logs.
If these are also sent to New Relic the data volumes increase quickly.

Overview of New Relic

New Relic is similar to InfluxDB in that Butler SOS pushes data to both systems.

The basic concepts are

• Metrics represent a measurement of some kind. Number or sessions in the Sense proxy, amount of free RAM on a Sense server etc.
• Events are something that happened. Warnings and errors in the Sense log files can be forwarded to New Relic as events.
  Various user activities (user session start/stop etc) in Sense can also be sent to New Relic as events.
• Attributes are conceptually tags that are attached to metrics or events. These act as dimensions for the data. Metrics in visualisations can be grouped by attributes, much in the same way Qlik Sense measurements are grouped by dimensions in Sense charts and tables.
  • Static attributes are defined in Butler SOS’ config file.
  • Dynamic attributes are determined at runtime.

In addition to the above these data formats exist but are not currently used by Butler SOS. This may change in the future.

• Logs are essentially regular lines in a log file, consisting of several fields.
• Distributed tracing collects data as requests travel from one service to another, recording each segment of the journey as a span. These spans contain important details about each segment of the request and are eventually combined into one trace. The completed trace gives you a picture of the entire request.

3.3 - Available Metrics: Prometheus

Prometheus

Metrics retrieved from the Sense servers can be stored in Prometheus. You don’t have to be a Prometheus expert to use Butler SOS, but understanding some basic concepts are helpful.

Storing metrics in Prometheus is not mandatory, but some kind of metrics storage - either in Prometheus, InfluxDB or New Relic - is needed to take full benefit of Butler SOS’ features.

Prometheus gathers metrics by “scraping” data from web pages (“endpoints”) on which metrics are displayed in a well specified format.
Most metrics from the Sense servers are exposed on a Prometheus compatible endpoint, but not all.
InfluxDB is more flexible for some types of data, while Prometheus provides more easily used features for data aggregation when data should be displaued in Grafana.

Prometheus endpoint

Prometheus is enabled/disabled in the Butler-SOS.prometheus section in the config file. Prometheus metrics are available on the /metrics URL on the IP and port specified in the config file.

For example, if the host is 0.0.0.0 and the port is 9842, Butler SOS will listen on port 9842 on all available network interfaces. If the Butler SOS’ server’s IP address is 192.168.1.168, a call from a web browser can look like this:

This is the web page Prometheus will scrape and ingest into it’s time-series database.

Overview of Prometheus

In contrast to InfluxDB, to which Butler SOS pushes data, Prometheus works the other way around.
The Prometheus server is responsible for gathering data exposed by the systems that should be monitored (for example Butler SOS).

The basic concepts are

• Metrics represent the measurements of interest. “fields” in InfluxDB.
• Labels are used to categorize metrics (similar to tags in InfluxDB).

Labels

The labels available for all Prometheus metrics are:

Metrics

Available metrics are similar to those in InfluxDB, with a few exceptions.

Prometheus is awesome when it comes to storing all kinds of measurements, but it doesn’t offer a good way to store strings.
For that reason Butler SOS metrics involving strings (for example list of apps loaded in memory) are not available on the Prometheus endpoint.
Most of the metrics come from Qlik Sense’ health check API.

Qlik Sense metrics

These are the Prometheus metrics exposed by Butler SOS:

Node.js metrics

A set of Node.js specific metrics are also available on Butler SOS’ Prometheus endpoint.
These are described in the “Default metrics” section on this page.