1 - Which config file to use

Butler SOS can use multiple config files. Here you learn to control which one is used by Butler SOS.

A description of the config file format is available here.

Select which config file to use

Butler SOS uses configuration files in YAML format.

Butler SOS comes with a default config file called production_template.yaml.
Make a copy of it, then rename the copy to butler-sos-config-prod.yaml, production.yaml, staging.yaml or something else suitable to your specific use case.

Update the config file as needed (see the config file reference page for details).

Trying to run Butler SOS with the default config file (the one included in the files download from GitHub) will not work - you must adapt it to your server environment. For example, you need to enter the IP or host name of you Sense server(s), the IP or host name where Butler SOS is running, where the Sense certificates are stored etc.

The name of the config file matters. Unless you specifically name which config to use when starting Butler SOS, it will look for an environment variable called “NODE_ENV” and then try to load a config file named with the value found in NODE_ENV.

Example 1:

  • Environment variable NODE_ENV = production
  • Butler SOS is started without specifying a config file: butler.exe --loglevel info
  • Butler SOS will look for a config file config/production.yaml.

Example 2:

  • Butler SOS is started with a command line option specifying a config file: butler.exe --configfile d:\some\path\butler-sos-config-prod.yaml --loglevel info
  • Butler SOS will not look at the NODE_ENV environment variable. Settings will be loaded from the butler-sos-config-prod.yaml instead.

Running several Butler SOS instances in parallel

If you have several Sense clusters (for example DEV, TEST and PROD environments) you can either monitor them all from a single Butler SOS instance, or set up separate instances for each Sense cluster.
The second case is implemented by creating several config files: butler-sos-dev.yaml, butler-sos-test.yaml and butler-sos-prod.yaml.

In this scenario three instances of Butler SOS should be started, each given a different config file by setting the NODE_ENV variable as needed when starting Butler SOS.
Or (this option is usually much easier!) use the --configfile command line option when starting Butler SOS.

Note: If running several Butler SOS instances in parallel, you must also ensure that each one uses unique port numbers for their respective UDP servers etc.

Setting environment variables

The method for setting environment variables varies between operating systems:

On Windows:

set NODE_ENV=production

Mac OS or Linux

export NODE_ENV=production

If using Docker, the NODE_ENV environment varible is set in the docker-compose.yml file (as already done in the template docker-compose files.)

2 - Config file verification

How to verify that the Butler SOS config file is valid.

A description of the config file format is available here.

Getting the config file correctly set up is usually the most challenging part of setting up Butler SOS.
The config file is written in an easy to read YAML format, but given the number of settings that can be configured, it can be a bit daunting to get it right.

Verify the config file

Config file verification is enabled by default.

Verification is done when Butler SOS is started, and if the config file is not valid, Butler SOS will not start.

Info

All settings in the config file are mandatory.

If you don’t want to use a specific Butler SOS feature, you must still include its settings in the config file, but you are free to disable the feature and set its setting to empty strings/values/arrays, or just leave the default values in place.

Skipping config file verification

If you want to skip config file verification, you can do so by starting Butler SOS with the ‘–skip-config-verificationsetting totrue` in the config file.

This will bypass all checks of the config file’s validity, and Butler SOS will try to start with the provided config file.
This can be useful if you are in the process of setting up Butler SOS and want to start it before the config file is complete, but for production scenarios it is recommended to leave config file verification enabled.

3 - Visualise the config file

Butler SOS can visualise its confile on a web page, using an internal web server.
This can be useful for troubleshooting and understanding how Butler SOS is configured.

The configuration can optionally be obfuscated to hide sensitive information.

What’s this?

Butler SOS can visualise its config file on a web page, using an internal web server. This can be useful for troubleshooting and understanding how Butler SOS is configured.

If enabled, the web server will serve a web page on the IP address and port specified in the config file.
The default IP address is localhost and the default port is 3100.

By clicking the “Download

JSON and YAML

The web page will show the config file in both JSON and YAML format.

The JSON format is useful if you want to copy the config file and paste it into a JSON validator, for example.
The YAML format is easier to read and understand for humans, and is also the format used in the config file.

Examples:

Butler SOS config file visualisation - JSON view

Butler SOS config file visualisation - YAML view

Obfuscation

The configuration can optionally be obfuscated to hide sensitive information.
This is useful if you need to share the config file with someone else, but don’t want to share sensitive information like IP addresses, user names or passwords.
Obfuscation is enabled/disabled in the config file.

For example, if asking for support on the Butler SOS forum, you can share the obfuscated config file without revealing sensitive information.

Disclaimer: Obfuscation is not foolproof, but it should be good enough for most use cases.
Always check the obfuscated config file before sharing it.

Settings in config file

Butler-SOS:
  ...
  ...
  # Should Butler SOS start a web server that serves an obfuscated view of the Butler SOS config file?
  configVisualisation:
    enable: true  
    host: localhost       # Hostname or IP address where the web server will listen. Should be localhost in most cases.
    port: 3100            # Port where the web server will listen. Change if port 3100 is already in use.
    obfuscate: true        # Should the config file shown in the web UI be obfuscated?
  ...
  ...

4 - Configuring Butler SOS logging

Butler SOS can log its activities to console and disk files.
Log files can be useful for retrospective troubleshooting of Butler SOS.

What’s this?

Butler SOS continuously logs what its doing.
Logging is always done to console and optionally also to disk files.

The top level section Butler-SOS in the config file has a set of settings that control logging.

Log level (verbosity) can be set, logging to disk can be enabled/disabled and the directory where log files are stored can be set.
Log level can also be set on the command line when starting Butler SOS, using the --loglevel option.

Log files are kept for 30 days, after which they are automatically deleted.

Settings in main config file

Butler-SOS:
  ...
  ...
  # Logging configuration
  logLevel: info          # Log level. Possible log levels are silly, debug, verbose, info, warn, error
  fileLogging: true       # true/false to enable/disable logging to disk file
  logDirectory: log       # Subdirectory where log files are stored
  ...
  ...

5 - Configuring Butler SOS heartbeats

Heartbeats provide a way to monitor that Butler SOS is running and working as intended.
Butler SOS can send periodic heartbeat messages to a monitoring tool, which can then alert if Butler SOS hasn’t checked in as expected.

What’s this?

A tool like Butler SOS should be viewed as mission critical, at least if it is used to monitor mission critical Sense apps.

But how can you know whether Butler SOS itself is working?
Somehow Butler SOS should be monitored.

Butler SOS (and most other tools in the Butler family) has a heartbeat feature.
It sends periodic messages to a monitoring tool, which can then alert if Butler SOS hasn’t checked in as expected.

Healthchecks.io is an example of such as tool. It’s open source and can be self-hosted, but also has a SaaS option if so preferred.

Uptime Kuma is another great tool that can be used, it has a somewhat slicker UI than Healthchecks.io - but it’s relally a matter of personal preference which one to use.

More info on using Healthchecks.io with Butler (Butler SOS works the same way) can be found in this blog post.

Settings in main config file

Butler-SOS:
  ...
  ...
  # Heartbeats can be used to send "I'm alive" messages to some other tool, e.g. an infrastructure monitoring tool
  # The concept is simple: The remoteURL will be called at the specified frequency. The receiving tool will then know 
  # that Butler SOS is alive.
  heartbeat:
    enable: true
    remoteURL: http://my.monitoring.server/some/path/
    frequency: every 1 hour         # https://bunkat.github.io/later/parsers.html#text
  ...
  ...

6 - Docker healthcheck

Docker has a concept of “health checks”, which is a way for Docker containers to tell the Docker runtime engine that the container is alive and well. Butler SOS can be configured to send such health check messages to Docker.

Note: Sending health check messages is only meaningful when running Butler SOS as a Docker container.

Settings in main config file

Butler-SOS:
  ...
  ...
  # Docker health checks are used when running Butler SOS as a Docker container. 
  # The Docker engine will call the container's health check REST endpoint with a set interval to determine
  # whether the container is alive/well or not.
  # If you are not running Butler SOS in Docker you can safely disable this feature. 
  dockerHealthCheck:
    enable: true                    # Control whether a REST endpoint will be set up to serve Docker health check messages
    port: 12398                     # Port the Docker health check service runs on (if enabled)
  ...
  ...

7 - Configuring Butler SOS uptime monitor

Butler SOS can optionally log how long it’s been running and how much memory it uses.
Optionally the memory usage can also be stored to an InfluxDB database or sent to New Relic, for later viewing/alerting in for example a Grafana dashboard or within New Relic.

What’s this?

In some cases - especially when investigating issues or bugs - it can be useful to get log messages telling how long Butler SOS has been running and how much memory it uses.

This feature is called “uptime monitoring” and can be enabled in the main config file. The feature is being added to more and more tools in the Butler family of tools for Qlik Sense.

The logging interval is configurable, as is the log level required for uptime messages to be shown in the console/file log.

Select a reasonable retention policy and logging frequency!
You will rarely if ever need to know how much memory Butler SOS used a month ago… A retention policy of 1-2 weeks is usually a good start, logging uptime metrics every few minutes.

InfluxDB

The memory usage data can optionally be written to InfluxDB, from where it can later be viewed in Grafana.
The metrics will be stored in the database specified in the Butler-SOS.influxdbConfig section of the config file.

Note that Butler-SOS.influxdbConfig.enable must also be set to true for any data to be sent to InfluxDB.

New Relic

Uptime metrics can be sent to zero or more New Relic accounts.

New Relic attributes (a concept where each data point sent to New Relic is tagged with a set of attributes) can be added to the metrics.
Attributes come in two forms: Static and dynamic.

  • Static attributes are hard-coded strings that don’t change over time. Could be used to distinguish metrics from DEV, TEST and PROD Sense environments.
  • Dynamic attributes may change each time Butler SOS is started, or even more often in the future if/when more dynamic attributes are added.
    An example is the Butler SOS version, which will change when Butler SOS is upgraded to a new version.

Log level

The log level does not affect storing uptime metrics in InfluxDB or New Relic.

Settings in main config file

Butler-SOS:
  ...
  ...
  # Uptime monitor
  uptimeMonitor:
    enable: true                    # Should uptime messages be written to the console and log files?
    frequency: every 15 minutes     # https://bunkat.github.io/later/parsers.html#text
    logLevel: verbose               # Starting at what log level should uptime messages be shown in console log and log files?
    storeInInfluxdb: 
      butlerSOSMemoryUsage: true    # Should data on Butler SOS' own memory use be stored in Infludb?
      instanceTag: PROD             # Tag that can be used to differentiate data from multiple Butler SOS instances
    storeNewRelic:
      enable: true
      destinationAccount:
        - First NR account
        - Second NR account
      metric:
        dynamic:
          butlerMemoryUsage:
            enable: true            # Should Butler SOS' memory/RAM usage be sent to New Relic?
          butlerUptime:
            enable: true            # Should Butler SOS' uptime (how long since it was started) be sent to New Relic?
      attribute: 
        static:                     # Static attributes/dimensions to attach to the data sent to New Relic.
          - name: metricType
            value: butler-sos-uptime
          - name: qs_service
            value: butler-sos
          - name: qs_environment
            value: prod
        dynamic:
          butlerVersion: 
            enable: true            # Should the Butler SOS version be included in the data sent to New Relic?
  ...
  ...

8 - Credentials to third party services

What’s this?

Butler SOS can interact with certain third party services, such as New Relic.
These services typically require some kind of authentication with associated credentials (username, password etc).
Those credentials are stored in the Butler-SOS.thirdPartyToolsCredentials section of the config file.

New Relic

Zero, one or more New Relic accounts with their respective credentials can be specified.
These accounts can then be used by Butler SOS’ various features.

Note that different Butler SOS features can send their data to different New Relic accounts.
This is specified in each feature’s section in the YAML config file.

Example:

  • Sense user events are sent to First NR account
  • Sense log events are sent to Second NR account
  • Sense RAM usage is sent to both First NR account and Second NR account

Note that the accountName setting is only used within Butler SOS to reference the diffferent New Relic accounts.
Specificallyt, it is not used by or within New Relic itself.

Settings in main config file

---
Butler-SOS:
  ...
  ...
  # Credentials for third party systems that Butler SOS integrate with.
  # These can also be specified via command line parameters when starting Butler SOS. 
  # Command line options takes precedence over settings in this config file.
  thirdPartyToolsCredentials:
    newRelic:         # Array of New Relic accounts/insert keys.
      - accountName: First NR account
        insertApiKey: <API key 1 (with insert permissions) from New Relic> 
        accountId: <New Relic account ID 1>
      - accountName: Second NR account
        insertApiKey: <API key 2 (with insert permissions) from New Relic> 
        accountId: <New Relic account ID 2>
  ...
  ...

9 - General Sense event settings

Butler SOS can act as a reciever of Qlik Sense events, sent as UDP messages from Qlik Sense Enterprise.
This section of the config file contains general settings for how Butler SOS should handle these events.

More specific settings for each event type (user, log, …) can be found in the respective sections of the config file.

What’s this?

Butler SOS can receive events from Qlik Sense Enterprise, sent as UDP messages.
Two kinds of events are supported: User events and log events.

  • User events are events that are generated when a user interacts with Qlik Sense, for example logging into Sense or opening an app.
  • Log events originate from the Sense logging framework itself, which is also responsible for logging things to Qlik Sense’s own log files.

Some aspects of these events are general in nature, i.e. shared between the different event types, and are configured in this section of the config file.

Counters for user and log events

If log and/or user events are enabled there is a risk that the number of events generated by a QSEoW cluster can be overwhelming.
To make it easier to understand the volume of events generated, Butler SOS can be configured to count the number of events generated by the Sense cluster.

The counters are stored in InfluxDB and can be used to create dashboards in Grafana.

Each InfluxDB datapoint has tags and fields as described in the reference section.

Rejected events

Butler SOS can be configured to reject certain events even though the event is correctly formatted and contains valid data.
The need to do this can arise if the Sense cluster generates a large number of events, and not all of them are relevant for the current monitoring use case.

For example, performance log events (event name qseow-qix-perf) will be emitted by Sense every 2 seconds during scheduled reloads. There is rarely a need to store all these events in InfluxDB, so they can be filtered out (=rejected) by Butler SOS.
But they can also be seen as valid and stored in InfluxDB, depending on the use case.

Rejected events are counted and the counters stored in InfluxDB.
They can be used to understand how many events are rejected by Butler SOS, versus how many are received in total (see the section above).

The InfluxDB measurement name is defined in the config file, Butler-SOS.qlikSenseEvents.rejectedEventCount.influxdb.measurementName.

Each event type and name may have its own rejection settings, defined in the respective sections of the config file.
They may also store different tags and fields in InfluxDB.

The currently defined rejection settings are:

Performance log events

These events come from the Qlik associative engine (the “QIX engine”) an contain very detailed performance data about apps, app objects, charts, user selections in apps etc.

Counters for rejected performance log events are enabled via the Butler-SOS.logEvents.enginePerformanceMonitor.trackRejectedEvents.enable setting in the config file.

Once this data is in InfluxDB it can be used in Grafana dashboards, for example showing how long each app takes to open:

Average time to open Sense apps

The data stored in InfluxDB for performance log events is described here.

Settings in main config file

---
Butler-SOS:
  ...
  ...
  # Shared settings for user and log events (see below)
  qlikSenseEvents:                  # Shared settings for user and log events (see below)
    influxdb:
      enable: false                 # Should summary (counter) of user/log events, and rejected events be stored in InfluxDB?
      writeFrequency: 20000         # How often (milliseconds) should rejected event count be written to InfluxDB?  
    eventCount:                     # Track how many events are received from Sense.
                                    # Some events are valid, some are not. Of the valid events, some are rejected by Butler SOS
                                    # based on the configuration in this file. 
      enable: false                 # Should event count be stored in InfluxDB?
      influxdb:
        measurementName: event_count # Name of the InfluxDB measurement where event count is stored
        tags:                       # Tags are added to the data before it's stored in InfluxDB
          - name: env
            value: DEV
          - name: foo
            value: bar
    rejectedEventCount:             # Rejected events are events that are received from Sense, that are correctly formatted, 
                                    # but that are rejected by Butler SOS based on the configuration in this file. 
                                    # An example of a rejected event is a performance log event that is filtered out by Butler SOS.
      enable: false                 # Should rejected events be counted and stored in InfluxDB?
      influxdb:
        measurementName: rejected_event_count # Name of the InfluxDB measurement where rejected event count is stored
  ...
  ...

Log appender XML files

Sample log appender files are available in the ZIP file available from the download page, in subfolders engine/proxy/repository/scheduler of config/log_appender_xml/ folder.

Note that the log appender files contain slightly different information for each Sense service (engine/proxy/repository/scheduler)!
Also keep in mind that the log appender files must be called LocalLogConfig.xml and placed in these directories on the all Sense servers (assuming the detfault installation path of Qlik Sense):

  • C:\ProgramData\Qlik\Sense\Engine
  • C:\ProgramData\Qlik\Sense\Proxy
  • C:\ProgramData\Qlik\Sense\Repository
  • C:\ProgramData\Qlik\Sense\Scheduler

Tip

If you have more than one Sense server you strictly speaking don’t have to deploy log appenders to all servers.

If you are only interested in receiving log events from some servers and/or services (engine, proxy, repository, scheduler) - deploy the log appender files there.

9.1 - Configuring user events

What’s this?

User events are among the most detailed bits of information retrieved from Sense by Butler SOS.
They capture session start/stop events (=users logging in/out) and connection open/close events (apps opened/closed in browser tabs).

These events rely on two things to be correctly configured:

  1. Settings in Butler SOS’ config file.
  2. Log appender XML file(s) being deployed on the Sense server(s) where user activity events should be captured.

Both are described below.

Tech deep-dive

The user events are created by hooking into Sense’s logging framework, which is called Log4Net.

By placing a carefully crafted XML file in the Qlik Sense proxy service’s configuration directory, we can instruct Log4Net to forward certain Sense log events that we are interested in to Butler SOS.
In this case we are interested in session start/stop and connection open/close events.

The XML file is also known as a “log appender file”.
It contains instructions that tell Log4Net to do various things when the specified filter matches the actual log data created by Sense. Examples include sending emails, writing log entries to disk (i.e. regular file logging!), sending the log row as a UDP message and more.
Here we’re interested in the UDP message feature.

So, by means of a log appender file we tell Log4Net to send certain log rows to Butler SOS as UDP messages.

We also have to specify in the log appender file what host/IP address and port Butler SOS listens to, i.e. where the UDP messages should be sent.
Finally we have to make sure firewalls are open and allow UDP traffic from the Sense server(s) to Butler SOS.

If everything is set up correctly UDP messages will arrive at Butler SOS within seconds after the actual event taking place in Qlik Sense, i.e. close to real-time.

Tagging of data

InfluxDB

The tags added to InfluxDB are described in the reference documentation for log events.

New Relic

The following attributes (which is New Relic lingo for tags) are added:

  1. A core set of attributes are added to all user events
    1. qs_host: Host name of the Sense server the event originated at.
    2. qs_event_action: What kind of user event that took place. Examples are “Start session”, “Stop session, “Open connection”, “Close connection”.
    3. qs_userFull: Full directory/user ID of the user the event is about. Will be scrambled if scrambling enabled in config file.
    4. qs_userDirectory: User directory of the user the event is about. Will be scrambled if scrambling enabled in config file.
    5. qs_userId: User ID of the user the event is about. Will be scrambled if scrambling enabled in config file.
    6. qs_origin: What kind of activity caused the event, for example “AppAccess”. May be empty for some user events.
    7. qs_appId: App ID of the app the event is about. May be empty for some user events.
    8. qs_appName: App name of the app the event is about. May be empty for some user events.
    9. qs_uaBrowserName: Browser name of the user agent that caused the event.
    10. qs_uaBrowserMajorVersion: Browser major version of the user agent that caused the event.
    11. qs_uaOsName: OS name of the user agent that caused the event.
    12. qs_uaOsVersion: OS version of the user agent that caused the event.
  2. Custom attributes defined in the Butler SOS config file’s Butler-SOS.userEvents.tags section of the config file.

Note: Attributes defined further down in the list above will overwrite already defined attributes if their names match.
To avoid problems you should make sure not to use already defined attributes.

Settings in config file

---
Butler-SOS:
  ...
  ...
  # Track individual users opening/closing apps and starting/stopping sessions. 
  # Requires log appender XML file(s) to be added to Sense server(s).
  userEvents:                       
    enable: false
    excludeUser:                    # Optional blacklist of users that should be disregarded when it comes to user events
      - directory: LAB
        userId: testuser1
      - directory: LAB
        userId: testuser2
    udpServerConfig:
      serverHost: <IP or FQDN>      # Host/IP where user event server will listen for events from Sense
      portUserActivityEvents: 9997  # Port on which user event server will listen for events from Sense
    tags:                           # Tags are added to the data before it's stored in InfluxDB
      - tag: env
        value: DEV
      - tag: foo
        value: bar
    sendToMQTT: 
      enable: false                 # Set to true if user events should be forwarded as MQTT messages
      postTo:                       # Control when and to which MQTT topics messages are sent 
        everythingTopic:            # Topic to which all user events are sent
          enable: true
          topic: qliksense/userevent
        sessionStartTopic:          # Topic to which "session start" events are sent
          enable: true
          topic: qliksense/userevent/session/start
        sessionStopTopic:           # Topic to which "session stop" events are sent
          enable: true
          topic: qliksense/userevent/session/stop
        connectionOpenTopic:        # Topic to which "connection open" events are sent
          enable: true
          topic: qliksense/userevent/connection/open
        connectionCloseTopic:       # Topic to which "connection close" events are sent
          enable: true
          topic: qliksense/userevent/connection/close
    sendToInfluxdb:
      enable: true                  # Set to true if user events should be stored in InfluxDB
    sendToNewRelic:
      enable: false                  # Should log events be sent to New Relic?
      destinationAccount:
        - First NR account
        - Second NR account
      scramble: true                # Should user info (user directory and user ID) be scrambled before sent to NR?
  ...
  ...

Log appender XML files

A sample log appender file LocalLogConfig.xml is available in the ZIP file available from the download page, in the config/log_appender_xml/proxy/LocalLogConfig.xml folder.

That file includes log appenders for both user and log events.
Looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <!-- Log appender finding user session events -->
    <appender name="EventSession" type="log4net.Appender.UdpAppender">
        <filter type="log4net.Filter.StringMatchFilter">
            <param name="stringToMatch" value="Start session for user" />
        </filter>
        <filter type="log4net.Filter.StringMatchFilter">
            <param name="stringToMatch" value="Stop session for user" />
        </filter>
        <filter type="log4net.Filter.DenyAllFilter" />
        <param name="remoteAddress" value="FQDN or IP of server where Butler SOS is running" />
        <param name="remotePort" value="9997" />
        <param name="encoding" value="utf-8" />
        <layout type="log4net.Layout.PatternLayout">
            <converter>
                <param name="name" value="hostname" />
                <param name="type" value="Qlik.Sense.Logging.log4net.Layout.Pattern.HostNamePatternConverter" />
             </converter>
            <param name="conversionpattern" value="/qseow-proxy-session/;%hostname;%property{Command};%property{UserDirectory};%property{UserId};%property{Origin};%property{Context};%message" />
        </layout>
    </appender>

    <!-- Log appender finding user connection events -->
    <appender name="EventConnection" type="log4net.Appender.UdpAppender">
        <filter type="log4net.Filter.StringMatchFilter">
            <param name="stringToMatch" value="connection Opened for session" />
        </filter>
        <filter type="log4net.Filter.StringMatchFilter">
            <param name="stringToMatch" value="connection Closed for session" />
        </filter>
        <filter type="log4net.Filter.DenyAllFilter" />
        <param name="remoteAddress" value="FQDN or IP of server where Butler SOS is running" />
        <param name="remotePort" value="9997" />
        <param name="encoding" value="utf-8" />
        <layout type="log4net.Layout.PatternLayout">
            <converter>
                <param name="name" value="hostname" />
                <param name="type" value="Qlik.Sense.Logging.log4net.Layout.Pattern.HostNamePatternConverter" />
             </converter>
            <param name="conversionpattern" value="/qseow-proxy-connection/;%hostname;%property{Command};%property{UserDirectory};%property{UserId};%property{Origin};%property{Context};%message" />
        </layout>
    </appender>

    <!-- Generic appender for detecting warnings and errors -->
    <appender name="LogEvent" type="log4net.Appender.UdpAppender">
        <param name="threshold" value="warn" />
        <param name="remoteAddress" value="FQDN or IP of server where Butler SOS is running" />
        <param name="remotePort" value="9996" />
        <param name="encoding" value="utf-8" />
        <layout type="log4net.Layout.PatternLayout">
            <converter>
                <param name="name" value="rownum" />
                <param name="type" value="Qlik.Sense.Logging.log4net.Layout.Pattern.CounterPatternConverter" /> 
            </converter> 
            <converter>
                <param name="name" value="hostname" />
                <param name="type" value="Qlik.Sense.Logging.log4net.Layout.Pattern.HostNamePatternConverter" />
            </converter>
            <converter>
                  <param name="name" value="longIso8601date" /> 
                  <param name="type" value="Qlik.Sense.Logging.log4net.Layout.Pattern.Iso8601TimeOffsetPatternConverter" /> 
            </converter>
            <converter> 
                  <param name="name" value="user" /> 
                  <param name="type" value="Qlik.Sense.Logging.log4net.Layout.Pattern.ServiceUserNameCachedPatternConverter" /> 
            </converter> 
            <converter> 
                  <param name="name" value="encodedmessage" /> 
                  <param name="type" value="Qlik.Sense.Logging.log4net.Layout.Pattern.EncodedMessagePatternConverter" /> 
            </converter> 
            <converter> 
                  <param name="name" value="encodedexception" /> 
                  <param name="type" value="Qlik.Sense.Logging.log4net.Layout.Pattern.EncodedExceptionPatternConverter" /> 
            </converter>
            <param name="conversionpattern" value="/qseow-proxy/;%rownum{9999};%longIso8601date;%date;%level;%hostname;%logger;%user;%encodedmessage;%encodedexception;%property{UserDirectory};%property{UserId};%property{Command};%property{Result};%property{Origin};%property{Context}" />
        </layout>
    </appender>

    <!-- Send UDP message to Butler SOS on user activity -->
    <logger name="AuditActivity.Proxy">
        <appender-ref ref="EventSession" />
        <appender-ref ref="EventConnection" />
    </logger>

    <!-- Send UDP message to Butler SOS on warnings and errors -->
    <logger name="Audit.Proxy">
        <appender-ref ref="LogEvent" />
    </logger>
    <logger name="AuditSecurity">
        <appender-ref ref="LogEvent" />
    </logger>
    <logger name="Security.Proxy">
        <appender-ref ref="LogEvent" />
    </logger>
    <logger name="System.Proxy">
        <appender-ref ref="LogEvent" />
    </logger>
</configuration>

Tip

If you have several servers in your Sense cluster you probably need several log appender files too.

More specifically, you should put a log appender file on each server where the Qlik Sense proxy service is running, i.e. on all servers via which end users access the Sense cluster.

Note the places where you need to fill in the IP/host where Butler SOS is running, as well as the port number to use (set to 9997 but can be changed if needed).

Make necessary changes so the file matches your environment, then deploy to C:\ProgramData\Qlik\Sense\Proxy\LocalLogConfig.xml (adapt path if you have a different installation path).
Note that the file must be called LocalLogConfig.xml!

Sense will usually detect and use the file without any restarts needed, but it can take a while. You can always restart the Sense proxy service to make sure the XML file is applied and used.

Once in place you should see events in the Butler SOS console/file logs if you set logging level to verbose, debug or silly.

9.2 - Configuring log events

What’s this?

Butler SOS log events are designed to be a replacement for the most important/useful aspects of Qlik Sense’ log database, which was removed from Qlik Sense Enterprise on Windows in mid 2021.

The log events capture warnings, errors and fatals from the various QSEoW subsystems.
These events used to be sent to the PostgreSQL logging database, most (but not all) are also sent to QSEoW’s log files.

Using Butler SOS’ log events is arguably even better than getting the same information from log db:
Log db had to be polled to detect new log events and this polling could realistically only be done every few minutes. It also put additional load on an often already struggling part of many QSEoW clusters.

With Butler SOS’ log events concept the notifications are almost instantaneous.
Errors and warnings show up in the Grafana or New Relic dashboards within seconds after taking place in QSEoW.

Log events rely on two things to work:

  1. Settings in the Butler SOS config file.
  2. Log appender XML files being deployed on the Sense servers where log events should be captured.

Both are described below.

Info

As of Butler SOS version 9.2, log events are captured in these QSEoW services:

  • Engine
  • Proxy
  • Repository
  • Scheduler

Support for additional modules is reasonably easy to add, please create a ticket if you believe some service should be added to the list above.

Tech deep-dive

The underlying mechanism is the same as described on the user events page.

Tagging of data

Categorising log events

Log events can optionally be categorised by Butler SOS.
The reason for categorising log events is to make it easier to make Sense of the potentially large number of log events that can be generated by a QSEoW cluster.

For example, if a QSEoW cluster creates 1000 warnings and errors per hour, it’s difficult to know which warnings are important and which are not.
By categorising the messages, for example by the subsystem that generated them and/or what they refer to, it’s easier to understand what’s going on.

Possible ways of categorising log events could be “access denied” issues, “user directory” issues, “app reload failed” issues, “general engine issues” etc.

Log events can be categorised in any number of ways, but the template config file contains a few examples.
Specifically:

  1. Access denied issues
    1. Captures log events of severity WARN and ERROR.
    2. Match log events aginst two filters.
      1. Log message starts with “Access was denied for User:”.
      2. Log message contains “was denied for User
    3. Categorises them by setting category tag qs_log_category to access-denied.
  2. AD issues
    1. Captures log events of severity WARN and ERROR.
    2. Match log events against one filter.
      1. Log message starts with “Duplicate entity with userId”.
    3. Categorises them by setting category tag qs_log_category to user-directory.
  3. Qlik Sense service down
    1. Captures log events of severity WARN.
    2. Match log events against two filters.
      1. Log message starts with “Failed to request service alive response from”.
      2. Log message contains “Unable to connect to the remote server”.
    3. Categorises them by setting category tag qs_log_category to qs-service.
  4. Reload task failed
    1. Captures log events of severity WARN and ERROR.
    2. Match log events against four filters.
      1. Log message starts with “Task finished with state FinishedFail”.
      2. Log message starts with “Task finished with state Error”.
      3. Log message ends with “Reload failed in Engine. Check engine or script logs.”.
      4. Log message starts with “Reload sequence was not successful (Result=False, Finished=True, Aborted=False) for engine connection with handle”.
    3. Categorises them by setting category tag qs_log_category to reload-failed.
  5. If no rules match the log event, it will be categorised as unknown.
    1. The default rule can be enabled/disabled in the config file via the ruleDefault.enable parameter.

Note 1: It is possible to assign one or more categories to a log event. This provides flexibility in how you later create dashboards in for example Grafana or New Relic.

Note 2: The sample/template config file may be updated with more examples in the future.

Tip

It is also possible to drop log events that match a certain pattern, for example if you are never interested in seeing them in Grafana or New Relic.
This is done by setting the action parameter to drop in the config file, for that particular rule.

InfluxDB

The tags added to InfluxDB are described in the reference documentation for log events.

Log categories are added as tags to InfluxDB datapoints.

New Relic

The following attributes (which is New Relic lingo for tags) are added:

  1. A core set of attributes are added to all user events. Note that some attributes will be empty for some/many log events.
    1. qs_ts_iso: Event timestamp in ISO format.
    2. qs_ts_local: Event timestamp in local (server) time zone.
    3. qs_log_source: Which Sense service the event originated in, for example “qseow-proxy”, “qseow-repository”.
    4. qs_log_level: Log level of the event. “WARN”, “ERROR”, or “FATAL”.
    5. qs_host: Host name of the Sense server the event originated at.
    6. qs_subsystem: Which part of each Sense service the event originated in, for example “System.Proxy.Proxy.Core.RequestListener”, “System.Engine.Engine”.
    7. qs_windows_user: Name of the Windows user that’s used to run the Window service where the event originated.
    8. qs_message: Event message.
    9. qs_exception_message: Additional information about the event.
    10. qs_user_full: Full directory/user ID of the user the event is about. Will be scrambled if scrambling enabled in config file.
    11. qs_user_directory: User directory of the user the event is about. Will be scrambled if scrambling enabled in config file.
    12. qs_user_id: User ID of the user the event is about. Will be scrambled if scrambling enabled in config file.
    13. qs_command: What command (if any) caused the event. Example: “Doc::DoSave”, “Doc::CreateObject”.
    14. qs_result_code: Result code as reported by Sense. Usually empty.
    15. qs_origin: What kind of activity caused the event, for example “AppAccess”.
    16. qs_context: Additional information about the event.
    17. qs_task_name: Task name (if any) causing the event.
    18. qs_app_name: App name (if any) causing the event.
    19. qs_task_id: Task ID (if any) causing the event.
    20. qs_app_id: App ID (if any) causing the event.
    21. qs_execution_id: Execution ID as reported by Sense.
    22. qs_proxy_session_id: Proxy session ID as reported by Sense.
    23. qs_engine_ts: Engine timestamp (if any) associated with the event.
    24. qs_process_id: Process ID of engine service.
    25. qs_engine_exe_version: Version of engine service’s EXE file.
    26. qs_server_started: Timestamp when Sense engine service was started.
    27. qs_entry_type: Entry type as reported by Sense. Usually empty.
    28. qs_session_id: Session ID as reported by Sense.
  2. Custom attributes defined in the Butler SOS config file’s Butler-SOS.logEvents.tags section.
  3. Custom attributes defined in the Butler SOS config file’s Butler-SOS.newRelic.event.attribute.static section.
  4. Dynamic attributes
    1. butlerSosVersion: Butler SOS version. Enabled by setting Butler-SOS.newRelic.event.attribute.dynamic.butlerSosVersion.enable to true in config file.

Note: Attributes defined further down in the list above will overwrite already defined attributes if their names match.
To avoid problems you should make sure not to use already defined attributes.

Log categories are currently NOT included in the data sent to New Relic.

Settings in main config file

Tip

The config snippet below comes from the production_template.yaml file.

Being a template, it contains examples on how configuration may be done - not necessarily how it should be done.
For example, the env/DEV and foo/bar tags are optional and can be changed to something else, or removed all together if not used.

Butler-SOS:
  ...
  ...
  # Log events are used to capture Sense warnings, errors and fatals in real time
  logEvents:
    udpServerConfig:
      serverHost: <IP or FQDN>         # Host/IP where log event server will listen for events from Sense
      portLogEvents: 9996              # Port on which log event server will listen for events from Sense
    tags:
      # - name: env
      #   value: DEV
      # - name: foo
      #   value: bar
    source:
      engine:
        enable: false                  # Should log events from the engine service be handled?
      proxy:
        enable: false                  # Should log events from the proxy service be handled?
      repository:
        enable: false                  # Should log events from the repository service be handled?
      scheduler:
        enable: false                  # Should log events from the scheduler service be handled?
    categorise:                        # Take actions on log events based on their content
      enable: false
      rules:                           # Rules are used to match log events to filters
        # - description: Find access denied errors
        #   logLevel:                    # Log events of this Log level will be matched. WARN, ERROR, FATAL. Case insensitive.
        #     - WARN
        #     - ERROR
        #   action: categorise           # Action to take on matched log events. Possible values are categorise, drop
        #   category:                    # Category to assign to matched log events. Name/value pairs. 
        #                                # Will be added to InfluxDB datapoints as tags.
        #     - name: qs_log_category
        #       value: access-denied
        #   filter:                      # Filter used to match log events. Case sensitive.
        #     - type: sw                 # Type of filter. sw = starts with, ew = ends with, so = substring of
        #       value: "Access was denied for User:"
        #     - type: so
        #       value: was denied for User
        # - description: Find AD issues
        #   logLevel:                    # Log events of this Log level will be matched. WARN, ERROR, FATAL. Case insensitive.
        #     - ERROR
        #     - WARN
        #   action: categorise           # Action to take on matched log events. Possible values are categorise, drop
        #   category:                    # Category to assign to matched log events. Name/value pairs. 
        #                                # Will be added to InfluxDB datapoints as tags.
        #     - name: qs_log_category
        #       value: user-directory
        #   filter:                      # Filter used to match log events. Case sensitive.
        #     - type: sw                 # Type of filter. sw = starts with, ew = ends with, so = substring of
        #       value: Duplicate entity with userId
        # - description: Qlik Sense service down
        #   logLevel:                    # Log events of this Log level will be matched. WARN, ERROR, FATAL. Case insensitive.
        #     - WARN
        #   action: categorise           # Action to take on matched log events. Possible values are categorise, drop
        #   category:                    # Category to assign to matched log events. Name/value pairs. 
        #                                # Will be added to InfluxDB datapoints as tags.
        #     - name: qs_log_category
        #       value: qs-service
        #   filter:                      # Filter used to match log events. Case sensitive.
        #     - type: sw                 # Type of filter. sw = starts with, ew = ends with, so = substring of
        #       value: Failed to request service alive response from
        #     - type: so                 # Type of filter. sw = starts with, ew = ends with, so = substring of
        #       value: Unable to connect to the remote server
        # - description: Reload task failed
        #   logLevel:                    # Log events of this Log level will be matched. WARN, ERROR, FATAL. Case insensitive.
        #     - WARN
        #     - ERROR
        #   action: categorise           # Action to take on matched log events. Possible values are categorise, drop
        #   category:                    # Category to assign to matched log events. Name/value pairs. 
        #                                # Will be added to InfluxDB datapoints as tags.
        #     - name: qs_log_category
        #       value: reload-failed
        #   filter:                      # Filter used to match log events. Case sensitive.
        #     - type: sw                 # Type of filter. sw = starts with, ew = ends with, so = substring of
        #       value: Task finished with state FinishedFail
        #     - type: sw                 # Type of filter. sw = starts with, ew = ends with, so = substring of
        #       value: Task finished with state Error
        #     - type: ew                 # Type of filter. sw = starts with, ew = ends with, so = substring of
        #       value: Reload failed in Engine. Check engine or script logs.
        #     - type: sw                 # Type of filter. sw = starts with, ew = ends with, so = substring of
        #       value: Reload sequence was not successful (Result=False, Finished=True, Aborted=False) for engine connection with handle
      ruleDefault:                     # Default rule to use if no other rules match the log event
        enable: true
        category:
          - name: qs_log_category
            value: unknown
    enginePerformanceMonitor:           # Detailed app performance data extraction from log events
      enable: false                     # Should app performance data be extracted from log events?
      appNameLookup:                    # Should app names be looked up based on app IDs?
        enable: false
      trackRejectedEvents: 
        enable: false                   # Should events that are rejected by the app performance monitor be tracked?
        tags:                           # Tags are added to the data before it's stored in InfluxDB
          # - name: env
          #   value: DEV
          # - name: foo
          #   value: bar
      monitorFilter:                    # What objects should be monitored? Entire apps or just specific object(s) within some specific app(s)?
                                        # Two kinds of monitoring can be done:
                                        # 1) Monitor all apps, except those listed for exclusion. This is defined in the allApps section.
                                        # 2) Monitor only specific apps. This is defined in the appSpecific section.
                                        # An event will be accepted if it matches any of the rules in the allApps section OR any of the rules in the appSpecific section.
        allApps:
          enable: false                 # Should all apps be monitored?
          appExclude:                   # What apps should be excluded from monitoring?
                                        # If both appId and appName are specified, both must match the event's data for it to be considered a match.
            # - appId: 5b817efe-472d-43ce-8a31-6cce34af7de9
            # - appName: Sales forecast
            # - appId: f42d6b16-8faf-45ca-a783-59f9da47db6e
            #   appName: Inventory analysis
          objectType:
            allObjectTypes: true        # Should all object types be monitored?
            allObjectTypesExclude:      # If allObjectTypes is set to true, the object types in this array are excluded from monitoring. 
                                        # someObjectTypesInclude (below) is ignored in that case.
              # - LoadModelList
              # - <Unknown>
              # - linechart
              # - map
            someObjectTypesInclude:     # What object types should be included in monitoring?
                                        # Only applicable if allObjectTypes is set to false.
              # - LoadModelList
              # - sheet
              # - barchart
          method:
            allMethods: true            # Should all methods be monitored?
            allMethodsExclude:          # If allMethods is set to true, the methods in this array are excluded from monitoring.
                                        # someMethodsInclude (below) is ignored in that case.
              # - Global::OpenApp
              # - Doc::GetAppLayout
              # - Doc::CreateSessionObject
            someMethodsInclude:         # What methods should be included in monitoring?
                                        # Only applicable if allMethods is set to false.
              # - GenericObject::GetLayout
              # - GenericObject::GetHyperCubeContinuousData
        appSpecific:
          enable: false                  # Should app specific monitoring be done?
          app:
            - include:                  # What apps should be monitored?
                                        # If both appId and appName are specified, both must match the event's data for it to be considered a match.
                # - appId: d7cf16f9-6a95-462a-9ff1-a6d413326de4
                # - appName: Budget 2025
                # - appId: 6931136d-c234-4358-a40c-e37153aba7c9
                #   appName: Sales basket analysis
              objectType:
                allObjectTypes: true   # Should all object types be monitored?
                allObjectTypesExclude:  # If allObjectTypes is set to true, the object types in this array are excluded from monitoring. 
                                        # someObjectTypesInclude (below) is ignored in that case.
                  # - table
                  # - map
                someObjectTypesInclude: # What object types should be included in monitoring?
                                        # Only applicable if allObjectTypes is set to false.
                  # - sheet
                  # - barchart
                  # - linechart
                  # - map
              appObject:
                allAppObjects: true     # Should all app objects be monitored?
                allAppObjectsExclude:   # If allAppObjects is set to true, the app objects in this array are excluded from monitoring.
                                        # someAppObjectsInclude (below) is ignored in that case.
                  # - objectId: AaBbCc
                  # - objectId: DdEeFf
                someAppObjectsInclude:  # What app objects should be included in monitoring?
                                        # Only applicable if allAppObjects is set to false.
                  # - objectId: YJEpPT    
              method: 
                allMethods: true        # Should all methods be monitored?
                allMethodsExclude:      # If allMethods is set to true, the methods in this array are excluded from monitoring.
                                        # someMethodsInclude (below) is ignored in that case.
                  # - Global::OpenApp
                  # - Doc::GetAppLayout
                  # - Doc::CreateSessionObject
                someMethodsInclude:     # What methods should be included in monitoring?
                                        # Only applicable if allMethods is set to false.
                  # - GenericObject::GetLayout
                  # - GenericObject::GetHyperCubeContinuousData
    sendToMQTT:
      enable: false                    # Should log events be sent as MQTT messages?
      baseTopic: qliksense/logevent    # What topic should log events be forwarded to? 
      postTo:
        baseTopic: true
        subsystemTopics: true          # Should log events be sent to subtopics corresponding to the QSEoW subsystems where the events originated?
    sendToInfluxdb:
      enable: false                    # Should log events be stored in InfluxDB?
    sendToNewRelic:
      enable: false                    # Should log events be sent to New Relic?
      destinationAccount:
        # - First NR account
        # - Second NR account
      source:
        engine:
          enable: true                 # Should log events from the engine service be handled?
          logLevel: 
            error: true                # Should error level log events be handled by Butler SOS?
            warn: true                 # Should warning level log events be handled by Butler SOS?
        proxy:
          enable: true                 # Should log events from the proxy service be handled?
          logLevel: 
            error: true                # Should error level log events be handled by Butler SOS?
            warn: true                 # Should warning level log events be handled by Butler SOS?
        repository:
          enable: true                 # Should log events from the repository service be handled?
          logLevel: 
            error: true                # Should error level log events be handled by Butler SOS?
            warn: true                 # Should warning level log events be handled by Butler SOS?
        scheduler:
          enable: true                 # Should log events from the scheduler service be handled?
          logLevel: 
            error: true                # Should error level log events be handled by Butler SOS?
            warn: true                 # Should warning level log events be handled by Butler SOS?
  ...
  ...

9.2.1 - Engine performance log events

Butler SOS can capture detailed performance data from the Qlik Sense engine service, based on log events generated by that service.

Due to the risk of generating a large number of log events, this feature comes with a comprensive set of settings to control what data is captured and stored in InfluxDB, and what is rejected.

Warning

Due to the risk of overwhelming the InfluxDB database with a large number of performance log events, this feature is disabled by default in the sample config file.
If you want to use this feature, you must enable it first.

It’s also recommended to start small and gradually increase the number of performance log events captured, to make sure the InfluxDB database can handle the load.

For example, start by capturing only a few specific types of objects types, methods etc for a limited set of apps, and then gradually increase/customise from there if needed.

What’s this?

“Performance log events” are a regular log events generated by the Qlik Sense engine service.
They get a special name in Butler SOS because they contain detailed performance data about the engine’s operations, something that can be very useful when monitoring the performance of the Qlik Sense engine.

These events make it possible to monitor how the engine is performing in close to real time.

Butler SOS can capture all or some of these events and store them in InfluxDB, where they can be used to create dashboards in Grafana - just like any other events or metrics captured by Butler SOS.

How it works

Once a log event has been identified as a performance log event (by looking at the event name, which is qseow-qix-perf in this case), Butler SOS will extract the data of interest from the event.

The data is then checked against a set of rules/filters to determine if the event should be stored in full detail in InfluxDB (=accepted) or not (=rejected).

For rejected events, a counter is incremented to keep track of how many events have been rejected.
The counters are written to InfluxDB at regular intervals, controlled by the Butler-SOS.qlikSenseEvents.influxdb.writeFrequency setting in the config file.

Once the data has been accepted, it is immediately stored in InfluxDB with a set of tags and fields, as described below.

Filtering of log events

There are two kinds of filters that can be defined in the config file:

  • Filters that apply to all apps
  • App specific filters that apply to specific apps

An event will be accepted if it matches either or both of the above filter types.
If an event does not match either of the two filter types, it will be rejected.

Filters applying to all apps

Global filters are defined in the Butler-SOS.logEvents.enginePerformanceMonitor.monitorFilter.allApps section of the config file.
All references to config file settings below are relative to this section.

The all-app filter consists of several parts/subfilters:

  • Apps for which events should be excluded, based on app ID and/or app name
  • Object types to include/exclude
  • Methods to include/exclude

An event must match all of the subfilters to be accepted by the all-app filter.

Enabling/disabling

It is possible to enabled/disable monitoring of all apps via the enable setting. If disabled no events will be accepted via the global all-apps filter type.

Excluding apps

The appExclude[] array can be used to exclude specific apps from monitoring.
The effect is that events from all apps, except the apps listed in appExclude[], will be accepted (unless the event is rejected by some of the other all-app filters).

appExclude[] can contain zero or more objects, each with either an appId or an appName properties, or both.
If both properties are present, both must match the event’s data for it to be considered a match.

Filtering by object type

The objectType section can be used to filter events based on the object type of the event.
“Object types” are things like barchart, sheet, map, but also internal Sense objects like AppPropsList and appprops.

  • If objectType.allObjectTypes is set to true, all object types are monitored, except those listed in allObjectTypesExclude[].
    • someObjectTypesInclude[] is not used in this case.
  • If objectType.allObjectTypes is set to false, only the object types listed in someObjectTypesInclude[] will result in events being accepted.
    • allObjectTypesExclude[] is not used in this case.

Put differently: You can either start by including all object types and then specify which ones to exclude, or start with an empty list and add the object types you want to include.

Filtering by method

“Methods” refer to the various operations that the Sense engine performs.
Examples include Global::OpenApp, Global::GetProgress, Doc::GetAppLayout, Doc::CreateSessionObject.

The concepts for filtering by method are the same as for filtering by object type (see above).

Filter applying to specific apps

If the all-app filters are not useful (they may generate too much data, for example), it is possible to define app-specific filters in the Butler-SOS.logEvents.enginePerformanceMonitor.monitorFilter.appSpecific section of the config file.
All references to config file settings below are relative to this section.

App-specific filters are defined in the array app[].
It includes zero or more objects, each of which contains the following subfilters:

  • Apps to include, based on app ID and/or app name.
  • Object types to include/exclude
  • App objects to include/exclude
  • Methods to include/exclude

The concepts for including all (and then excluding some) or starting with an empty list and adding what you want to include are the same as for the all-app filters.

Enabling/disabling

It is possible to enabled/disable app specific monitoring via the enable setting. If disabled no events will be accepted via the app specific filter type.

Including apps

The app[].include[] contains the app IDs and/or app names of the apps to include in the filter.

Example:

appSpecific:
  enable: true                  # Should app specific monitoring be done?
  app:
    - include:                  # What apps should be monitored?
                                # If both appId and appName are specified, both must match the event's data for it to be considered a match.
        - appId: d7cf16f9-6a95-462a-9ff1-a6d413326de4
        - appName: Budget 2025
        - appId: 6931136d-c234-4358-a40c-e37153aba7c9
          appName: Sales basket analysis

Filtering by object type

The objectType section can be used to filter events based on the object type of the event.

The concepts for filtering by object type are the same as for the all-app filters.

Filtering by app object

The appObject section can be used to filter events based on the object ID of the app object that the event is related to.
This is useful if you want to monitor only specific objects within an app, for example certain chart or table.

Getting the ID of an app object is a bit tricky, but for charts, tables and other UI elements it can be done in a few reasonably easy ways.
The below works for Qlik Sense 2024-May - your mileage may vary for other versions.

Share object from within app
  1. Open the app, then move to the sheet the chart is on.
  2. Right click the chart and select “Share”. Click “Embed”.
  3. The object ID is shown under the preview image of the chart. It is also available in the Iframe URL as the obj parameter.

Get the object ID from the "Share" dialog within the Sense app itself.

Use a Chrome extension

This works if you are using Chrome and have the Add Sense extension installed.

  1. Open the app, then move to the sheet the chart is on.
  2. Click the Add Sense icon in the Chrome toolbar, then “Show” in the menu that appears.
  3. Popup windows will appear with information - including the object ID - about all UI objects on the sheet.

Get the object ID from the Add Sense extension in Chrome.

Use the “Single configurator” in the Qlik Sense Dev Hub

While it is possible to get the object ID from the Dev Hub, it is not recommended as the Dev Hub will be removed in a future version of Qlik Sense.

Still, if you are using a version of Sense that has the Dev Hub, you can get the object ID like this:

  1. Open the Dev Hub, then the “Single configurator” tool (usually found at https://mysense.some.domain/dev-hub/single-configurator).
  2. Select the app and the object you want to get the ID for in the dropdown to the left.
  3. Select the chart or table you are interested in. The object ID is found in the URL.

Get the object ID from the URL in the Single configurator tool in the Dev Hub.

The concepts for filtering by app object are the same as for other filter types.

Filtering by method

The method section can be used to filter events based on the method that generated the event.

The concepts for filtering by method are the same as for other filter types.

Metrics for accepted performance log events

The accepted performance log events stored in InfluxDB are described here.

Metrics for rejected performance log events

The rejected performance log events stored in InfluxDB are described here.

Settings in config file

Tip

The config snippet below comes from the production_template.yaml file.

Being a template, it contains examples on how configuration may be done - not necessarily how it should be done.
For example, the env/DEV and foo/bar tags are optional and can be changed to something else, or removed all together if not used.

Butler-SOS:
  ...
  ...
  # Log events are used to capture Sense warnings, errors and fatals in real time
  logEvents:
    ...
    ...
    enginePerformanceMonitor:           # Detailed app performance data extraction from log events
      enable: false                     # Should app performance data be extracted from log events?
      appNameLookup:                    # Should app names be looked up based on app IDs?
        enable: false
      trackRejectedEvents: 
        enable: false                   # Should events that are rejected by the app performance monitor be tracked?
        tags:                           # Tags are added to the data before it's stored in InfluxDB
          # - name: env
          #   value: DEV
          # - name: foo
          #   value: bar
      monitorFilter:                    # What objects should be monitored? Entire apps or just specific object(s) within some specific app(s)?
                                        # Two kinds of monitoring can be done:
                                        # 1) Monitor all apps, except those listed for exclusion. This is defined in the allApps section.
                                        # 2) Monitor only specific apps. This is defined in the appSpecific section.
                                        # An event will be accepted if it matches any of the rules in the allApps section OR any of the rules in the appSpecific section.
        allApps:
          enable: false                 # Should all apps be monitored?
          appExclude:                   # What apps should be excluded from monitoring?
                                        # If both appId and appName are specified, both must match the event's data for it to be considered a match.
            # - appId: 5b817efe-472d-43ce-8a31-6cce34af7de9
            # - appName: Sales forecast
            # - appId: f42d6b16-8faf-45ca-a783-59f9da47db6e
            #   appName: Inventory analysis
          objectType:
            allObjectTypes: true        # Should all object types be monitored?
            allObjectTypesExclude:      # If allObjectTypes is set to true, the object types in this array are excluded from monitoring. 
                                        # someObjectTypesInclude (below) is ignored in that case.
              # - LoadModelList
              # - <Unknown>
              # - linechart
              # - map
            someObjectTypesInclude:     # What object types should be included in monitoring?
                                        # Only applicable if allObjectTypes is set to false.
              # - LoadModelList
              # - sheet
              # - barchart
          method:
            allMethods: true            # Should all methods be monitored?
            allMethodsExclude:          # If allMethods is set to true, the methods in this array are excluded from monitoring.
                                        # someMethodsInclude (below) is ignored in that case.
              # - Global::OpenApp
              # - Doc::GetAppLayout
              # - Doc::CreateSessionObject
            someMethodsInclude:         # What methods should be included in monitoring?
                                        # Only applicable if allMethods is set to false.
              # - GenericObject::GetLayout
              # - GenericObject::GetHyperCubeContinuousData
        appSpecific:
          enable: false                  # Should app specific monitoring be done?
          app:
            - include:                  # What apps should be monitored?
                                        # If both appId and appName are specified, both must match the event's data for it to be considered a match.
                # - appId: d7cf16f9-6a95-462a-9ff1-a6d413326de4
                # - appName: Budget 2025
                # - appId: 6931136d-c234-4358-a40c-e37153aba7c9
                #   appName: Sales basket analysis
              objectType:
                allObjectTypes: true   # Should all object types be monitored?
                allObjectTypesExclude:  # If allObjectTypes is set to true, the object types in this array are excluded from monitoring. 
                                        # someObjectTypesInclude (below) is ignored in that case.
                  # - table
                  # - map
                someObjectTypesInclude: # What object types should be included in monitoring?
                                        # Only applicable if allObjectTypes is set to false.
                  # - sheet
                  # - barchart
                  # - linechart
                  # - map
              appObject:
                allAppObjects: true     # Should all app objects be monitored?
                allAppObjectsExclude:   # If allAppObjects is set to true, the app objects in this array are excluded from monitoring.
                                        # someAppObjectsInclude (below) is ignored in that case.
                  # - objectId: AaBbCc
                  # - objectId: DdEeFf
                someAppObjectsInclude:  # What app objects should be included in monitoring?
                                        # Only applicable if allAppObjects is set to false.
                  # - objectId: YJEpPT    
              method: 
                allMethods: true        # Should all methods be monitored?
                allMethodsExclude:      # If allMethods is set to true, the methods in this array are excluded from monitoring.
                                        # someMethodsInclude (below) is ignored in that case.
                  # - Global::OpenApp
                  # - Doc::GetAppLayout
                  # - Doc::CreateSessionObject
                someMethodsInclude:     # What methods should be included in monitoring?
                                        # Only applicable if allMethods is set to false.
                  # - GenericObject::GetLayout
                  # - GenericObject::GetHyperCubeContinuousData
  ...
  ...

Log appender XML files

Sample log appender files are available in the ZIP file available from the download page, in subfolders engine/proxy/repository/scheduler of config/log_appender_xml/ folder.

Note that the log appender files contain slightly different information for each Sense service (engine/proxy/repository/scheduler)!
Also keep in mind that the log appender files must be called LocalLogConfig.xml and placed in these directories on the all Sense servers:

  • C:\ProgramData\Qlik\Sense\Engine
  • C:\ProgramData\Qlik\Sense\Proxy
  • C:\ProgramData\Qlik\Sense\Repository
  • C:\ProgramData\Qlik\Sense\Scheduler

Tip

If you have more than one Sense server you strictly speaking don’t have to deploy log appenders to all servers.

If you are only interested in receiving log events from some servers and/or services (engine, proxy, repository, scheduler) - deploy the log appender files there.

10 - Configuring the log database

Warning

Support for Qlik Sense log db was removed in Butler SOS version 11.0.0.

It’s recommended to use log events instead, they provide a more flexible and scalable way to capture log events from Qlik Sense.

What’s this?

Up until mid 2021 Qlik Sense Enterprise on Windows included a logging database to which log events were sent. It was removed from the product due to mainly performance reasons - it could be difficult to scale properly for large Sense clusters.

Butler SOS offers a replacement for log db, in the form of log events.

11 - Connecting to a Qlik Sense server

Details on how to configure the connection from Butler SOS to Qlik Sense Enterprise on Windows.

What’s this?

In order to interact with a Qlik Sense Enterprise on Windows (QSEoW) environment, Butler SOS needs to know a few things about that environment. This is true no matter if the Sense cluster consists of a single Sense server or many.

Settings in main config file

---
Butler-SOS:
  ...
  ...
  # Certificates to use when connecting to Sense. Get these from the Certificate Export in QMC.
  cert:
    clientCert: <path/to/cert/client.pem>
    clientCertKey: <path/to/cert/client_key.pem>
    clientCertCA: <path/to/cert/root.pem>
    clientCertPassphrase: <certificate key password, if one was specified when exporting certificates from Sense QMC >
    # If running Butler in a Docker container, the cert paths MUST be the following
    # clientCert: /nodeapp/config/certificate/client.pem
    # clientCertKey: /nodeapp/config/certificate/client_key.pem
    # clientCertCA: /nodeapp/config/certificate/root.pem
    # clientCertPassphrase: 
  ...
  ...

Qlik Sense certificates

Butler SOS uses certificates to authenticate with Qlik Sense.
These certificates must be exported from the Qlik Management Console (QMC).

Qlik Sense certificate export

To export certificates you need to provide a few pieces of information:

  1. The IP or full host name that Butler SOS’ will use when calling Butler SOS APIs. For example, if Butler SOS get data from server1.my.domain (i.e. the config setting Butler.serverToMonitor.server.host is set to server1.my.domain), the value server1.my.domain should be entered as “machine name” when exporting the certificates from the QMC.
  2. You only need to export certificate from one server in multi-server Sense clusters. The exported certificate can be used to access and get data from any server in the cluster.
  3. Butler SOS can handle certificates with or without password protection. If you choose to use a password, you must enter that password in the Butler SOS config file.
  4. Check the “Include secret key” check box.
  5. Export certificates in PEM format.

Qlik Sense certificate export

Then click the “Export certificates” button. If all goes well the certificates are now exported to a folder on the Sense server to which you are connected (i.e. the server hosting the virtual proxy you are connected to):

Qlik Sense certificate export

The exported certificate files will be used when configuring Butler SOS.

12 - Setting up MQTT messaging

Butler SOS can use MQTT as a channel for pub-sub style M2M (machine to machine) messages. This page describes how to configure MQTT in Butler SOS.

What’s this?

MQTT is a light weight messaging protocol based on a publish-subscribe metaphore. It is widely used in Internet of Things and telecom sectors.

MQTT has features such as guaranteed delivery of messages, which makes it very useful for communicating between Sense and both up- and downstream source/destination systems.

Butler SOS can be configured to forward various metrics and events from Sense as MQTT messages. In order to do so, some shared configuration needs to be in place first. This section covers that configuration.

Specifically, a MQTT broker/gateway has to be configured. All MQTT messages from Butler SOS will be sent to this broker.

Settings in main config file

Butler-SOS:
  ...
  ...
  # MQTT config parameters
  mqttConfig:
    enable: false
    # Items below are mandatory if mqttConfig.enable=true
    brokerHost: <IP of MQTT broker/server>
    brokerPort: 1883
    baseTopic: butler-sos/          # Default topic used if not not oherwise specified elsewhere. Should end with /
  ...
  ...

13 - Setting up the New Relic integration

Butler SOS can send metrics and events to New Relic.
This way it’s possible use their SaaS solution for storing and visualising Butler SOS data.

What’s this?

New Relic offers a suite of online/SaaS products that collectively form a very complete obeservability stack.

From a Butler SOS perspective the interesting parts are metrics, event and log handling.
By forwarding such data to New Relic it’s not necessary to run local InfluxDB and Grafana instances.

That said, New Relic is a commercial service and while their free tier is very generous, there will be a trade-off between a local/lower cost InfluxDB/Grafana setup and using New Relic.
With InfluxDB/Grafana you also get more fine grained control over both data storage and visualisations, while New Relic offer ease of setup and no need to host InfluxDB/Grafana yourself.

Below the settings for sending Qlik Sense health metrics to New Relic are described.

Tagging of data

The following attributes (which is New Relic lingo for tags) are added.

Note: Attributes defined further down in the list will overwrite already defined attributes if their names match.
To avoid problems you should make sure not to use already defined attributes.

Tags for Qlik Sense health metrics

  1. Static attributes defined in the config file’s Butler-SOS.newRelic.metric.attribute.static section.
  2. Dynamic attributes
    1. butlerSosVersion: Butler SOS version
  3. Dynamic attributes based on whether each item in Butler-SOS.newRelic.metric.dynamic section is enabled/disabled.
    1. If Butler-SOS.newRelic.metric.dynamic.engine.memory section is enabled
      1. qs_memCommited
      2. qs_memAllocated
      3. qs_memFree
    2. If Butler-SOS.newRelic.metric.dynamic.engine.cpu section is enabled
      1. qs_cpuTotal
    3. If Butler-SOS.newRelic.metric.dynamic.engine.calls section is enabled
      1. qs_engineCalls
    4. If Butler-SOS.newRelic.metric.dynamic.engine.selections section is enabled
      1. qs_engineSelections
    5. If Butler-SOS.newRelic.metric.dynamic.engine.sessions section is enabled
      1. qs_engineSessionsActive
      2. qs_engineSessionsTotal
    6. If Butler-SOS.newRelic.metric.dynamic.engine.users section is enabled
      1. qs_engineUsersActive
      2. qs_engineUsersTotal
    7. If Butler-SOS.newRelic.metric.dynamic.engine.saturated section is enabled
      1. qs_engineSaturated
    8. If Butler-SOS.newRelic.metric.dynamic.apps.docCount section is enabled
      1. qs_docsActiveCount
      2. qs_docsLoadedCount
      3. qs_docsInMemoryCount
    9. If Butler-SOS.newRelic.metric.dynamic.cache.cache section is enabled
      1. qs_cacheHits
      2. qs_cacheLookups
      3. qs_cacheaAdded
      4. qs_cacheReplaced
      5. qs_cacheBytesAdded

Tags for Qlik Sense proxy session metrics

If and how Butler SOS should extract proxy session metrics is controlled in the Butler-SOS.userSessions section of the config file.

  1. Static attributes defined in the config file’s Butler-SOS.newRelic.metric.attribute.static section.
  2. Dynamic attributes
    1. butlerSosVersion: Butler SOS version

Settings in main config file

Butler-SOS:
  ...
  ...
  # New Relic config
  # If enabled, select Butler SOS metrics will be sent to New Relic.
  newRelic:
    enable: false
    event:
      # There are different URLs depending on whther you have an EU or US region New Relic account.
      # The available URLs are listed here: https://docs.newrelic.com/docs/accounts/accounts-billing/account-setup/choose-your-data-center/
      #
      # Note that the URL path should *not* be included in the url setting below!
      # As of this writing the valid options are
      # https://insights-collector.eu01.nr-data.net
      # https://insights-collector.newrelic.com 
      url: https://insights-collector.eu01.nr-data.net
      header:                   # Custom http headers
        - name: X-My-Header
          value: Header value
      attribute: 
        static:                 # Static attributes/dimensions to attach to the events sent to New Relic.
          - name: service
            value: butler-sos
          - name: environment
            value: prod
        dynamic:
          butlerSosVersion: 
            enable: true       # Should the Butler SOS version be included in the events sent to New Relic?
    metric:
      destinationAccount:
        - First NR account
        - Second NR account
      # There are different URLs depending on whther you have an EU or US region New Relic account.
      # The available URLs are listed here: https://docs.newrelic.com/docs/accounts/accounts-billing/account-setup/choose-your-data-center/
      # As of this writing the options for the New Relic metrics API are
      # https://insights-collector.eu01.nr-data.net/metric/v1
      # https://metric-api.newrelic.com/metric/v1 
      url: https://insights-collector.eu01.nr-data.net/metric/v1   # Where should uptime data be sent?
      header:                   # Custom http headers
        - name: X-My-Header
          value: Header value
      dynamic:
        engine:
          memory:               # Engine RAM (free/committed/allocated).
            enable: true
          cpu:                  # Engine CPU.
            enable: true        
          calls:                # Total number of requests made to the engine.
            enable: true        
          selections:           # Total number of selections made to the engine.
            enable: true        
          sessions:             # Engine session metrics (active and total number of engine sessions).
            enable: true        
          users:                # Engine user metrics (active and total number of users in engine.
            enable: true        
          saturated:            # Engine saturation status (tracks whether engine has high or low load).
            enable: true
        apps:
          docCount:
            enable: true
        cache:
          cache:                # Cache metrics.
            enable: true
        proxy:
          sessions:             # Session metrics as reported by the Sense proxy service
            enable: true
      attribute: 
        static:                 # Static attributes/dimensions to attach to the data sent to New Relic.
          - name: service
            value: butler-sos
          - name: environment
            value: prod
        dynamic:
          butlerSosVersion: 
            enable: true       # Should the Butler SOS version be included in the data sent to New Relic?
  ...
  ...

14 - Setting up Prometheus

Butler SOS can store metrics in Prometheus.

What’s this?

Prometheus is the de-facto standard, open source tool for achieving observability of both small, large and huge IT systems.

At its heart Prometheus contains a time-series databas optimized for storing various kinds of measurements. It has strong support for doing dimensional queries, great integrations with incident managament tools and more.

Looking at the visualisation side of things, Prometheus is Grafana’s preferred source for time-series data. Put differently, Prometheus has some query features that InfluxDB lack, thus making some Grafana diagrams easier to create using Prometheus vs InfluxDB. The difference is minor though.

Settings in main config file

Butler-SOS:
  ...
  ...
  # Prometheus config
  # If enabled, select Butler SOS metrics will be exposed on a Prometheus compatible URL from where they can be scraped.
  prometheus:
    enable: false                                    # Default false
    host: <IP or FQDN where Butler SOS is running>  # On what IP/FQDN should the Prometheus metrics be exposed? Default 0.0.0.0, i.e. all available IPs
    port: 9842      
  ...
  ...

15 - Setting up InfluxDB time series database

Butler SOS can store metrics in InfluxDB.

Warning

Butler SOS supports InflixDB 1.x and 2.x.
There are reports that InfluxDB’s cloud product also works with Butler SOS, that has however not been tested by the Butler SOS team.

Version 3 (in beta at the time of this writing) is not supported.

What’s this?

InfluxDB is a time series database. This means it is optimised for storing data that’s somehow linked to a timestamp.
Measurements and metrics are some of the most obvious kinds of data for which InfluxDB was created.

Butler SOS stores data in InfluxDB in full detail, i.e. Butler SOS doesn’t do any aggregation of older data points.
This has a few consequences:

  • If you are monitoring many Sense servers and/or query Sense for health metrics very frequently and/or have a long InfluxDB retention policy (many months or even years) you will eventually end up with lots of data.
  • You should ask yourself how far back you need to look at operational data such as the one collected by Butler SOS. In most cases 30 or 45 days history will be more than enough. 10-14 days are usually a good starting point. Use the Butler-SOS.influxdbConfig.retentionPolicy section of the config file to create a retention policy for Butler SOS.
  • If you need longer history you should consider using InfluxDB’s excellent aggregation features. These can assist in aggregating older data points, with the effect that you can then keep virtually unlimited history. The older data will not be as detailed (fewer samples per time period) - but you will still have an averaged view of what the history looked like.

Note 1: Instructions for how to aggregate old data is beyond the scope of this documentation.

Note 2: The retention policy specified in the config file will only be created if the InfluxDB database specified in the config file does NOT exist when Butler SOS is started.
I.e. if you store data to an existing InfluxDB database, the retention policy will not be created.

Settings in main config file

Butler-SOS:
  ...
  ...
  # Influx db config parameters
  influxdbConfig:
    enable: true
    # Items below are mandatory if influxdbConfig.enable=true
    host: influxdb.mycompany.com    # InfluxDB host, hostname, FQDN or IP address
    port: 8086                      # Port where InfluxDBdb is listening, usually 8086
    version: 1                      # Is the InfluxDB instance version 1.x or 2.x? Valid values are 1 or 2
    v2Config:                       # Settings for InfluxDB v2.x only, i.e. Butler-SOS.influxdbConfig.version=2
      org: myorg
      bucket: mybucket
      description: Butler SOS metrics
      token: mytoken
      retentionDuration: 10d
    v1Config:                       # Settings below are for InfluxDB v1.x only, i.e. Butler-SOS.influxdbConfig.version=1
      auth:
        enable: false               # Does influxdb instance require authentication (true/false)?
        username: <username>        # Username for Influxdb authentication. Mandatory if auth.enable=true
        password: <password>        # Password for Influxdb authentication. Mandatory if auth.enable=true
      dbName: SenseOps
      # Default retention policy that should be created in InfluxDB when Butler SOS creates a new database there. 
      # Any data older than retention policy threshold will be purged from InfluxDB.
      retentionPolicy:
        name: 10d
        duration: 10d                 # Possible duration units here: https://docs.influxdata.com/influxdb/v1.8/query_language/spec/#durations
    # Control whether certain metrics are stored in InfluxDB or not
    # Use with caution! Enabling activeDocs, loadedDocs or inMemoryDocs may result in lots of data sent to InfluxDB.
    includeFields:
      activeDocs: false              # Should data on what docs are active be stored in Influxdb (true/false)? 
      loadedDocs: false              # Should data on what docs are loaded be stored in Influxdb (true/false)?
      inMemoryDocs: false            # Should data on what docs are in memory be stored in Influxdb (true/false)?

  ...
  ...

16 - Configuring extraction of app names from Qlik Sense

What’s this?

Qlik Sense’s APIs return all its app metrics relative to an app ID, not the app name.
This is fine as the ID is guaranteed to be unique, but the downside is that the ID doesn’t tell us humans much.

Butler SOS therefore provides an app ID-to-app name mapping with a configurable update interval.
The Butler-SOS.appNames section of the config file controls if this mapping should be done at all, how often and which Sense server should be used to get it.

If an app ID for some reason can’t be mapped to an app name, Butler SOS will use the app ID as the app name.

A more comprehensive description of Butler SOS’ strategy for getting correct names of Sense apps is available in the Concepts section.

Settings in main config file

Butler-SOS:
  ...
  ...
  # Extract app names
  appNames: 
    enableAppNameExtract: true    # Extract app names in addition to app IDs (tue/false)?
    extractInterval: 60000        # How often (milliseconds) should app names be extracted?
    hostIP: <IP or FQDN>          # What Sense server should be queried for app names?
  ...
  ...

Setting anonTelemetry to true enables telemetry, setting it to false disables telemetry.

17 - Configuring user sessions

What’s this?

A description of what user sessions are is available in the Concepts section.

Detailed user session metrics are retrieved for all virtual proxies specified in the Butler-SOS.serversToMonitor.servers[].userSessions.VirtualProxies[] array.

In other words: For each monitored Sense server it is possible to specify which of the server’s proxy service’s virtual proxies should be monitored with respect to per-user session metrics.
Right, that’s a long sentence…

Let’s try again: For each monitored Sense server, decide which virtual proxies should be monitored.
Enter those virtual proxies in the Butler-SOS.serversToMonitor.servers[].userSessions.VirtualProxies[] array for the server in question.

Note

In order to get detailed, per-user and virtual proxy session info you need to

  1. Configure the Butler-SOS.userSessions section of the config file with general parameters about how often sessions should be polled, user blacklist etc.
    Don’t forget to set Butler-SOS.userSessions.enableSessionExtract to true.
  2. For each server then set Butler-SOS.serversToMonitor.servers[].userSessions.enable to true and specify which virtual proxies should be monitored.

You will only get user session info if you configure both the points above.

Settings in main config file

Tip

The config snippet below comes from the production_template.yaml file.

Being a template, it contains examples on how configuration may be done - not necessarily how it should be done.
For example, the LAB/testuser1 and LAB/testuser2 user are optional and can be changed to something else, or removed all together if not used.

Butler-SOS:
  ...
  ...
  # Sessions per virtual proxy
  userSessions:
    enableSessionExtract: true      # Query unique user IDs of what users have sessions open (true/false)?
    # Items below are mandatory if enableSessionExtract=true    
    pollingInterval: 30000        # How often (milliseconds) should session data be polled?
    excludeUser:                  # Optional blacklist of users that should be disregarded when it comes to session monitoring.
                                  # Blacklist is only applied to data in InfluxDB. All session data will be sent to MQTT.
      - directory: LAB
        userId: testuser1
      - directory: LAB
        userId: testuser2
  ...
  ...

18 - Configure which Sense servers to monitor

What’s this?

This part of the config file contains information on what Sense servers should be monitored and details about those servers.

Note that there is a dependency to the Butler-SOS.userSessions section. Please see the Configuring user sessions page for more info.

Tags

It’s possible to define server specific tags that will be stored together with the Sense metrics in InfluxDB.
The tags can then be used when creating Grafana dashboards, for example to distinguish between DEV, TEST and PROD servers, where servers are located physically etc.

You can define zero or more tags in the Butler-SOS.serversToMonitor.serverTagsDefinition section.
Those tags are then given values for each server.

If a tag is defined it must in fact be given a value for each server!
If nothing else just set it to an empty string or a hyphen, ‘-’.

Settings in main config file

Tip

The config snippet below comes from the production_template.yaml file.

Being a template, it contains examples on how configuration may be done - not necessarily how it should be done.
For example, the list serverTagsDefinition is optional and can be changed to something else, or removed all together if not used.

Same thing for the list of servers and virtual proxies. Update them so they match your own Sense environment.

Butler-SOS:
  ...
  ...
  serversToMonitor:
    pollingInterval: 30000          # How often (milliseconds) should the healthcheck API be polled?

    # If false, Butler SOS will accept TLS certificates on the server without verifying them with the CA. 
    # If true, data will only be retrieved from the Sense server if that server's TLS cert verifies 
    # successfuully against the list of CAs available on the computer where Butler SOS is running.
    rejectUnauthorized: true 

    # List of extra tags for each server. Useful for creating more advanced Grafana dashboards.
    # Each server below MUST include these tags in its serverTags property.
    # The tags below are just examples - define your own as needed
    # These tags will also be used to label data exposed on the Prometheus endpoint (if it is enabled)
    # NOTE: Prometheus only allows label names consisting of ASCII letters, numbers, as well as underscores. They must match the regex [a-zA-Z_][a-zA-Z0-9_]*. 
    # I.e. if the Prometheus endpoint is enabled, the tag names below must follow the label naming standard of Prometheus. 
    serverTagsDefinition: 
      - server_group
      - serverLocation
      - server_type
      - serverBrand

    # Sense Servers that should be queried for healthcheck data 
    servers:
      - host: <server1.my.domain>:4747        # Example: 10.34.3.45:4747
        serverName: <server1>
        serverDescription: <description>
        logDbHost: <host name as used in QLogs db>
        userSessions:
          enable: true
          # Items below are mandatory if userSessions.enable=true
          host: <server1.my.domain>:4243      # Example: 10.34.3.45:4243
          virtualProxies:
            - virtualProxy: /                 # Default virtual proxy
            - virtualProxy: /hdr              # "hdr" virtual proxy
            - virtualProxy: /sales            # "sales" virtual proxy
        serverTags:
          server_group: DEV
          serverLocation: Asia
          server_type: virtual
          serverBrand: Dell
        headers: 
          X-My-Header-1: Header value 1
          X-My-Header-2: Header value 2
      - host: <server2.my.domain>:4747        # Example: 10.34.3.46:4747
        serverName: <server2>
        serverDescription: <description>
        logDbHost: <host name as used in QLogs db>
        userSessions:
          enable: true
          # Items below are mandatory if userSessions.enable=true
          host: <server2.my.domain>:4243      # Example: 10.34.3.46:4243
          virtualProxies:
            - virtualProxy: /finance          # "finance" virtual proxy
        serverTags:
          server_group: PROD
          serverLocation: Europe
          server_type: physical
          serverBrand: HP
        headers: 
          X-My-Header-1: Header value 3
          X-My-Header-2: Header value 4
  ...
  ...

19 - Configuring telemetry

What’s this?

A description of Butler’s telemetry feature is available here.

Settings in main config file

---
Butler:
  # Logging configuration
  ...
  ...
  anonTelemetry: true     # Can Butler SOS send anonymous data about what computer it is running on? 
                          # More info on whata data is collected: https://butler-sos.ptarmiganlabs.com/docs/about/telemetry/
                          # Please consider leaving this at true - it really helps future development of Butler SOS!
  ...
  ...

Setting anonTelemetry to true enables telemetry, setting it to false disables telemetry.