About

• 1: Butler SOS
• 2: The Butler family
• 3: Use cases
• 4: Versions
• 5: Contribution guidelines
• 6: Telemetry

1 - Butler SOS

The Butler SOS project is about adding best-in-class monitoring to the client-managed Windows version of Qlik Sense Enterprise, also known as QSEoW (Qlik Sense Enterprise on Windows).

The goal is to provide a close to real-time view into what’s happening in a Qlik Sense environment.

At different times different metrics will be of interest.
For that reason Butler SOS stores all metrics from Sense in a time-series databse (InfluxDB and Prometheus both supported), from which dashboards, reports etc can be created using tools such as Grafana.
Grafana is an open source, world-class visualisation tool for time series data. It also has great alerting features and integrate with all kinds of alerting solutions, email, Teams, Slack and more.

If you don’t fancy InfluxDB or Prometheus, Qlik Sense metrics and events can also be sent to New Relic for storage and visualisation.
They offer a free tier that will go a long way towards testing out a cloud-based visualisation solution for Butler SOS.

Metrics are a major component of operational monitoring, but it’s also important to keep on top of what errors and warning occur in the system.
As of late 2021 the log database is no longer part of QSEoW, with the log files the only place where you can find those errors and warnings.

Butler SOS address this by sending log events in real-time from QSEoW to Butler SOS, which then stores them in InfluxDB, and/or send them to New Relic, and/or re-publish them as MQTT messages.
This basically means you will very rarely have to plow through endless log files to find information about warnings and errors that have occured.
Warnings and errors can alse be categorised by Butler SOS, making it much easier to understand what is happening in your Sense environment.

By listening in on the Qlik Sense log events from the associative engine, Butler SOS gets very detailed information about what is happening in the engine (which is where the magic happens).
This means you get very detailed information about what is happening in your Sense apps, sheets and charts - as it happens.
A chart that takes too long to render, an app that opens slowly, a sheet that is slow to load - all these things can be monitored in real-time.

Butler SOS also tracks user agents of all users accessing your Sense environment. This means you get real-time insights into what operating systems and browsers are used to access your Sense environment.

There is a clear goal that Butler SOS should be very configurable.
In practice this means that features can be turned on/off as needed, improving security and lowering memory usage.

Butler SOS is written in Node.js and runs on most modern operating systems.
Stand-alone binary files are created for Windows, Linux and macOS.
Windows and macOS binaries are signed with a code-signing certificates, making it easier to install and run on those platforms.

You can run Butler SOS on the same server as Qlik Sense, in a Docker container on a Linux server, in Kubernetes, on Mac OS, on Raspberry Pi (not a good idea.. but possible and proven to work).

Butler SOS is a member of a group of tools collectively referred to as the “Butler family”, more info is available here.

2 - The Butler family

Butler started out with a very specific need to start Sense reloads from outside systems.
Over the years a few projects (for example Butler SOS, which simplifies day 2 operations ([1], [2]) have spun off from the original Butler project, and still other projects have been created from scratch to solve specific challenges around developing Sense apps and running Qlik Sense server environments.

All members of the Butler family are available on Ptarmigan Labs’ GitHub page.

Projects with production grade release status are (as of this writing):

Butler

The original Butler. Offers various utilities that make it easier to develop Sense apps, as well as simplifying day 2 operations.

butler.ptarmiganlabs.com.

Butler SOS

Real-time operational metrics for Qlik Sense.
A must-have if you are responsible for a Sense environment with more than a dozen or so users.

Butler SOS makes it possible to detect and alert on issues as they happen, rather than in retrospect much later.

butler-sos.ptarmiganlabs.com. (This site!)

Butler Sheet Icons

Automates the creation of sheet icons for Qlik Sense Enterprise on Windows (QSEoW) applications.

It’s a cross platform command line tool which given the correct Sense credentials will take screen shots of all sheets in a Sense app (or all apps on a Sense server!), then create thumbnail versions of those screenshots.
Finally those thumbnails will be set as sheet icons.

No more manual screenshot taking, resizing images, navigating hundreds of sheets in dozens of apps.
Start Butler Sheet Icons instead and go get a nice fika.

The tool can be used stand-along or as part of an automated release process.

https://github.com/ptarmiganlabs/butler-sheet-icons

Ctrl-Q

Given the name of this tool it doesn’t sound like a member of the Butler family.
Let’s say Ctrl-Q is a sibling of the Butler bunch.

While the Butler tools are (usually) intended to solve and simplify rather specific use cases, Ctrl-Q is aimed at being the lazy Qlik developer’s best friend.

Let’s say there is some manual, tedious, time consuming and error prone activity that a Qlik Sense developer is faced with.
For example importing dozens of apps from QVF files and creating a hundred associated reload tasks.
Ctrl-Q lets you do this with a single command, using definitions in an Excel file. Instead of spending a day on this the actual execution takes a minute or so.

In other words: Ctrl-Q focus on high-value use cases that are difficult or impossible to solve using other tools.

github.com/ptarmiganlabs/ctrl-q

Butler CW

Butler Cache Warmer. Cache warming is the process of proactively forcing Sense apps to be loaded into RAM, so they are readily available when users open them.
Using Butler CW is an easy way to make your end users’ experience of Sense a little better.

github.com/ptarmiganlabs/butler-cw

Butler App Duplicator

No matter if you are a single developer creating Sense apps, or have lots of developers doing this, having app templates is a good idea:

• Lowered barrier of entry for new Sense developers.
• Productivity boost when developing Sense apps.
• Encouraging a common coding standard across all apps.

github.com/ptarmiganlabs/butler-app-duplicator

Butler Spyglass

This tool is mainly of interest if you have lots of QVDs and apps, but when that’s the case it’s of paramount importance to understand what apps use which QVDs. In other words what data lineage looks like.

Butler Spyglass also extracts full load scripts for all Sense apps, creating a historical record of all load scripts for all Sense apps.

github.com/ptarmiganlabs/butler-spyglass

Butler Notifier

This tool makes it easy to tap into the Qlik Sense notification API. From there you can get all kinds of notifications, including task reload failures and changes in session state (user login/logout etc).

github.com/ptarmiganlabs/butler-notifier

Butler Icon Uploader

Visual looks is important when it comes to analytics, and this holds true also for Sense apps.

The Butler Icon Uploader makes it easy to upload icon libraries (for example Font Awesome) to Qlik Sense Enterprise. With such icons available it is then easy for app developers to use professional quality sheet and app icons in their Sense apps.

github.com/ptarmiganlabs/butler-icon-upload

3 - Use cases

A complete list of metrics is available in the reference section.

Monitor Qlik Sense metrics

This is the main use case for Butler SOS, with a large number of monitored metrics.
Butler SOS can be configured to store these metrics in InfluxDB/Prometheus and/or send them as MQTT messages.

Session count

A session (or more specifically, a “proxy session”) is created when a user logs into Sense.
The session is typically reused when a user opens additional Sense apps from the same browser.
Knowing how many users are logged at any given time gives a Sense admin an understanding of when peak hours are, when service windows should be planed, whether the server(s) is too small or too big etc.

The session metrics are arguably among the most important ones provided by Butler SOS

Applications

Butler SOS tracks how many and what applications (IDs and app names) are in loaded into the monitored Qlik engine(s). In-memory and active applications are also tracked in the same way.

These metrics provide useful insights into the degree to which loaded apps are actually used and to what degree caching is in effect.

A couple of bonus metrics are also included: Number of calls made to the Qlik associative engine and number of selections done by users in the engine. Not really useful as such, but they do serve as a good relative measurement of how active users are between days/weeks/months.

Cache status

The caching of applications in RAM is one of Qlik’s classic selling points. But is it really working?

Butler SOS provides a set of metrics (bytes added to cache, lookups, cache hits/misses etc) that give hard numbers to the question of whether the cache is working or not.

While this is probably not as interesting from an operational perspective as user session counts, RAM usage and errors/warnings from the Sense logs, the cache metrics should definitely be monitored over medium timespans.

For example, if the cache hit ration goes down over weeks or months, that could mean a poorer user experience over that same time period.

Monitor server metrics

Some basic server metrics (free RAM and CPU load) are monitored by Butler SOS. You may have other, dedicated server monitoring tools too - Butler SOS does not replace these.
It’s however often convenient to have both server and Sense metrics side by side, thus Butler SOS includes some of the more important server metrics in addition to the Sense ones.

These metrics are only stored in InfluxDB/Prometheus, i.e. not sent as MQTT messages.

Available memory/RAM

As Qlik Sense is an in-memory analytics tool you really want to ensure that there is always available memory for users’ apps.
If your Sense server runs out of memory it’s basically game over. Now, Sense usually does a very good job reclaiming unused memory, but it’s still critically important to monitor memory usage.

One error scenario that’s hard or impossible to catch without Butler SOS style monitoring is that of apps with cartesian products in them.
They can easily consume tens of hundreds GByte of RAM within seconds, bringing a Sense server to a halt.
Butler SOS has more than once proven its value when debugging this specific issue.

CPU load

If a server is heavily loaded it will eventually be seen as slow(er) by end users, with associated badwill accumulating.

Log events: Qlik Sense errors & warning

The Sense logs are always available on the Sense servers, the problem is that they are hard to reach there - at least in real time.
Retrospective analysis is also cumbersome, you basically have to manually dig up the specific log files of interest and then search them for the needed information.
Qlik does provide good analysis apps for the logs, but they are not real-time and they must be reloaded (which tends to be slow and resource intensive) to show new data.

Butler SOS simplifies this greatly by having select log events (warnings, errors and fatals by default) sent from the Sense servers to Butler SOS.
Once such a log event message arrives, Butler SOS will store it in its database (for example InfluxDB or New Relic), from where the log event can be visualised using Grafana or within New Relic.

Log events from several Qlik Sense servics can selectively be forwarded to Butler SOS:

• Engine
• Proxy
• Repository
• Scheduler

A sample use case of log events:

1. Create a Grafana or New Relic real-time chart showing number of warnings/errors per 5-minute window.
2. Set up an alert in those tools to notify you when number of events during past 5 minutes go above some limit.

This is trivial to set up, but gives you a very capable, close to real-time error/warning monitoring solution.

Log events can also be re-published as MQTT messages. This makes it possible for 3rd party systems to trigger actions when certain log events occur in Qlik Sense.

Categorisation of log events

By categorising log events, Butler SOS makes it much easier to understand what is happening in your Sense environment.
For example, if you categorise log events as follows:

• reload failures as “reload_failures”
• engine related errors/warnings as"engine"
• Active Directory related errors as “active_directory”
• permission denied related messages as “permission”

…you can then easily create Grafana charts that show number of warnings/errors per category, and also clearly show how many unknown log events there are.
It’s also easy to track volume of warnings/errors over time, and to set up alerts when number of events go above some threshold.

Engine performance log events

Butler SOS can also process performance log events from the Qlik associative engine (the “Qix” engine as it was called in older versions of Sense).

These events provide very detailed insights into how the engine is performing, including how long it takes to calculate expressions, how long it takes to open apps, how long it takes to calculate charts etc.
And how much memory each of those operations consume.

This is a very powerful feature, but when enabled in Qlik Sense (from the QMC) it will generate a lot of data that will all be store in the Sense log files.

By configuring Qlik Sense to send these events to Butler SOS instead of storing it in ever growing log files, you gain several advantages:

• The data is stored in a time series database, making it easy to visualise, analyse and alert on in real time.
• By applying retention policies in the database, you keep only the most recent data, thus saving (lots of!) disk space compared to storing all data in log files on the Sense servers.

Let’s look at a few examples of what data volumes are generated by these events:

• A user opening a Sense app will generate 15-20 log events.
• Making a selection in a chart will generate 10-15 log events.
• A scheduled reload will generate 5-10 initial log events, then one progress event every 2 seconds, and finally 5-10 log events when the reload is done. For long running reloads this can easily generate 1000+ log events.

When enabled via the QMC, the associated Sense log files can easily reach millions of lines per day week even on a relatively small Sense environment.
Butler SOS lets you handle this data in a more efficient way.

The log database

The log database in Qlik Sense Enterprise on Windows is deprecated as of late 2021.
Butler SOS still maintains support for older QSEoW versions. At some point in the future this support is however likely to be removed.

This feature relies on Butler SOS querying log db with certain intervals (typically every few minutes).
A list of recent log events are returned to Butler SOS. The events are de-duplicated before stored in InfluxDB.

This essentially achieve the same thing as the more modern log event feature of Butler SOS - but with longer delays. The log event model is almost instantaneous whereas the log db polling will be its very nature result in non real-time data.

User activity events

Detailed events are avilable for all users:

• Session start
• Session stop
• Connection open
• Connection close

These events are both stored in InfluxDB and re-published as MQTT messages.

Usually it’s enough to track how many users are currently using the Qlik Sense system.
Exactly what users are usually of less interest.

At times you may want more detailed insights though. Then these events are increadibly useful.

For example:

• Sometimes network issues cause some users’ browsers to start many new sessions instead of re-using existing sessions.
  This can result in the proxy service overloading and making access to Sense slow for all users.
  Butler SOS makes it easy to detect this. Just create a Grafana chart that shows number of session start events over time, attach an alert that goes off if number of new sessions per minute is too high.
• Let’s say a specific user has troubles using Sense apps as intended, or a user is suspected of causing excessive RAM/CPU usage.
  Subscribe to the MQTT user activity messages coming from Butler SOS, filtering out just the user(s) of interest.
  Get a notification the very moment the user connects to Sense after which you can follow in real time what happens with the system. Does CPU go up? Free RAM goes down? Which apps are loaded?

You can also use the MQTT message to create your personal disco light, controlled by your Sense users connecting and dropping off the Sense server…

Green = User opening a connection to Sense
Red = User closing connection to Sense

From YouTube.

4 - Versions

5 - Contribution guidelines

Butler SOS is developed in Node.js, with support from various NPM modules.

We use Hugo to format and generate this documentation site, the Docsy theme for styling and site structure.
Hugo is an open-source static site generator that provides us with templates, content organisation in a standard directory structure, and a website generation engine. You write the pages in Markdown (or HTML if you want), and Hugo wraps them up into a website.

All submissions, including submissions by project members, require review. We use GitHub pull requests for this purpose. Consult GitHub Help for more information on using pull requests.

Creating an issue

If you’ve found a problem - or have a feature suggestion - with Butler SOS itself or the documentation, but you’re not sure how to fix it yourself, please create an issue in the Butler SOS repo. You can also create an issue about a specific doc page by clicking the Create Issue button in the top right hand corner of the page.

Development concepts

Some of the main tools/processes used during development of Butler SOS are:

Visual Studio Code (=VSC)

Any IDE supporting Node.js can be used, but VSC works really well. Open Source and a huge ecosystem of extensions.

GitHub

Used to store source code, track issues, change requests etc.

GitHub Actions used to build Docker images.

Release Please

Release Please is used to create release notes.
It also enforces consistent versioning when new (sometimes breaking) features are added, bugs fixed etc.

This menas thatas of Butler SOS 6.0 you can have more trust in the semantic versioning of Butler SOS releases.

ESLint + Prettier

Used to enforce a uniform source code format that also follow best practices defined in ESLint.

ESLint shows code issues within Visual Studio Code, but standalone reports can also be created:

➜ npx eslint . --format table

/Users/goran/code/butler-sos/src/butler-sos.js

║ Line     │ Column   │ Type     │ Message                                                │ Rule ID              ║
╟──────────┼──────────┼──────────┼────────────────────────────────────────────────────────┼──────────────────────╢
║ 31       │ 18       │ error    │ Unexpected require().                                  │ global-require       ║

/Users/goran/code/butler-sos/src/lib/appnamesextract.js

║ Line     │ Column   │ Type     │ Message                                                │ Rule ID              ║
╟──────────┼──────────┼──────────┼────────────────────────────────────────────────────────┼──────────────────────╢
║ 29       │ 37       │ error    │ A constructor name should not start with a             │ new-cap              ║
║          │          │          │ lowercase letter.                                      │                      ║

/Users/goran/code/butler-sos/src/lib/heartbeat.js

║ Line     │ Column   │ Type     │ Message                                                │ Rule ID              ║
╟──────────┼──────────┼──────────┼────────────────────────────────────────────────────────┼──────────────────────╢
║ 3        │ 1        │ error    │ Unexpected var, use let or const instead.              │ no-var               ║
║ 6        │ 1        │ error    │ Unexpected var, use let or const instead.              │ no-var               ║
║ 6        │ 21       │ warning  │ Unexpected unnamed function.                           │ func-names           ║
║ 9        │ 15       │ error    │ Unexpected function expression.                        │ prefer-arrow-        ║
║          │          │          │                                                        │ callback             ║
║ 9        │ 15       │ warning  │ Unexpected unnamed function.                           │ func-names           ║
║ 9        │ 25       │ error    │ 'response' is defined but never used.                  │ no-unused-vars       ║
║ 13       │ 16       │ error    │ Unexpected function expression.                        │ prefer-arrow-        ║
║          │          │          │                                                        │ callback             ║
║ 13       │ 16       │ warning  │ Unexpected unnamed function.                           │ func-names           ║
║ 27       │ 9        │ error    │ All 'var' declarations must be at the top of the       │ vars-on-top          ║
║          │          │          │ function scope.                                        │                      ║
║ 27       │ 9        │ error    │ Unexpected var, use let or const instead.              │ no-var               ║
║ 28       │ 9        │ error    │ All 'var' declarations must be at the top of the       │ vars-on-top          ║
║          │          │          │ function scope.                                        │                      ║
║ 28       │ 9        │ error    │ Unexpected var, use let or const instead.              │ no-var               ║
║ 28       │ 13       │ error    │ 't' is assigned a value but never used.                │ no-unused-vars       ║
║ 28       │ 35       │ error    │ Unexpected function expression.                        │ prefer-arrow-        ║
║          │          │          │                                                        │ callback             ║
║ 28       │ 35       │ warning  │ Unexpected unnamed function.                           │ func-names           ║

╔════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗
║ 13 Errors                                                                                                      ║
╟────────────────────────────────────────────────────────────────────────────────────────────────────────────────╢
║ 4 Warnings                                                                                                     ║
╚════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝

➜ 

To format all source code files:

➜  npm run format:prettier

> butler-sos@5.6.0 format:prettier
> npx prettier --config .prettierrc.yaml "./**/*.{ts,css,less,js}" --write

butler-sos.js 97ms
docker-healthcheck.js 13ms
globals.js 90ms
lib/appnamesextract.js 35ms
lib/healthmetrics.js 27ms
lib/heartbeat.js 8ms
lib/logdb.js 34ms
lib/post-to-influxdb.js 85ms
lib/post-to-mqtt.js 27ms
lib/prom-client.js 45ms
lib/servertags.js 5ms
lib/service_uptime.js 14ms
lib/sessionmetrics.js 29ms
lib/telemetry.js 18ms
lib/udp_handlers.js 18ms
➜ 

6 - Telemetry

What’s telemetry

From Wikipedia:

Telemetry is the in situ collection of measurements or other data at remote points and their automatic transmission to receiving equipment (telecommunication) for monitoring.

In the context of software tools (including Butler) telemetry is often used to describe the process of sending information about the tool itself to some monitoring system.

Why telemetry in Butler SOS?

This is a good question.

For several years there was no telemetry at all in Butler SOS.

Development of new features were driven mainly by what features were needed at the time.
Or the fact that Qlik released some new feature in Sense and Butler SOS was a way to test that new feature from the perspective of the Sense APIs.

That’s all good but today Butler SOS is a rather significant tool with features spanning monitoring, alerting and more.

This multitude of features is also one of the core reasons for adding telemetry to Butler SOS:

Without this information the Butler SOS developers will keep working in the dark, not really knowing where to focus their efforts.

On the other hand - with access to telemetry data a lot of possibilities open up for the Butler SOS developers:

• If telemetry shows that no one uses a particular feature, maybe that feature should be scheduled for deprecation?
• The opposite of the previous: If lots of users use a specific Butler SOS feature, then that feature is a candidate for future focus and development.
• Telemetry will show if lots of users run Butler SOS on old Node.js versions. Knowing this its possible to set a migration schedule for what Node.js versions are supported - avoiding hard errors when some old Node.js version is no longer supported by Butler SOS.
• Same thing for understanding what operating systems Butler SOS runs (and should be supported) on.

Configuring Butler SOS’ telemetry

Instructions here.

The details

Where is telemetry data sent

Butler SOS uses PostHog to collect telemetry data.

PostHog is an open source telemetry platform that is used by many open source projects. The data is stored in the EU.

Deleting telemetry data

Even though no-one (not even Ptarmigan Labs who runs the telemetry database!) has any way of ever connecting the data sent by your Butler SOS instance to you (it’s all anonymized, remember?), there can be cases where telemetry data must be deleted.

The legal page has more information about this.

Field level description of telemetry data

A telemetry message from Butler SOS contains the information below.

{
  "system": {
    "id": "62f221c94dc72823ebba9488a74006ccf69da8f8c6cfaa896a60d9a02186cc2e",
    "arch": "x64",
    "platform": "linux",
    "release": "22.04.4 LTS",
    "distro": "Ubuntu",
    "codename": "jammy",
    "virtual": true,
    "isRunningInDocker": false,
    "nodeVersion": "v20.15.0"
  },
  "enabledFeatures": {
    "feature": {
      "heartbeat": true,
      "dockerHealthCheck": true,
      "uptimeMonitor": true,
      "uptimeMonitorNewRelic": false,
      "udpServer": true,
      "eventCount": true,
      "rejectedEventCount": true,
      "userEvents": true,
      "userEventsMQTT": true,
      "userEventsInfluxdb": true,
      "userEventsNewRelic": false,
      "logEventsProxy": true,
      "logEventsScheduler": true,
      "logEventsRepository": true,
      "logEventCategorise": true,
      "logEventCategoriseRuleCount": 4,
      "logEventCategoriseRuleDefault": true,
      "logEventEnginePerformanceMonitor": true,
      "logEventEnginePerformanceMonitorNameLookup": true,
      "logEventEnginePerformanceMonitorTrackRejected": true,
      "logEventsMQTT": true,
      "logEventsInfluxdb": true,
      "logEventsNewRelic": false,
      "logdb": false,
      "mqtt": true,
      "newRelic": false,
      "prometheus": true,
      "influxdb": true,
      "influxdbVersion": 2,
      "appNames": true,
      "userSessions": true
    }
  }
}

The anonymous ID field

The id field deserves a bit more explanation.

It’s purpose is to uniquely identify the Butler SOS instance - nothing else.
If Butler SOS is stopped and started agagin the same ID will be used.
If reinstalled on a new server, or if the server’s network configuration changes, a new ID will be created.

Some sensitive information is used to create the ID, but as the ID is anonymized before sent as part of the telemetry data, no sensitive information leaves your servers.

The ID field is created as follows:

1. Combine the following information to a single string

   1. MAC address of default network interface
   2. IPv4 address of default network interface
   3. IP or FQDN of Sense server where repository service is running
   4. System unique ID as reported by the OS. Not all OSs support this though, which is why field 1-3 above are also needed to get a unique ID.

2. Run the created string through a one-way hashing/message digest function. Butler SOS uses Node.js’ own Crypto library to create a SHA-256 hash, using the default network interface’s MAC address as salt.
   Security is increased due to the fact that the salt never leaves the server where Butler is running.

   The bottom line is that it’s impossible to reverse the process and get the IP, host name etc used in step 1 above.
   Then again - this is cryptography, and there are no guarantees.
   But if you trust the certificates securing Sense itself, then the ID anonymization in Butler SOS should be ok too. Both are built on the same concepts of one-way cryptographic functions.

3. The result is a string that uniquely identifies the Butler SOS instance at hand, without giving away any sensitive data about the system where Butler is running.

See above for an example of what the id field looks like.
The id field is always shown during Butler startup.

Telemetry FAQ

1. What data is included in the telemetry messages?
   See above.
   The telemetry includes information about which Butler SOS features are enabled vs disabled. A unique, anonymized ID is included too, it’s unique to each Butler SOS instance and is used soley to distinguish between different Butler SOS instances.
   Finally some information about Butler SOS’s execution environment is included. Things like operating system, Node.js version used etc.

2. Can my Sense environment be identified via telemetry data?
   Short answer: No.
   Longer answer: No information about your Sense environment is sent as part of telemetry. No IP addresses or server names, no IDs of Sense apps/tasks/etc, no information about what actual data passed through Butler SOS, or any other data that can be linked to your Sense environment is included in the telemetry data.