Configuring log events

Optional

These settings are optional.

If you don’t do anything log events are turned off by default.

What’s this?

Butler SOS log events are designed to be a replacement for the most important/useful aspects of Qlik Sense’ log database, which was removed from Qlik Sense Enterprise on Windows in mid 2021.

The log events capture warnings, errors and fatals from the various QSEoW subsystems.
These events used to be sent to the PostgreSQL logging database, most (but not all) are also sent to QSEoW’s log files.

Using Butler SOS’ log events is arguably even better than getting the same information from log db:
Log db had to be polled to detect new log events and this polling could realistically only be done every few minutes. It also put additional load on an often already struggling part of many QSEoW clusters.

With Butler SOS’ log events concept the notifications are almost instantaneous.
Errors and warnings show up in the Grafana or New Relic dashboards within seconds after taking place in QSEoW.

Log events rely on two things to work:

Settings in the Butler SOS config file.
Log appender XML files being deployed on the Sense servers where log events should be captured.

Both are described below.

Info

As of Butler SOS version 9.2, log events are captured in these QSEoW services:

Engine
Proxy
Repository
Scheduler

Support for additional modules is reasonably easy to add, please create a ticket if you believe some service should be added to the list above.

Tech deep-dive

The underlying mechanism is the same as described on the user events page.

Tagging of data

Categorising log events

Log events can optionally be categorised by Butler SOS.
The reason for categorising log events is to make it easier to make Sense of the potentially large number of log events that can be generated by a QSEoW cluster.

For example, if a QSEoW cluster creates 1000 warnings and errors per hour, it’s difficult to know which warnings are important and which are not.
By categorising the messages, for example by the subsystem that generated them and/or what they refer to, it’s easier to understand what’s going on.

Possible ways of categorising log events could be “access denied” issues, “user directory” issues, “app reload failed” issues, “general engine issues” etc.

Log events can be categorised in any number of ways, but the template config file contains a few examples.
Specifically:

Access denied issues
1. Captures log events of severity WARN and ERROR.
2. Match log events aginst two filters.
  1. Log message starts with “Access was denied for User:”.
  2. Log message contains “was denied for User
3. Categorises them by setting category tag qs_log_category to access-denied.
AD issues
1. Captures log events of severity WARN and ERROR.
2. Match log events against one filter.
  1. Log message starts with “Duplicate entity with userId”.
3. Categorises them by setting category tag qs_log_category to user-directory.
Qlik Sense service down
1. Captures log events of severity WARN.
2. Match log events against two filters.
  1. Log message starts with “Failed to request service alive response from”.
  2. Log message contains “Unable to connect to the remote server”.
3. Categorises them by setting category tag qs_log_category to qs-service.
Reload task failed
1. Captures log events of severity WARN and ERROR.
2. Match log events against four filters.
  1. Log message starts with “Task finished with state FinishedFail”.
  2. Log message starts with “Task finished with state Error”.
  3. Log message ends with “Reload failed in Engine. Check engine or script logs.”.
  4. Log message starts with “Reload sequence was not successful (Result=False, Finished=True, Aborted=False) for engine connection with handle”.
3. Categorises them by setting category tag qs_log_category to reload-failed.
If no rules match the log event, it will be categorised as unknown.
1. The default rule can be enabled/disabled in the config file via the ruleDefault.enable parameter.

Note 1: It is possible to assign one or more categories to a log event. This provides flexibility in how you later create dashboards in for example Grafana or New Relic.

Note 2: The sample/template config file may be updated with more examples in the future.

Tip

It is also possible to drop log events that match a certain pattern, for example if you are never interested in seeing them in Grafana or New Relic.
This is done by setting the action parameter to drop in the config file, for that particular rule.

Detailed engine performance metrics

App perforamnce

Reload and internal tasks performance

InfluxDB

The tags added to InfluxDB are described in the reference documentation for log events.

Log categories are added as tags to InfluxDB datapoints.

New Relic

The following attributes (which is New Relic lingo for tags) are added:

A core set of attributes are added to all user events. Note that some attributes will be empty for some/many log events.
1. qs_ts_iso: Event timestamp in ISO format.
2. qs_ts_local: Event timestamp in local (server) time zone.
3. qs_log_source: Which Sense service the event originated in, for example “qseow-proxy”, “qseow-repository”.
4. qs_log_level: Log level of the event. “WARN”, “ERROR”, or “FATAL”.
5. qs_host: Host name of the Sense server the event originated at.
6. qs_subsystem: Which part of each Sense service the event originated in, for example “System.Proxy.Proxy.Core.RequestListener”, “System.Engine.Engine”.
7. qs_windows_user: Name of the Windows user that’s used to run the Window service where the event originated.
8. qs_message: Event message.
9. qs_exception_message: Additional information about the event.
10. qs_user_full: Full directory/user ID of the user the event is about. Will be scrambled if scrambling enabled in config file.
11. qs_user_directory: User directory of the user the event is about. Will be scrambled if scrambling enabled in config file.
12. qs_user_id: User ID of the user the event is about. Will be scrambled if scrambling enabled in config file.
13. qs_command: What command (if any) caused the event. Example: “Doc::DoSave”, “Doc::CreateObject”.
14. qs_result_code: Result code as reported by Sense. Usually empty.
15. qs_origin: What kind of activity caused the event, for example “AppAccess”.
16. qs_context: Additional information about the event.
17. qs_task_name: Task name (if any) causing the event.
18. qs_app_name: App name (if any) causing the event.
19. qs_task_id: Task ID (if any) causing the event.
20. qs_app_id: App ID (if any) causing the event.
21. qs_execution_id: Execution ID as reported by Sense.
22. qs_proxy_session_id: Proxy session ID as reported by Sense.
23. qs_engine_ts: Engine timestamp (if any) associated with the event.
24. qs_process_id: Process ID of engine service.
25. qs_engine_exe_version: Version of engine service’s EXE file.
26. qs_server_started: Timestamp when Sense engine service was started.
27. qs_entry_type: Entry type as reported by Sense. Usually empty.
28. qs_session_id: Session ID as reported by Sense.
Custom attributes defined in the Butler SOS config file’s Butler-SOS.logEvents.tags section.
Custom attributes defined in the Butler SOS config file’s Butler-SOS.newRelic.event.attribute.static section.
Dynamic attributes
1. butlerSosVersion: Butler SOS version. Enabled by setting Butler-SOS.newRelic.event.attribute.dynamic.butlerSosVersion.enable to true in config file.

Note: Attributes defined further down in the list above will overwrite already defined attributes if their names match.
To avoid problems you should make sure not to use already defined attributes.

Log categories are currently NOT included in the data sent to New Relic.

Settings in main config file

Tip

The config snippet below comes from the production_template.yaml file.

Being a template, it contains examples on how configuration may be done - not necessarily how it should be done.
For example, the env/DEV and foo/bar tags are optional and can be changed to something else, or removed all together if not used.

Butler-SOS:
  ...
  ...
  # Log events are used to capture Sense warnings, errors and fatals in real time
  logEvents:
    udpServerConfig:
      serverHost: <IP or FQDN>         # Host/IP where log event server will listen for events from Sense
      portLogEvents: 9996              # Port on which log event server will listen for events from Sense
    tags:
      # - name: env
      #   value: DEV
      # - name: foo
      #   value: bar
    source:
      engine:
        enable: false                  # Should log events from the engine service be handled?
      proxy:
        enable: false                  # Should log events from the proxy service be handled?
      repository:
        enable: false                  # Should log events from the repository service be handled?
      scheduler:
        enable: false                  # Should log events from the scheduler service be handled?
    categorise:                        # Take actions on log events based on their content
      enable: false
      rules:                           # Rules are used to match log events to filters
        # - description: Find access denied errors
        #   logLevel:                    # Log events of this Log level will be matched. WARN, ERROR, FATAL. Case insensitive.
        #     - WARN
        #     - ERROR
        #   action: categorise           # Action to take on matched log events. Possible values are categorise, drop
        #   category:                    # Category to assign to matched log events. Name/value pairs. 
        #                                # Will be added to InfluxDB datapoints as tags.
        #     - name: qs_log_category
        #       value: access-denied
        #   filter:                      # Filter used to match log events. Case sensitive.
        #     - type: sw                 # Type of filter. sw = starts with, ew = ends with, so = substring of
        #       value: "Access was denied for User:"
        #     - type: so
        #       value: was denied for User
        # - description: Find AD issues
        #   logLevel:                    # Log events of this Log level will be matched. WARN, ERROR, FATAL. Case insensitive.
        #     - ERROR
        #     - WARN
        #   action: categorise           # Action to take on matched log events. Possible values are categorise, drop
        #   category:                    # Category to assign to matched log events. Name/value pairs. 
        #                                # Will be added to InfluxDB datapoints as tags.
        #     - name: qs_log_category
        #       value: user-directory
        #   filter:                      # Filter used to match log events. Case sensitive.
        #     - type: sw                 # Type of filter. sw = starts with, ew = ends with, so = substring of
        #       value: Duplicate entity with userId
        # - description: Qlik Sense service down
        #   logLevel:                    # Log events of this Log level will be matched. WARN, ERROR, FATAL. Case insensitive.
        #     - WARN
        #   action: categorise           # Action to take on matched log events. Possible values are categorise, drop
        #   category:                    # Category to assign to matched log events. Name/value pairs. 
        #                                # Will be added to InfluxDB datapoints as tags.
        #     - name: qs_log_category
        #       value: qs-service
        #   filter:                      # Filter used to match log events. Case sensitive.
        #     - type: sw                 # Type of filter. sw = starts with, ew = ends with, so = substring of
        #       value: Failed to request service alive response from
        #     - type: so                 # Type of filter. sw = starts with, ew = ends with, so = substring of
        #       value: Unable to connect to the remote server
        # - description: Reload task failed
        #   logLevel:                    # Log events of this Log level will be matched. WARN, ERROR, FATAL. Case insensitive.
        #     - WARN
        #     - ERROR
        #   action: categorise           # Action to take on matched log events. Possible values are categorise, drop
        #   category:                    # Category to assign to matched log events. Name/value pairs. 
        #                                # Will be added to InfluxDB datapoints as tags.
        #     - name: qs_log_category
        #       value: reload-failed
        #   filter:                      # Filter used to match log events. Case sensitive.
        #     - type: sw                 # Type of filter. sw = starts with, ew = ends with, so = substring of
        #       value: Task finished with state FinishedFail
        #     - type: sw                 # Type of filter. sw = starts with, ew = ends with, so = substring of
        #       value: Task finished with state Error
        #     - type: ew                 # Type of filter. sw = starts with, ew = ends with, so = substring of
        #       value: Reload failed in Engine. Check engine or script logs.
        #     - type: sw                 # Type of filter. sw = starts with, ew = ends with, so = substring of
        #       value: Reload sequence was not successful (Result=False, Finished=True, Aborted=False) for engine connection with handle
      ruleDefault:                     # Default rule to use if no other rules match the log event
        enable: true
        category:
          - name: qs_log_category
            value: unknown
    enginePerformanceMonitor:           # Detailed app performance data extraction from log events
      enable: false                     # Should app performance data be extracted from log events?
      appNameLookup:                    # Should app names be looked up based on app IDs?
        enable: false
      trackRejectedEvents: 
        enable: false                   # Should events that are rejected by the app performance monitor be tracked?
        tags:                           # Tags are added to the data before it's stored in InfluxDB
          # - name: env
          #   value: DEV
          # - name: foo
          #   value: bar
      monitorFilter:                    # What objects should be monitored? Entire apps or just specific object(s) within some specific app(s)?
                                        # Two kinds of monitoring can be done:
                                        # 1) Monitor all apps, except those listed for exclusion. This is defined in the allApps section.
                                        # 2) Monitor only specific apps. This is defined in the appSpecific section.
                                        # An event will be accepted if it matches any of the rules in the allApps section OR any of the rules in the appSpecific section.
        allApps:
          enable: false                 # Should all apps be monitored?
          appExclude:                   # What apps should be excluded from monitoring?
                                        # If both appId and appName are specified, both must match the event's data for it to be considered a match.
            # - appId: 5b817efe-472d-43ce-8a31-6cce34af7de9
            # - appName: Sales forecast
            # - appId: f42d6b16-8faf-45ca-a783-59f9da47db6e
            #   appName: Inventory analysis
          objectType:
            allObjectTypes: true        # Should all object types be monitored?
            allObjectTypesExclude:      # If allObjectTypes is set to true, the object types in this array are excluded from monitoring. 
                                        # someObjectTypesInclude (below) is ignored in that case.
              # - LoadModelList
              # - <Unknown>
              # - linechart
              # - map
            someObjectTypesInclude:     # What object types should be included in monitoring?
                                        # Only applicable if allObjectTypes is set to false.
              # - LoadModelList
              # - sheet
              # - barchart
          method:
            allMethods: true            # Should all methods be monitored?
            allMethodsExclude:          # If allMethods is set to true, the methods in this array are excluded from monitoring.
                                        # someMethodsInclude (below) is ignored in that case.
              # - Global::OpenApp
              # - Doc::GetAppLayout
              # - Doc::CreateSessionObject
            someMethodsInclude:         # What methods should be included in monitoring?
                                        # Only applicable if allMethods is set to false.
              # - GenericObject::GetLayout
              # - GenericObject::GetHyperCubeContinuousData
        appSpecific:
          enable: false                  # Should app specific monitoring be done?
          app:
            - include:                  # What apps should be monitored?
                                        # If both appId and appName are specified, both must match the event's data for it to be considered a match.
                # - appId: d7cf16f9-6a95-462a-9ff1-a6d413326de4
                # - appName: Budget 2025
                # - appId: 6931136d-c234-4358-a40c-e37153aba7c9
                #   appName: Sales basket analysis
              objectType:
                allObjectTypes: true   # Should all object types be monitored?
                allObjectTypesExclude:  # If allObjectTypes is set to true, the object types in this array are excluded from monitoring. 
                                        # someObjectTypesInclude (below) is ignored in that case.
                  # - table
                  # - map
                someObjectTypesInclude: # What object types should be included in monitoring?
                                        # Only applicable if allObjectTypes is set to false.
                  # - sheet
                  # - barchart
                  # - linechart
                  # - map
              appObject:
                allAppObjects: true     # Should all app objects be monitored?
                allAppObjectsExclude:   # If allAppObjects is set to true, the app objects in this array are excluded from monitoring.
                                        # someAppObjectsInclude (below) is ignored in that case.
                  # - objectId: AaBbCc
                  # - objectId: DdEeFf
                someAppObjectsInclude:  # What app objects should be included in monitoring?
                                        # Only applicable if allAppObjects is set to false.
                  # - objectId: YJEpPT    
              method: 
                allMethods: true        # Should all methods be monitored?
                allMethodsExclude:      # If allMethods is set to true, the methods in this array are excluded from monitoring.
                                        # someMethodsInclude (below) is ignored in that case.
                  # - Global::OpenApp
                  # - Doc::GetAppLayout
                  # - Doc::CreateSessionObject
                someMethodsInclude:     # What methods should be included in monitoring?
                                        # Only applicable if allMethods is set to false.
                  # - GenericObject::GetLayout
                  # - GenericObject::GetHyperCubeContinuousData
    sendToMQTT:
      enable: false                    # Should log events be sent as MQTT messages?
      baseTopic: qliksense/logevent    # What topic should log events be forwarded to? 
      postTo:
        baseTopic: true
        subsystemTopics: true          # Should log events be sent to subtopics corresponding to the QSEoW subsystems where the events originated?
    sendToInfluxdb:
      enable: false                    # Should log events be stored in InfluxDB?
    sendToNewRelic:
      enable: false                    # Should log events be sent to New Relic?
      destinationAccount:
        # - First NR account
        # - Second NR account
      source:
        engine:
          enable: true                 # Should log events from the engine service be handled?
          logLevel: 
            error: true                # Should error level log events be handled by Butler SOS?
            warn: true                 # Should warning level log events be handled by Butler SOS?
        proxy:
          enable: true                 # Should log events from the proxy service be handled?
          logLevel: 
            error: true                # Should error level log events be handled by Butler SOS?
            warn: true                 # Should warning level log events be handled by Butler SOS?
        repository:
          enable: true                 # Should log events from the repository service be handled?
          logLevel: 
            error: true                # Should error level log events be handled by Butler SOS?
            warn: true                 # Should warning level log events be handled by Butler SOS?
        scheduler:
          enable: true                 # Should log events from the scheduler service be handled?
          logLevel: 
            error: true                # Should error level log events be handled by Butler SOS?
            warn: true                 # Should warning level log events be handled by Butler SOS?
  ...
  ...

Log appender XML files

Sample log appender files are available in the ZIP file available from the download page, in subfolders engine/proxy/repository/scheduler of config/log_appender_xml/ folder.

Note that the log appender files contain slightly different information for each Sense service (engine/proxy/repository/scheduler)!
Also keep in mind that the log appender files must be called LocalLogConfig.xml and placed in these directories on the all Sense servers:

C:\ProgramData\Qlik\Sense\Engine
C:\ProgramData\Qlik\Sense\Proxy
C:\ProgramData\Qlik\Sense\Repository
C:\ProgramData\Qlik\Sense\Scheduler

Tip

If you have more than one Sense server you strictly speaking don’t have to deploy log appenders to all servers.

If you are only interested in receiving log events from some servers and/or services (engine, proxy, repository, scheduler) - deploy the log appender files there.