InfluxDB & Grafana
Warning
Butler SOS was developed with InfluxDB version 1.x in mind.
InfluxDB is currently available in version 2.x and while this version brings lots of new goodies, it’s not out-of-the-box compatible with Butler SOS.
For that reason you should use the latest 1.x version of InfluxDB, which at the time of this writing is 1.8.4.
In due time Butler SOS will be updated to support InfluxDB 2.x too.
If you already have InfluxDB and/or Grafana running you can skip this section.
Running in Docker using docker-compose
The easiest way to get started is to run these tools in Docker containers, controlled by docker-compose files.
Running them under Kubernetes will give you a whole other level of fault tolerance, scalability etc - but this also requries much more when it comes to Kubernetes skills. Use the setup that’s relevant to your use case.
You can use a single docker-compose file for Butler SOS, InfluxDB and Grafana - or separate docker-compose files for each tool.
The advantage of using a single docker-compose file is that the entire stack of tools will be launched in unison. You can create dependencies between the tools if needed etc - very convenient. On the other hand, having separate docker-compose files makes it easier to restart (or upgrade or in other ways change) a single service without affecting other services.
Full stack docker-compose file
Let’s start Butler SOS, InfluxDB and Grafana from a single docker-compose_fullstack_influxdb.yml file:
➜ butler-sos-docker cat docker-compose_fullstack_influxdb.yml
# docker-compose_fullstack_influxdb.yml
version: "3.3"
services:
butler-sos:
image: ptarmiganlabs/butler-sos:latest
container_name: butler-sos
restart: always
volumes:
# Make config file and log files accessible outside of container
- "./config:/nodeapp/config"
- "./log:/nodeapp/log"
environment:
- "NODE_ENV=production_influxdb" # Means that Butler SOS will read config data from production_influxdb.yaml
logging:
driver: "json-file"
options:
max-file: "5"
max-size: "5m"
networks:
- senseops
influxdb:
image: influxdb:1.8.10
container_name: influxdb
restart: always
volumes:
- ./influxdb/data:/var/lib/influxdb # Mount for influxdb data directory
- ./influxdb/config/:/etc/influxdb/ # Mount for influxdb configuration
ports:
# The API for InfluxDB is served on port 8086
- "8086:8086"
- "8082:8082"
environment:
# Disable usage reporting
- "INFLUXDB_REPORTING_DISABLED=true"
networks:
- senseops
grafana:
image: grafana/grafana:latest
container_name: grafana
restart: always
ports:
- "3000:3000"
volumes:
- ./grafana/data:/var/lib/grafana
networks:
- senseops
networks:
senseops:
driver: bridge
➜ butler-sos-docker
Assuming you’ve already completed the setup of Butler SOS, the result of running the docker-compose_fullstack_influxdb.yml file above is something like this:
➜ butler-sos-docker docker-compose -f docker-compose_fullstack_influxdb.yml up
Creating network "butler-sos-docker_senseops" with driver "bridge"
Creating influxdb ... done
Creating butler-sos ... done
Creating grafana ... done
Attaching to butler-sos, grafana, influxdb
...
...
grafana | logger=grafanaStorageLogger t=2022-08-21T18:13:42.76538465Z level=info msg="storage starting"
grafana | logger=ngalert t=2022-08-21T18:13:42.780004463Z level=info msg="warming cache for startup"
grafana | logger=http.server t=2022-08-21T18:13:42.796364325Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl= socket=
grafana | logger=ngalert.multiorg.alertmanager t=2022-08-21T18:13:42.807894344Z level=info msg="starting MultiOrg Alertmanager"
butler-sos | 2022-08-21T18:13:42.908Z info: CONFIG: Influxdb enabled: true
butler-sos | 2022-08-21T18:13:42.911Z info: CONFIG: Influxdb host IP: influxdb
butler-sos | 2022-08-21T18:13:42.912Z info: CONFIG: Influxdb host port: 8086
butler-sos | 2022-08-21T18:13:42.912Z info: CONFIG: Influxdb db name: senseops
influxdb | ts=2022-08-21T18:13:43.139047Z lvl=info msg="Executing query" log_id=0cSPbmJG000 service=query query="SHOW DATABASES"
influxdb | [httpd] 172.24.0.2 - - [21/Aug/2022:18:13:43 +0000] "GET /query?p=&q=show+databases&u= HTTP/1.1" 200 84 "-" "-" fd854ac5-217c-11ed-8001-0242ac180003 1084
influxdb | ts=2022-08-21T18:13:43.169398Z lvl=info msg="Executing query" log_id=0cSPbmJG000 service=query query="CREATE DATABASE senseops"
influxdb | [httpd] 172.24.0.2 - - [21/Aug/2022:18:13:43 +0000] "POST /query?p=&q=create+database+%22senseops%22&u= HTTP/1.1 " 200 33 "-" "-" fd89e529-217c-11ed-8002-0242ac180003 2940
butler-sos | 2022-08-21T18:13:43.177Z info: CONFIG: Created new InfluxDB database: senseops
influxdb | ts=2022-08-21T18:13:43.219945Z lvl=info msg="Executing query" log_id=0cSPbmJG000 service=query query="CREATE RETENTION POLICY \"10d\" ON senseops DURATION 10d REPLICATION 1 DEFAULT"
influxdb | [httpd] 172.24.0.2 - - [21/Aug/2022:18:13:43 +0000] "POST /query?p=&q=create+retention+policy+%2210d%22+on+%22senseops%22+duration+10d+replication+1+default&u= HTTP/1.1 " 200 33 "-" "-" fd91ac84-217c-11ed-8003-0242ac180003 2299
butler-sos | 2022-08-21T18:13:43.242Z info: CONFIG: Created new InfluxDB retention policy: 10d
butler-sos | 2022-08-21T18:13:43.391Z info: --------------------------------------
butler-sos | 2022-08-21T18:13:43.391Z info: Starting Butler SOS
butler-sos | 2022-08-21T18:13:43.392Z info: Log level: verbose
butler-sos | 2022-08-21T18:13:43.393Z info: App version: 9.2.0
butler-sos | 2022-08-21T18:13:43.394Z info: Instance ID : 964cbd0a36bc....
butler-sos | 2022-08-21T18:13:43.394Z info:
butler-sos | 2022-08-21T18:13:43.395Z info: Node version : v18.7.0
butler-sos | 2022-08-21T18:13:43.396Z info: Architecture : x64
butler-sos | 2022-08-21T18:13:43.396Z info: Platform : linux
butler-sos | 2022-08-21T18:13:43.396Z info: Release : 11
butler-sos | 2022-08-21T18:13:43.397Z info: Distro : Debian GNU/Linux
butler-sos | 2022-08-21T18:13:43.397Z info: Codename : bullseye
butler-sos | 2022-08-21T18:13:43.398Z info: Virtual : false
butler-sos | 2022-08-21T18:13:43.398Z info: Processors : 4
butler-sos | 2022-08-21T18:13:43.399Z info: Physical cores : 4
butler-sos | 2022-08-21T18:13:43.399Z info: Cores : 4
butler-sos | 2022-08-21T18:13:43.400Z info: Docker arch. : undefined
butler-sos | 2022-08-21T18:13:43.400Z info: Total memory : 6233055232
butler-sos | 2022-08-21T18:13:43.401Z info: Standalone app : false
butler-sos | 2022-08-21T18:13:43.401Z info: --------------------------------------
butler-sos | 2022-08-21T18:13:43.402Z info: Client cert: /nodeapp/config/certificate/client.pem
butler-sos | 2022-08-21T18:13:43.402Z info: Client cert key: /nodeapp/config/certificate/client_key.pem
butler-sos | 2022-08-21T18:13:43.402Z info: CA cert: /nodeapp/config/certificate/root.pem
butler-sos | 2022-08-21T18:13:43.421Z verbose: MAIN: Anonymous telemetry reporting has been set up.
butler-sos | 2022-08-21T18:13:43.423Z verbose: MAIN: Starting Docker healthcheck server...
butler-sos | 2022-08-21T18:13:43.428Z info: USER EVENT: UDP server listening on 0.0.0.0:9997
butler-sos | 2022-08-21T18:13:43.429Z info: LOG EVENT: UDP server listening on 0.0.0.0:9996
butler-sos | 2022-08-21T18:13:43.461Z info: MAIN: Started Docker healthcheck server on port 12398.
butler-sos | 2022-08-21T18:13:43.462Z info: MAIN: Starting Prometheus Butler SOS endpoint on 0.0.0.0:9842.
butler-sos | 2022-08-21T18:13:43.464Z verbose: PROM: Setting up Prometheus client for server: sense1
butler-sos | 2022-08-21T18:13:43.465Z verbose: PROM: Setting up Prometheus client for server: sense2
butler-sos | 2022-08-21T18:13:43.482Z info: PROM: Prometheus Butler SOS metrics server now listening on port 9842
butler-sos | 2022-08-21T18:13:43.483Z info: PROM: Prometheus Node.js metrics server now listening on port 0.0.0.0:9001
butler-sos | 2022-08-21T18:13:45.080Z verbose: --------------------------------
butler-sos | 2022-08-21T18:13:45.081Z verbose: Iteration # 1, Uptime: 0 months, 0 days, 0 hours, 0 minutes, 2.007 seconds, Heap used 31.56 MB of total heap 60.81 MB. External (off-heap): 2.98 MB. Memory allocated to process: 102.28 MB.
influxdb | [httpd] 172.24.0.2 - - [21/Aug/2022:18:13:45 +0000] "POST /write?db=senseops&p=&precision=n&rp=&u= HTTP/1.1 " 204 0 "-" "-" feaf181f-217c-11ed-8004-0242ac180003 44267
butler-sos | 2022-08-21T18:13:45.137Z verbose: MEMORY USAGE INFLUXDB: Sent Butler SOS memory usage data to InfluxDB
butler-sos | 2022-08-21T18:13:45.198Z verbose: UPTIME NEW RELIC: Sent Butler SOS memory usage data to New Relic account 123456789 ("Ptarmigan Labs NR account")
...
...
From a separate shell we can then ensure that the expected Docker containers are running:
➜ ~ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2311d17d1285 ptarmiganlabs/butler-sos:latest "docker-entrypoint.s…" About a minute ago Up About a minute (healthy) butler-sos
a22307d12263 influxdb:1.8.10 "/entrypoint.sh infl…" About a minute ago Up About a minute 0.0.0.0:8082->8082/tcp, 0.0.0.0:8086->8086/tcp influxdb
81df665545d0 grafana/grafana:latest "/run.sh" About a minute ago Up About a minute 0.0.0.0:3000->3000/tcp grafana
➜ ~
That’s great, we now have a single command (docker-compose -f docker-compose_fullstack_influxdb.yml up -d for background/daemon mode) to bring up all the tools needed to monitor a Qlik Sense cluster!
Now, let’s see if any data has arrived in InfluxDB.
Let’s check this by going into Grafana, which is available on port 3000.
First time logging into a new Grafana instance you can use the default admin acount (username=admin, password=admin).
You will be asked to change that password during first login.
First add a data source in Grafana, pointing it to the local InfluxDB server.
Adding an InfluxDB datasource in Grafana
Now we can create a basic chart in Grafana, showing for example Butler SOS’ own memory usage.
After a while we should see some data in the chart:
Butler SOS' own memory usage, stored in InfluxDB and visualised in Grafana
Need to stop the entire stack of tools?
Easy - just run docker-compose -f docker-compose_fullstack_influxdb.yml down:
➜ butler-sos-docker docker-compose -f docker-compose_fullstack_influxdb.yml down
Stopping butler-sos ... done
Stopping influxdb ... done
Stopping grafana ... done
Removing butler-sos ... done
Removing influxdb ... done
Removing grafana ... done
Removing network butler-sos-docker_senseops
➜ butler-sos-docker
➜