I was researching how to monitor systemd services using Grafana/Prometheus when I came across this article. I learned systemd monitoring comes pre-loaded with node_exporter and just needs to be enabled. Then I discovered the Ansible role I use already enables it for me.

Neat! No work to do.

Or so I thought. Punching node_systemd_unit_state{instance="192.168.1.8:9100"} into Prometheus did not show user services, units that would be shown by systemctl --user.

Turns out node_exporter only collects for root.

systemd monitoring for USER services

I found the community collector systemd_exporter which has this capability, although it isn’t documented anywhere.

It’s a completely different binary than node_exporter, and thus has to be installed and enabled. Follow the install instructions or use the cloudalchemy/ansible-systemd-exporter Ansible role. The role starts another root systemd exporter, which is not what we’re interested in.

After installing the binary, set up yet-one-more systemd service in ~/.config/systemd/user/systemd_exporter.service which runs the same collector at a user level with some changes:

  1. Add --collector.user argument.
  2. Change the metrics port (I use 9559).
  3. Remove the User= and Group= lines from the [Service].

It should look something like this:

~/.config/systemd/user/systemd_exporter

#
# Ansible managed
#

[Unit]
Description=Prometheus SystemD Exporter
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/systemd_exporter \
    --collector.user \
    --web.listen-address=0.0.0.0:9559

SyslogIdentifier=systemd_exporter
Restart=always
RestartSec=1
StartLimitInterval=0

[Install]
WantedBy=multi-user.target

For completeness, this is my install-systemd_exporter.yml ansible playbook:

- hosts: all
  become: true
  roles:
    - cloudalchemy.systemd_exporter
- hosts: all
  tasks:
    - name: Copy user systemd_exporter.service
      ansible.builtin.copy:
        src: ../../templates/systemd/systemd_exporter.service
        dest: ~/.config/systemd/user/systemd_exporter.service
    - name: Enable systemd_exporter.service
      ansible.builtin.systemd:
        daemon-reload: true
        enabled: true
        state: started
        name: systemd_exporter
        scope: user

After running the playbook, or otherwise --user enable --now-ing systemd_exporter, I now have metrics on 0.0.0.0:9559 describing user services.

Neat.