Homelab

The source code for this simple site plus some standout snippets.

I was researching how to monitor systemd services using Grafana/Prometheus when I came across this article. I learned systemd monitoring comes pre-loaded with node_exporter and just needs to be enabled. Then I discovered the Ansible role I use already enables it for me. Neat! No work to do. Or so I thought. Punching node_systemd_unit_state{instance="192.168.1.8:9100"} into Prometheus did not show user services, units that would be shown by systemctl --user. Turns out node_exporter only collects for root. systemd monitoring for USER services I found the community collector systemd_exporter which has this capability, although it isn’t documented anywhere. It’s a completely different binary than node_exporter, and thus has to be installed and enabled. Follow the install instructions or use the cloudalchemy/ansible-systemd-exporter Ansible role. The role starts another root systemd exporter, which is not what we’re interested in. After installing the binary, set up yet-one-more systemd service in ~/.config/systemd/user/systemd_exporter.service which runs the same collector at a user level with some changes: Add --collector.user argument. Change the metrics port (I use 9559). Remove the User= and Group= lines from the [Service]. It should look something like this: ~/.config/systemd/user/systemd_exporter # # Ansible managed # [Unit] Description=Prometheus SystemD Exporter After=network-online.target [Service] Type=simple ExecStart=/usr/local/bin/systemd_exporter \ --collector.user \ --web.listen-address=0.0.0.0:9559 SyslogIdentifier=systemd_exporter Restart=always RestartSec=1 StartLimitInterval=0 [Install] WantedBy=multi-user.target For completeness, this is my install-systemd_exporter.yml ansible playbook: - hosts: all become: true roles: - cloudalchemy.systemd_exporter - hosts: all tasks: - name: Copy user systemd_exporter.service ansible.builtin.copy: src: ../../templates/systemd/systemd_exporter.service dest: ~/.config/systemd/user/systemd_exporter.service - name: Enable systemd_exporter.service ansible.builtin.systemd: daemon-reload: true enabled: true state: started name: systemd_exporter scope: user After running the playbook, or otherwise --user enable --now-ing systemd_exporter, I now have metrics on 0.0.0.0:9559 describing user services. Neat.

System crash when using igpu Documenting for future use. This is still not resolved as of: Date: 2023-01-21 Kernel: 6.1.6-060106-generic Mesa: mesa-va-drivers/kinetic 23.1~git2301210600.797b83~oibaf~k amd64 i915 Firmware: linux-firmware/kinetic-updates,now 20220923.gitf09bebf3-0ubuntu1.3 all [installed] I’m using a 13900k for my homelab, which has mostly been excellent. However the embedded GPU has major issues hardware transcoding video. I’m testing hardware transcoding on Jellyfin, as well as Tdarr’s Boosh-Transcode using QSV GPU & FFMPEG. Summary: the GPU eventually crashes, more frequently when using more streams. jellyfin-ffmpeg sometimes recovers and reverts to software transcoding, but more often just hangs until killed. FWIW this was also an issue on the 12th gen CPU I recently upgraded from, I’m not expecting this to be fixed anytime soon. So much for 2 codec engines 🌩 ecodes when transcoding using QSV: [ 534.980513] i915 0000:00:10.0: [drm] GPU HANG: ecode 12:1:a014184f, in tdarr-ffmpeg [2058] [ 534.980612] i915 0000:00:10.0: [drm] Resetting chip for stopped heartbeat on rcs0 [ 535.082049] i915 0000:00:10.0: [drm] tdarr-ffmpeg[1974] context reset due to GPU hang [ 535.082057] i915 0000:00:10.0: [drm] tdarr-ffmpeg[2058] context reset due to GPU hang [ 535.094994] i915 0000:00:10.0: [drm] GuC firmware i915/tgl_guc_70.bin version 70.5.1 [ 535.094998] i915 0000:00:10.0: [drm] HuC firmware i915/tgl_huc.bin version 7.9.3 [ 301.022234] i915 0000:00:10.0: [drm] GPU HANG: ecode 12:4:44df4a95, in ffmpeg [5460] [ 301.051882] i915 0000:00:10.0: [drm] GPU HANG: ecode 12:4:98f287b8, in ffmpeg [5262] [ 194.711129] i915 0000:00:10.0: [drm] GPU HANG: ecode 12:1:a9fa002d, in ffmpeg [5383] [ 194.712922] i915 0000:00:10.0: [drm] Resetting rcs0 for CS error [ 194.744843] i915 0000:00:10.0: [drm] GPU HANG: ecode 12:4:ec85561a, in ffmpeg [5383] [ 1182.201301] i915 0000:00:10.0: [drm] ffmpeg[13947] context reset due to GPU hang [ 1182.201328] i915 0000:00:10.0: [drm] GPU HANG: ecode 12:8:cc3768fb, in ffmpeg [13947] [ 135.855164] i915 0000:00:10.0: [drm] ffmpeg[5065] context reset due to GPU hang [ 143.000691] i915 0000:00:10.0: [drm] GPU HANG: ecode 12:4:9c5a5653, in ffmpeg [5065] [ 143.004613] i915 0000:00:10.0: [drm] Resetting vcs0 for CS error [ 143.004641] i915 0000:00:10.0: [drm] ffmpeg[5065] context reset due to GPU hang [ 150.907517] i915 0000:00:10.0: [drm] GPU HANG: ecode 12:4:9c595551, in ffmpeg [5065] [ 157.992504] i915 0000:00:10.0: [drm] GPU HANG: ecode 12:0:00000000 ecodes when transcoding using VAAPI: [ 272.803882] i915 0000:00:10.0: [drm] GPU HANG: ecode 12:4:cc2b051d, in ffmpeg [4529] Somewhat relevant links https://community.frame.work/t/hard-freezing-on-fedora-36-with-the-new-12th-gen-system/20675/23 https://gitlab.freedesktop.org/drm/intel/-/wikis/How-to-file-i915-bugs https://gitlab.freedesktop.org/drm/intel/-/issues/4858 https://community.frame.work/t/hard-freezing-on-fedora-36-with-the-new-12th-gen-system/20675/47 https://community.frame.work/t/hard-freezing-on-fedora-36-with-the-new-12th-gen-system/20675/138 https://forums.servethehome.com/index.php?threads/lga-1700-alder-lake-servers.35719/page-9

I’ve already been plugging in data from my Emporia Vue, which reads the meter from Southern California Edison and allows me to query my usage in kW/h from Home Assistant. I now need to calculate cost, which is variable based on what SCE calls TOU (Time Of Use). This essentially means peak hours (typically 4PM - 9PM) have higher rates than off-peak. I broke this problem down into 3 entities to solve: TOU Season (Summer vs Winter) TOU Peak (Current hourly block) TOU Cost ($ per kW/h) I can use Cost as the dynamic entity in our Energy Usage dashboard. This, combined with the kW/h reading, can be used to calculate total energy costs per day. This defines the 3 entities in my sce.yaml. If you’re copying this, plug your peak costs into the last few lines of this snippet. You can find updated rates here: https://www.sce.com/residential/rates/Time-Of-Use-Residential-Rate-Plans template: - sensor: - name: "TOU Season" state: "{{ ['Winter', 'Summer'][now().month >= 6 and now().month < 10] }}" icon: mdi:weather-cloudy-clock - sensor: - name: "TOU Peak" icon: mdi:calendar-clock state: > {% set is_weekend = now().strftime("%w") == 0 or now().strftime("%w") == 6 %} {% if states('sensor.tou_season') == "Summer" %} {% if now().hour >= 16 and now().hour < 21 %} {% if is_weekend %} {{ "Mid-Peak" }} {% else %} {{ "On-Peak" }} {% endif %} {% else %} {{ "Off-Peak" }} {% endif %} {% else %} {% if now().hour >= 16 and now().hour < 21 %} {{ "Mid-Peak" }} {% elif now().hour >= 21 or now().hour < 8 %} {{ "Off-Peak" }} {% else %} {{ "Super Off-Peak" }} {% endif %} {% endif %} - sensor: - name: "TOU Cost" icon: mdi:currency-usd device_class: monetary state: > {% if states('sensor.tou_season') == "Summer" %} {{ {"Off-Peak": 0.15, "On-Peak": 0.41, "Mid-Peak": 0.30}[states('sensor.tou_peak')] }} {% else %} {{ {"Super Off-Peak": 0.15, "Off-Peak": 0.15, "Mid-Peak": 0.38}[states('sensor.tou_peak')] }} {% endif %} Updated values for Option 1 TOUD-5-8PM: template: - sensor: - name: "TOU Season" state: "{{ ['Winter', 'Summer'][now().month >= 6 and now().month < 10] }}" icon: mdi:weather-cloudy-clock - sensor: - name: "TOU Peak" icon: mdi:calendar-clock state: > {% set is_weekend = now().strftime("%w") == 0 or now().strftime("%w") == 6 %} {% if states('sensor.tou_season') == "Summer" %} {% if now().hour >= 17 and now().hour < 20 %} {% if is_weekend %} {{ "Mid-Peak" }} {% else %} {{ "On-Peak" }} {% endif %} {% else %} {{ "Off-Peak" }} {% endif %} {% else %} {% if now().hour >= 17 and now().hour < 20 %} {{ "Mid-Peak" }} {% elif now().hour >= 20 or now().hour < 8 %} {{ "Off-Peak" }} {% else %} {{ "Super Off-Peak" }} {% endif %} {% endif %} - sensor: - name: "TOU Cost" icon: mdi:currency-usd device_class: monetary state: > {% if states('sensor.tou_season') == "Summer" %} {{ {"Off-Peak": 0.33, "On-Peak": 0.67, "Mid-Peak": 0.50}[states('sensor.tou_peak')] }} {% else %} {{ {"Super Off-Peak": 0.32, "Off-Peak": 0.36, "Mid-Peak": 0.55}[states('sensor.tou_peak')] }} {% endif %}

As of Dec 2021, this is now outdated. I am no longer using Dendron, nor nomad to deploy the site. I’ll have a follow up shortly. This expands on the DroneCI -> Nomad pipeline I examined earlier I explained why I use Dendron, today I will show how. The glue that binds Dendron, Quantified Self reports, and Hugo is DroneCI. The CI build runs every night and immediately when a commit is made to the Hugo site. I wrote a custom Pod for exporting notes. It’s similar to the built-in markdown pod, but includes frontmatter and renders a hierarchical structure which makes Hugo happy. The server completes these build steps: Clone the required build repos from Github / Gitea: Hugo site workspace Hugo site theme Dendron workspace and notes Dendron <-> Hugo export pod Nomad config files Once the files are in order, we install dendron-cli and link the hugo-pod in an node container Apply the included patch that skips validation of the pod Run the export pod, creating a well formed /garden/ subdirectory in Hugo’s content folder Build the Hugo site Build the final Docker image with the new public directory Publish the final Docker image Send a redeploy command to Nomad using the config cloned in step 1. Prune old Docker images from our registry Done! Simple, no? 😆 The complexity stems from the sources being so scattered (Dendron, Hugo, Nomad…). Obviously this isn’t practical for casual note writing. But in my head, if it’s worth doing it’s also worth overdoing. This is the full .drone.yml spec: kind: pipeline type: docker name: default clone: git: image: plugins/git recursive: true submodule_override: "themes/novela": https://git.quinncasey.com/qcasey/hugo-theme-novela steps: - name: "Clone hugo theme" image: alpine/git commands: - sed -ir 's$:$/$' .gitmodules - sed -ir 's$git@ssh.$https://$g' .gitmodules - git submodule update --recursive --init - name: "Clone Dendron notes and export pod" image: alpine/git commands: - git clone https://git.quinncasey.com/qcasey/hugo-pod.git - git clone https://git.quinncasey.com/qcasey/notes-framework.git notes - cd notes - git clone https://git.quinncasey.com/qcasey/notes.git garden - name: "Export Dendron notes" image: node:current-alpine3.11 commands: - npm install -g @dendronhq/dendron-cli - cd hugo-podand - npm install # Skip validation error "data must NOT have additional properties" - patch node_modules/@dendronhq/pods-core/lib/utils.js < utils.js.patch - npm link - cd ../notes/ - npm link hugo-pod - dendron-cli exportPod --wsRoot ./ --podId hugo --podPkg hugo-pod --podSource custom --config "fname=dendron,vaultName=vault,dest=../content,includeStubs=true" - name: "Build site with hugo" image: klakegg/hugo:ext-alpine commands: - hugo - name: publish image: plugins/docker settings: registry: registry.quinncasey.com repo: registry.quinncasey.com/garden-quinncasey tags: - latest - ${DRONE_BUILD_NUMBER} username: casey password: from_secret: registry_password - name: clone deployment dotfiles image: alpine/git commands: - git clone -b abathur https://git.quinncasey.com/qcasey/dotfiles - name: redeploy nomad image: multani/nomad commands: - nomad run -address=http://192.168.1.196:4646 -var=image_id=${DRONE_BUILD_NUMBER} dotfiles/nomad/webserver/quinncasey.com.nomad - name: prune images image: anoxis/registry-cli:latest commands: - /registry.py -l casey:$PASSWORD -r https://registry.quinncasey.com -i garden-quinncasey --delete --num 10 --keep-tags "stable" "latest" environment: PASSWORD: from_secret: registry_password

Redeploying container images built in CI/CD to a Nomad cluster.

Digital Garden I like the idea of writing with the garage door open. I write notes to help myself a year from now. But if I publish an obscure thing that helps someone do something cool, I wouldn’t complain. Our natural fear of being judged leads most people to build, learn, and think privately. But seeking validation should not be the goal of learning in public. ~ Anne-Laure Le Cunff My sites are hosted in my Homelab, these notes are written in Dendron. I publish the site using Hugo. Plain text is wonderful. It allows me to easily find ideas, references, links, personals notes, tasks, thoughts and everything else I want to keep handy. Even short jots and thoughts deserve their space. Being useful for me is the primary use case for this space on the internet. It’s not that I don’t care about you, but this is for me. It’s here so I can record what I think and know and preserve it in time and space. It’s my garden, but I’m happy for you to hang around and eat tomatos with me. https://joelhooks.com/on-writing-more In it’s current iteration, digital gardens take active effort on the part of the reader to wade through links to related pieces of content. This is very unlike a real garden: you don’t have to be an expert at horticulture or garden design to appreciate the overall landscape. You can easily engage with many different levels of a real garden at the same time, telescoping in and out at will to first examine the minutae of informational placards or specific plants and then returning to a broad perspective of the scenery as a whole. https://vivqu.com/blog/2020/10/18/digital-gardens/ Links to other gardens nikitavoloboev You and your mind garden Building a digital garden A Renaissance of Open Thinking and Curated Writing on the Web Digital gardens Work with the garage door up brendex beepb00p My blog is a digital garden, not a blog Julian’s lifelog and digital playground Alex’s Notes YouTube Creators Linux The Linux Experiment Linux For Everyone Techno Tim ExplainingComputers Makers Estefannie Explains It All JetsonHacks MickMake N-O-D-E Zack’s Lab Strange Parts Maker’s Muse Make Anything

I try to self-host the software and services I use on a daily basis.

To get data from SCE power meter, some options: https://www.amazon.com/gp/product/B084T6HGNR/ref=ox_sc_act_title_1?smid=A2MZON57HPVTEJ&psc=1 Integrations to Home Assistant: https://community.home-assistant.io/t/emporia-vue/178737 https://github.com/magico13/ha-emporia-vue with $25 rebate: https://www.sce.com/residential/rebates-savings/hanlogin Reddit thread: https://www.reddit.com/r/homeassistant/comments/im49ud/is_it_possible_to_get_data_from_my_utilitys_meters/ with generic utility radio hack: https://github.com/bemasher/rtlamr

Home Assistant only creates sensors for VM states, and does not yet provide a way to control those VMs. If you want to simply shutdown/startup a VM from Home Assistant, the rest_command is your friend.

A Ulauncher extension to view and control devices in your Home Assistant instance.

Python script for decoding Sunpower inverter packets and recording solar production, power consumption, and temperature in Home Assistant.

Generate boot image from here: https://github.com/kholia/OSX-KVM With this as a guide for proxmox: https://manjaro.site/how-to-install-macos-big-sur-on-proxmox-ve/ GenSMBIOS, ProperTree, Hackintool are REQUIRED! Post Install Use Clover Configurator to mount EFI disk of root and OpenCore install disk. Copy OpenCore install disk EFI files to root efi. Open config.plist into GenSMBIOS, use the settings there. Use Hackintool to check that en0 is built-in. If not, use ProperTree to edit the plist as described here FIXME NSKeyedArchiver Notes

The Home Assistant Kodi Integration is nice, but as of writing it becomes slower the longer Home Assistant runs. This sucks, especially when automations (like turning on a power hungry receiver) don't run because of it.