[{"content":"I have multiple containers which need to write into the same media folders. Copyparty runs as UID 924, the arr applications run as UID 911 and Frigate uses UID/GID 914 on the host. They need to create, rename and delete each other\u0026rsquo;s files without running all services as the same user.\nI faced three related issues:\nCopyparty ran as UID 924 and shared media with the arr stack at UID 911. Frigate had to initialize as root in its container while retaining host ownership as UID/GID 914. A mergerfs view combined two branches whose copies of the same directory had different modes. Fixing only the owner or group was not enough. I needed idmapped mounts, correct directory modes, application-specific chmod settings and inherited ACLs.\nWhy a writable file could not be replaced For a process to replace movies/title/video.mkv, it needs write and execute on movies/title/. Write permission on video.mkv controls modification of the file\u0026rsquo;s contents; it does not grant permission to remove that directory entry.\nThree inputs decide the normal Unix permission check:\nUID/GID ownership chooses the owner, group, or other class. The inode mode or access ACL says what that class may do. The containing directory controls create, rename, and delete. flowchart TB PROC[Radarr uid 911, media group] --\u003e CLASS{Matching class?} CLASS --\u003e|directory gid media| GROUP[Use group mode or ACL] GROUP --\u003e|r-x on directory| DENY[Read and traverse, cannot replace] GROUP --\u003e|rwx on directory| ALLOW[Create, rename, and delete] FILE[Writable target file] -. does not decide unlink .-\u003e DENY Ownership chooses a permission class; directory permissions decide whether another service can rename or delete an entry. So the first check was the owner and mode of the parent directory, not only the movie file.\nCopyparty rename blocked a Radarr import Radarr failed while upgrading a movie:\n1 Access to the path \u0026#39;.../Run (2002)/Run (2002) SDTV.mp4\u0026#39; is denied The destination directory told the story:\n1 drwxr-sr-x 924 media Run (2002)/ UID 924 was Copyparty. GID media was correct, and the setgid bit ensured new children inherited that group. But the group had r-x, not rwx. Radarr\u0026rsquo;s UID 911 could enter the directory and read the old file but could not unlink it.\nI had set UMASK=002 on the container. Copyparty\u0026rsquo;s image did not honor that for this operation and produced a 0755 directory. The permanent fix was application-level creation policy:\n1 2 3 flags: chmod_f: 664 chmod_d: 775 I applied that policy to the Copyparty volumes shared with the arr stack, qBittorrent, and Picard. Existing entries needed a one-time repair because creation policy is not retroactive.\nFixing both mergerfs branches My first repair changed the affected directory under the btank backing tree. The error persisted. The visible media path was a mergerfs union of btank and btwo, and the directory existed on both branches:\n1 2 3 btank branch: drwxrwsr-x 924 media btwo branch: drwxr-sr-x 924 media merged view: drwxr-sr-x mergerfs surfaced metadata from the stale btwo copy. With default_permissions, the kernel enforced the mode visible through the union. The branch containing the media file was not necessarily the branch supplying the directory metadata.\nflowchart TB VIEW[\"/mnt/merged-media/title\"] --\u003e PICK{Metadata selected by mergerfs} B1[\"/mnt/btank/@media/title: 2775\"] --\u003e PICK B2[\"/mnt/btwo/@media/title: 2755\"] --\u003e PICK PICK --\u003e|stale branch wins| MODE[Visible mode 2755] MODE --\u003e DENIED[Radarr group write denied] Repairing only the branch with the file left a stale directory mode on the other mergerfs branch. The correct repair swept every raw backing branch, never only the merged view:\n1 2 find /mnt/btank/@media/movies /mnt/btwo/@media/movies \\ -type d ! -perm -2775 -exec chmod 2775 {} + After both branch copies were corrected, Radarr deleted the old file and completed the import.\nFrigate runs as root inside the container Frigate\u0026rsquo;s official image starts through s6-overlay and needs container root for initialization. Forcing --user 914:914 broke startup. Running it rootful with a plain bind mount would create host root:root state, which would break the native rollback path and Copyparty access.\nA per-bind idmapped mount solved ownership without changing the process:\n1 idmap=uids=914-0-1;gids=914-0-1 Podman\u0026rsquo;s triplet is backing ID, mapped ID, length. Through that mount, host 914:914 appears as 0:0 to the container. Frigate can initialize as root and its files still land on btrfs and ZFS as host 914:914.\nflowchart LR F[Frigate container root 0:0] --\u003e M[Idmapped bind mount] M --\u003e|uids 914-0-1, gids 914-0-1| DISK[Backing files 914:914] N[Native rollback process 914:914] --\u003e DISK C[Copyparty uid 924 plus group 914] --\u003e DISK A per-bind idmap changes the ownership view for one mount; it does not recursively mutate the backing files. This is different from the alternatives:\nMechanism What it changes Why I did or did not use it --user 914:914 Process identity Broke Frigate\u0026rsquo;s root-requiring init :U Recursively changes backing ownership Mutates data and is expensive Whole-container user namespace Ownership view for the container Complicated GPU and group mappings Per-bind idmap Ownership view for one mount Preserved both image and host contracts s6 reset the process umask In a disposable directory, --umask=0002 produced directories at 0775 and files at 0664. Copyparty, running as UID 924 with supplementary GID 914, could create, rename, and delete them.\nThe real Frigate processes produced 2755 directories and 0644 files. s6-overlay had reset the live process umask to 0022 after Podman launched the container. A shell test that bypassed the supervisor had proved the kernel and idmap mechanics, not the application\u0026rsquo;s final runtime behavior.\nThe reliable check was the live process state:\n1 grep \u0026#39;^Umask:\u0026#39; /proc/PID/status Because the image did not preserve the desired umask, I used default ACLs on the recordings tree. Newly created directories inherit group write even when the process requests a stricter base mode. Copyparty also joins supplementary GID 914, and its own creation policy emits group-writable entries when it writes into the same tree.\nFinal permission setup The working setup uses:\nstable host ownership through Frigate\u0026rsquo;s per-bind idmap; shared or supplementary groups selecting the intended permission class; setgid directories preserving group ownership; application chmod_f and chmod_d policy where the image ignored UMASK; inherited default ACLs where s6 reset the process umask; repairs performed on every mergerfs backing branch. For each shared mount, I test read, create, modify, rename, delete, mkdir and rmdir using the real service UIDs and groups. I also test files created by the actual application, not only a shell started in the same image. This was important for Frigate because s6 changed the umask after Podman started the container.\nIf mergerfs is involved, check and repair every backing branch. The permissions shown through the merged path may come from a different branch than the file being modified.\n","date":"2026-08-17T00:00:00+05:30","permalink":"https://blog.dexome.com/post/shared-file-permissions-across-containers/","title":"Sharing Files Between Containers with Different UIDs"},{"content":"I run Netdata as a native NixOS service and restrict its network and device access using systemd. After updating to Netdata 2.10.3, the dashboard worked but the SMART, ZFS pool and Traefik collectors did not create any charts.\nAll three appeared as missing collectors in the UI, but each one had a different problem. Below are the settings I needed on NixOS while keeping the service restricted.\nRestricting network and device access The service could reach only the network ranges it actually used:\n1 2 3 4 5 6 IPAddressDeny = [ \u0026#34;any\u0026#34; ]; IPAddressAllow = [ \u0026#34;localhost\u0026#34; rproxyBridgeSubnet notifyBridgeSubnet ]; That blocked cloud connectivity, telemetry, and arbitrary egress independently of package build flags. Device access was also explicit so the Intel GPU collector could read its render node:\n1 DeviceAllow = [ \u0026#34;/dev/dri/renderD128 rw\u0026#34; ]; Once DeviceAllow is set, all devices not listed there are denied. This caused the first collector failure.\nSMART collector could not open the disk Adding any DeviceAllow entry makes systemd\u0026rsquo;s device policy deny-by-default. The render node was allowed; every unlisted block and character device was not.\nNetdata\u0026rsquo;s privileged helper ran smartctl as root and still received:\n1 Smartctl open device: /dev/sda failed: Operation not permitted UID 0 could not override the device cgroup. The denial applied to every process in the service cgroup, including privileged helpers.\nThe narrow fix was to allow the disk collector\u0026rsquo;s device too:\n1 2 3 4 5 DeviceAllow = [ \u0026#34;/dev/dri/renderD128 rw\u0026#34; \u0026#34;/dev/sda rw\u0026#34; \u0026#34;/dev/zfs rw\u0026#34; ]; flowchart LR N[Netdata service cgroup] --\u003e GPU[\"/dev/dri/renderD128 allowed\"] N --\u003e S{smartctl as root} S --\u003e|before| DENY[\"/dev/sda denied by cgroup\"] S --\u003e|after explicit allow| DISK[\"/dev/sda readable\"] N --\u003e|after explicit allow| ZFS[\"/dev/zfs accessible\"] Once DeviceAllow is present, the service cgroup denies every device that is not named, regardless of the helper\u0026#39;s UID. Running smartctl as root did not bypass this rule because the device cgroup applies to the complete Netdata service.\nZFS collector used /usr/bin/zpool After allowing /dev/zfs, the ZFS pool collector still registered nothing. Its initialization error was precise:\n1 stat /usr/bin/zpool: no such file or directory NixOS does not install zpool under /usr/bin. Netdata\u0026rsquo;s go.d collector in this version used that fixed default instead of searching the service path.\nThe job needed an explicit binary location:\n1 2 3 jobs: - name: zfspool binary_path: /run/current-system/sw/bin/zpool Now the collector could execute zpool, and the previous /dev/zfs exception allowed the command to reach the kernel module. Fixing only the path would have changed the failure from \u0026ldquo;not found\u0026rdquo; to a timeout or permission error.\nTraefik was not ready during the first check During the same activation, containerized Traefik was restarting. Netdata checked its metrics endpoint before it was ready and reported:\n1 check failed: unexpected metrics (not Traefik) The endpoint recovered. The charts did not. A failed initial go.d check() can disable that job until Netdata itself restarts.\nThe fix was not another systemd dependency. Netdata and Traefik may legitimately restart independently. The collector needed to tolerate temporary absence:\n1 2 3 4 jobs: - name: traefik url: http://10.89.0.2:8083/metrics autodetection_retry: 60 I used the same retry behavior for NUT, whose endpoint may appear in a later configuration phase.\nsequenceDiagram participant N as Netdata go.d participant T as Traefik metrics N-\u003e\u003eT: initial check during restart T--\u003e\u003eN: not ready Note over N: Without retry, job remains disabled N-\u003e\u003eT: retry after 60 seconds T--\u003e\u003eN: valid Prometheus metrics Note over N: Collector registers charts A one-shot discovery check turns a temporary dependency outage into permanently missing charts; periodic autodetection heals it. How I checked each failure The three symptoms all looked like absent charts, but their evidence lived at different layers:\nLayer Question Evidence Collector Did the job parse and initialize? go.d debug log Executable Does its configured binary exist? stat and exact error Device cgroup Can the service open the device? EPERM despite helper UID Filesystem permissions Is the node accessible after cgroup policy? mode and group Network filter Can this cgroup reach the endpoint? request from service context Lifecycle Was the endpoint merely late? retry succeeds without config change Instead of removing all restrictions, I added only what each collector needed:\none device-node allow entry; one NixOS-native binary path; one retry interval; only the bridge ranges used for dashboard ingress and local notifications. After these changes, SMART could read /dev/sda, ZFS used the NixOS zpool path and Traefik registered itself after its endpoint became ready. The useful check here is the go.d collector log; a working Netdata service and dashboard do not mean every configured collector started successfully.\n","date":"2026-08-10T00:00:00+05:30","permalink":"https://blog.dexome.com/post/hardening-monitoring-agent-nixos/","title":"Running Netdata with systemd Hardening on NixOS"},{"content":"I run two OPNsense VMs, one on metalbox and another on heavymetal. I wanted the second firewall to take over the gateway, DNS, DHCP and existing connections when the first host was unavailable.\nCARP itself was the easy part. The difficult parts were the services around it. When I first booted the backup firewall, the network reached around 160,000 packets per second and became unusable. Later, CARP moved the IP correctly but DNS stopped and existing TCP connections were lost.\nBelow is how I configured the full setup and the issues I found while testing it.\nThe HA setup The finished design runs two OPNsense VMs on separate NixOS hypervisors. Node A runs on metalbox; node B runs on heavymetal. Clients keep using the familiar .1 gateways on each internal VLAN, now implemented as CARP VIPs. The nodes have real .6 and .7 management addresses, and a dedicated VLAN 67 sync link at 10.100.67.6/28 and 10.100.67.7/28.\nflowchart TB CLIENTS[LAN, THINGS, QUANTUM and SERVERS clients] --\u003e VIPS[CARP gateway VIPs ending in .1] INTERNET[PPPoE uplink] --\u003e WANVIP[WAN CARP VIP 192.168.1.5] subgraph M[metalbox NixOS hypervisor] A[OPNsense A - preferred MASTER] AREAL[Real addresses ending in .6] A --- AREAL end subgraph H[heavymetal NixOS hypervisor] B[OPNsense B - BACKUP] BREAL[Real addresses ending in .7] B --- BREAL end VIPS --\u003e A VIPS -. failover .-\u003e B WANVIP --\u003e A WANVIP -. failover .-\u003e B A ==\u003e|VLAN 67: XMLRPC, pfsync, Kea HA| B The completed OPNsense HA topology separates client VIPs from node management and the VLAN 67 control plane. There are four separate parts in this setup:\nCARP moves gateway and service IP addresses between nodes. Configuration sync keeps rules and services aligned. State sync copies the firewall state table so established connections live. Application-level HA makes services such as DHCP and mDNS behave correctly. I configured and tested them one at a time.\nConfigure CARP on the first node Before introducing a backup, I converted the existing firewall into a single CARP master. Its old gateway addresses became virtual IPs, while the firewall received separate real addresses for management. The WAN source-NAT rule also changed to use the WAN virtual IP, so outbound traffic would retain the same source after a failover.\nThis stage sounds redundant: why configure failover with only one node? Because it isolates the address migration from the redundancy problem. I could prove that clients still reached their familiar gateways, outbound NAT used the expected address, and management remained available before another system was allowed to advertise anything.\nIt also established a useful naming pattern:\none stable name for the firewall service, resolving to the virtual IP; one node-specific name per firewall, resolving to its real address. When HA itself is broken, the node-specific addresses are the way back in.\nBooting the backup caused an mDNS storm The second node started from a copy of the first node\u0026rsquo;s configuration. Its CARP advertisement priority was lower, so it should have stayed in BACKUP. Within roughly 30 to 60 seconds, however, the network flooded.\nThe timing made a split-brain theory persuasive. I checked the usual suspects:\nshared CARP passwords matched; virtual host IDs matched; the master and backup had the intended advertisement skew; multicast advertisements arrived on every VLAN; the backup remained silent in BACKUP instead of advertising as another master. These checks showed CARP was working as expected. I then captured the traffic causing the packet storm.\nThe breakthrough was to stop looking only at CARP packets and inspect the storm itself. Almost all of it was UDP port 5353: multicast DNS. Both node MAC addresses were flooding at similar rates.\nThe copied configuration had enabled an mDNS repeater on both firewalls. Each reflector received packets repeated by the other and reflected them again across the same interfaces. A small amount of multicast became an exponential loop.\nsequenceDiagram participant Device as mDNS device participant A as OPNsense A reflector participant B as OPNsense B reflector participant LAN as Other VLANs Device-\u003e\u003eA: Multicast query on UDP 5353 A-\u003e\u003eLAN: Reflect query LAN-\u003e\u003eB: Reflected query arrives B-\u003e\u003eLAN: Reflect it again LAN-\u003e\u003eA: Re-reflected query arrives loop Exponential amplification A-\u003e\u003eLAN: Reflect B's copy LAN-\u003e\u003eB: Deliver A's copy B-\u003e\u003eLAN: Reflect A's copy LAN-\u003e\u003eA: Deliver B's copy end OPNsense B inherited the active mDNS repeater configuration. Each node reflected the other\u0026#39;s reflected packets until the LAN reached roughly 160,000 packets per second. The firewall software had the correct feature for this situation: enable the repeater only while the node is CARP master. I configured the service identically on both nodes but enabled its CARP-aware failover mode. The backup retained the configuration without running the reflector until promotion.\nThe same check is needed for any service that broadcasts or reflects traffic. Copying the configuration to the backup can make both instances active at the same time.\nDedicated network between the firewalls I added VLAN 67 between the firewalls for control traffic. It had no client gateway and no virtual IP. OPNsense A used 10.100.67.6/28, OPNsense B used 10.100.67.7/28, and a tightly scoped firewall rule allowed traffic only within that sync subnet.\nThe lack of a default pass rule on a newly assigned firewall interface was an early trap. Both addresses existed, but all layer-3 traffic was silently dropped until the sync-network rule was installed. Link state and correct addresses do not prove that the control plane can communicate.\nThe dedicated link carried config sync, pfsync, and DHCP peer communication. It also kept that traffic away from client VLANs and gave packet captures a much cleaner place to answer \u0026ldquo;did the peers actually talk?\u0026rdquo;\nflowchart LR A[OPNsense A 10.100.67.6] ==\u003e|XMLRPC: configuration A to B| B[OPNsense B 10.100.67.7] A \u003c--\u003e|pfsync: firewall states| B A \u003c--\u003e|Kea HA: leases and peer health| B CARP[CARP advertisements on client VLANs] -.-\u003e A CARP -.-\u003e B CARP is only the address-ownership layer; three separate protocols cross VLAN 67 to preserve configuration, leases, and live connections. Problems with configuration sync The config-sync page looked straightforward: peer address, username, password, and a list of areas to synchronize. It failed for five independent reasons.\nManagement ports were different One node exposed its GUI directly on the default HTTPS port. The other listened on a non-default local port behind a reverse proxy. Config sync contacted the peer\u0026rsquo;s real address, not the public proxy name, so both web services needed a reachable and explicitly matching port.\nGUI was not listening on the sync interface The backup\u0026rsquo;s web server listened only on its LAN interface. A TCP connection over the dedicated sync address therefore had nowhere to land. Adding the sync interface to the GUI\u0026rsquo;s listen scope fixed the transport without widening client access.\nCertificate verification failed The peers used a self-signed management certificate over a private, dedicated L2 link. Certificate verification failed before credentials were considered. In this topology I disabled peer verification for the sync call. A better option, when supported, is to give each node a certificate chaining to an internal CA.\nThe sync user needed another privilege The user could open the HA configuration page but could not call the XMLRPC library. \u0026ldquo;High Availability\u0026rdquo; GUI access and XMLRPC execution were separate permissions. Granting only the narrowly required library privilege fixed the API call without turning the account into an administrator.\nChanges were not pushed automatically The largest conceptual surprise was that saving a configuration did not necessarily push it to the backup. The GUI\u0026rsquo;s explicit \u0026ldquo;synchronize all\u0026rdquo; action worked, but ordinary changes and infrastructure-as-code applies did not invoke it.\nI added a post-apply hook that calls the firewall\u0026rsquo;s sync endpoint. GUI-only edits still require the operator to trigger synchronization, so the operating rule is simple: one node is authoritative; the backup is not a second place to edit.\nThe reusable lesson is to test config sync as an action, not as a checkbox. Change a harmless object on the primary, trigger the documented sync path, and prove it appears on the backup.\nKea DHCP hot standby CARP moving the gateway does not automatically make the backup DHCP server safe. Two independent DHCP servers with copied configuration can both answer clients, while a master-only DHCP service can leave an availability gap during promotion.\nI used Kea\u0026rsquo;s hot-standby mode over the sync network. The primary serves leases under normal conditions; the standby receives lease updates and takes over after it decides the partner is down.\nThe configuration failed when both peers used the same server name. Kea requires this-server-name to match one of the declared peer names. Explicitly writing the primary\u0026rsquo;s name into synchronized configuration made both nodes identify as the primary.\nI gave the firewalls distinct hostnames, declared those as peer names, and left this-server-name empty so each node derives its identity locally. The node identity cannot be copied from primary to backup.\nHot standby also has a detection interval. In this case the standby waited about a minute before entering partner-down mode. Existing clients with leases stayed online; a new or renewing client might wait. That is not a broken failover, but it is part of the service-level objective and should be measured rather than assumed.\nUnbound stopped answering after failover With CARP working, the virtual gateway moved cleanly to the backup. Clients could route, but DNS queries to the gateway timed out.\nThe resolver had been configured to bind selected interfaces. On the backup, virtual IPs do not exist while it is in BACKUP. When the resolver configuration was regenerated, the absent virtual address disappeared from the generated listen set. CARP later promoted the node and created the address, but the resolver was not listening there.\nThe fix was to let the resolver listen on all local interfaces using its automatic interface behavior. Firewall policy, rather than a brittle application bind list, controlled which networks could reach port 53.\nThis is a broader HA pattern: applications should not permanently compile a list of addresses whose presence changes with ownership. Either bind wildcard/automatic and enforce access in the firewall, or use a service hook that reacts correctly to every promotion and demotion.\nKeeping connections with pfsync At this point a cable pull promoted the backup, moved the virtual IPs, and kept DNS and DHCP available. Yet established TCP sessions still died.\nThat was expected in retrospect. A stateful firewall tracks sequence numbers, NAT translations, timeouts, and policy decisions in a state table. The backup cannot infer that state from a newly arrived midstream packet.\nI enabled pfsync on both nodes using explicit unicast peer addresses on the sync network. Configuring only one direction was not sufficient; each node needed to send and receive state because either could be active after maintenance or a second failure.\nThe acceptance test was deliberately mundane: start a long-lived transfer, pull the active firewall\u0026rsquo;s network cable, and watch whether the transfer continues. After bidirectional state sync was working, it did.\nThat test proved something a successful ping never could.\nHow I tested the failover I ended with a matrix instead of a single \u0026ldquo;HA works\u0026rdquo; result:\nTest What it proves Primary enters maintenance mode Graceful CARP demotion and backup promotion Primary loses its cable Failure detection without operator help Fail back to the original node Prior master can safely resume ownership DNS query during promotion Resolver listens correctly on the new owner New DHCP request after peer timeout Standby has leases and enters partner-down Long TCP transfer during cable pull State and NAT translations were synchronized Config change on primary Explicit sync path updates the backup Boot both nodes from cold Broadcast services do not duplicate or loop Running the matrix in both directions exposed assumptions hidden by testing only the preferred primary.\nFinal checks Moving the virtual IP was only one of the checks. The setup was complete when:\nthere is exactly one active owner for each singleton or broadcast service; both nodes carry compatible configuration without becoming competing writers; DHCP and DNS continue according to measured failover timing; established stateful connections survive a hard loss; operators can reach either node through a real, node-specific address; the behavior has been proven by removing the active node, not inferred from a green status page. The main issue I found was assuming that copying a service configuration also made the service HA-aware. The mDNS repeater had to run only on the CARP master, Kea needed its own peer state and Unbound needed to listen on an address which appears only after promotion.\nFinally, test with a real cable pull and a long-running connection. A successful ping after pressing the CARP maintenance button does not verify pfsync or the hard-failure path.\n","date":"2026-08-04T00:00:00+05:30","permalink":"https://blog.dexome.com/post/ha-firewall-carp-state-sync/","title":"Setting up OPNsense HA with CARP and pfsync"},{"content":"I wanted to move the WAN connection from a dedicated port on OPNsense to VLAN 999 through my Omada switch. This would allow both OPNsense nodes to reach the same ONT connection and was needed for the HA setup.\nThe PPPoE username, password and ONT were not changing. I had already configured the VLAN and expected the Internet to be down only for the time needed to move the cable and connect again.\nIt did not work that way. The first connection failed because I had left the old PPPoE session active at the ISP. Once I fixed that, the connection dropped again because the Omada switch was sending loop-detection packets towards the ONT.\nThis post covers both issues and the order I now use for a PPPoE cutover.\nThe setup The old topology was direct. The Syrotech GPON ONT connected to the bare-metal OPNsense firewall\u0026rsquo;s igc0 interface. The new design moved that Ethernet segment onto VLAN 999 through a TP-Link Omada TL-SG2008P, then carried the tag over a trunk to OPNsense as igc2_vlan999.\nflowchart TB BEFORE[\"BeforeSyrotech ONT to OPNsense igc0 to PPPoE\"] AFTER[\"AfterSyrotech ONT to Omada TL-SG2008P to VLAN 999 to OPNsense igc2_vlan999 to PPPoE\"] BEFORE --\u003e|Move WAN onto the managed-switch trunk| AFTER The WAN migration inserted an Omada switch and VLAN 999 between the Syrotech ONT and OPNsense. The VLAN setup looked correct. A test device on the same path could reach the management IP of the ONT and packet captures showed the expected VLAN traffic. So I pulled the cable from the old port, moved it and enabled the new interface.\nThe connection never completed.\nPacket captures showed PADI discovery frames leaving the firewall, but the expected PADO response did not come back. At other points the logs showed LCP negotiation repeating with new magic numbers, never reaching a usable session. It looked close enough to working to keep sending me down the wrong paths.\nChecking the VLAN first VLANs are an obvious suspect because they can fail silently. An access port can be untagged in the wrong VLAN, a trunk can omit the tag, or a native VLAN can consume traffic that was supposed to remain tagged.\nI tested the transport independently of PPPoE. A client attached to the same logical path could reach the optical terminal\u0026rsquo;s management network. Captures on the firewall showed the expected Ethernet discovery frames on the expected VLAN. That did not prove every switch behavior was correct, but it proved that this was not simply a missing tag.\nChecking the WAN MAC Many providers bind service to a router or ONT MAC address, so MAC locking was the next theory. I tried preserving the previous WAN MAC and inspected the provider-facing frames. It was a reasonable theory, but it did not explain the behavior.\nSome ISPs lock the connection to a MAC address, so this was still worth checking. There are three separate things which can cause a new router or port to fail:\nDoes the provider authenticate only with a username and password? Does it also remember the client MAC? Does it permit more than one simultaneous PPPoE session on the line? All three look similar from OPNsense. In my case, the third one was the problem.\nClosing the old PPPoE session I had moved the cable while PPPoE was still active. That is not a clean session shutdown.\nA graceful PPPoE disconnect sends a PADT (PPPoE Active Discovery Terminate), or terminates the link through PPP before removing the carrier. A cable pull sends neither. From the ISP access concentrator\u0026rsquo;s point of view, the old session can remain alive until its own timeout expires.\nThe provider allowed one session on the line. My new firewall interface was not replacing the old session; it was asking for a second one while the first still existed upstream.\nThat explained why waiting sometimes appeared to fix the problem. A long power-off interval gave the stale session time to age out. It also explained why rechecking credentials and VLAN tags accomplished nothing. The obstacle was not in my current configuration at all.\nThe reliable cutover was:\nLeave the old WAN connected. Disconnect or disable PPPoE on the old interface. Verify that a PADT leaves the old interface. Move the cable or change the VLAN assignment. Enable PPPoE on the new interface. sequenceDiagram participant Old as OPNsense igc0 participant BRAS as ISP access concentrator participant New as OPNsense igc2_vlan999 Old-\u003e\u003eBRAS: PPPoE session established Note over Old,BRAS: Cable pulled without LCP Terminate or PADT New-\u003e\u003eBRAS: PADI from VLAN 999 BRAS--xNew: One-session policy blocks new session Note over BRAS: Old session remains until timeout Old-\u003e\u003eBRAS: PADT during controlled retry BRAS--\u003e\u003eOld: Old session removed New-\u003e\u003eBRAS: PADI then PADR BRAS--\u003e\u003eNew: PADO then PADS Note over New,BRAS: New PPPoE session succeeds The cable pull left the old session alive at the ISP; a PADT-first cutover explicitly closed it before the new interface dialled. On a BSD-based firewall, a capture like this is enough to observe the discovery traffic:\n1 tcpdump -ni \u0026lt;wan-interface\u0026gt; -e ether proto 0x8863 The exact command used to disconnect varies by platform. I prefer the firewall\u0026rsquo;s normal interface disconnect action because it follows the same control path the system uses in production. The capture is there to prove the shutdown emitted a termination frame before any cable moved.\nWith the old session closed first, the new interface authenticated normally.\nThe connection came up on VLAN 999 after this. But it did not stay up.\nOmada loop detection on the ONT port The PPPoE session later failed again, this time after it had already been stable. The pattern was different: discovery could be blocked outright, or an established session would disappear several minutes later.\nThe managed switch between the firewall and optical terminal had loop-detection features enabled. It injected control traffic into the provider-facing bridged segment: first spanning-tree-style frames, then vendor loopback-detection probes. On an ordinary LAN those features can be useful. On a transparent WAN transport, the ISP side is not an ordinary LAN and did not appreciate the extra frames.\nThe fix was not to disable protection across the whole switch. I disabled loopback control only on the port connected to the optical terminal. The WAN VLAN remained isolated and tagged exactly as before; only the switch\u0026rsquo;s active probing stopped.\nflowchart TD MOVE[Move ONT cable from igc0 to VLAN 999] --\u003e IMMEDIATE{When does it fail?} IMMEDIATE --\u003e|Immediately| STALE[Old PPPoE session still held by ISP] STALE --\u003e PADT[Disconnect old WAN and verify PADT] PADT --\u003e UP[PPPoE connects on igc2_vlan999] IMMEDIATE --\u003e|Minutes after connecting| PROBES[Omada loopback-control probes enter WAN segment] PROBES --\u003e DROP[Established PPPoE session drops] DROP --\u003e PORT[Disable Loopback Control on ONT-facing port] PORT --\u003e STABLE[Session remains stable through soak and reboot] Two independent faults occupied the same WAN path and failed on different timelines. After that per-port change, the PPPoE session stayed up through the observation window and survived reboots of the relevant equipment.\nInitially I thought the earlier fix was incomplete, but this was a separate problem on the same path:\nthe cutover failed immediately because the old PPPoE session was never closed; the repaired connection later failed because the switch injected control traffic into the WAN bridge. The migration steps I use now Below is the order I now use when moving a live PPPoE WAN between ports, VLANs or firewall nodes.\nBefore the maintenance window Record the working parent interface, VLAN ID, MTU, credentials source, and WAN MAC. Confirm whether the provider documents MAC binding or single-session limits. Preconfigure the destination interface without enabling it. Verify the VLAN at layer 2 independently of PPPoE when possible. Review the ISP-facing switch port for spanning tree, loop detection, LLDP, discovery, or other active control protocols. Keep a management path that does not depend on the WAN being migrated. During the cutover 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 old PPPoE connected │ ▼ disconnect old PPPoE cleanly │ ▼ observe PADT / termination │ ▼ move cable or VLAN ownership │ ▼ enable new PPPoE interface │ ▼ verify discovery, LCP, IPCP, route and DNS Do not clone the MAC reflexively. Preserve it only when you know the provider requires it or when a controlled test demonstrates that it matters. Unnecessary MAC duplication is especially hazardous if old and new devices can be online at the same time.\nAfter the session comes up Do more than check that a public IP appeared:\ninspect the PPP logs for successful LCP and IPCP negotiation; confirm the default route uses the new interface; test DNS through the normal client path; verify the negotiated MTU with don\u0026rsquo;t-fragment pings; keep the session under observation long enough to catch periodic switch probes; reboot the switch or firewall if reboot survival is part of the acceptance criteria. The distinction between \u0026ldquo;connected once\u0026rdquo; and \u0026ldquo;operationally complete\u0026rdquo; saved me from calling the migration finished before the loop-detection failure surfaced.\nKey things to check The immediate failure and the delayed failure had different causes. If PPPoE does not connect after moving a cable, first check whether the old session was closed properly. If it connects and drops later, check what the switch is sending on the WAN VLAN.\nAlso, a managed switch is not always transparent. Disable STP, loop detection or similar active features on the ONT-facing port unless you know the ISP path can handle them.\nMost importantly, disconnect PPPoE before moving the cable and verify the PADT in a packet capture. Waiting for the ISP timeout also worked, but it made a simple cutover take much longer than needed.\n","date":"2026-07-30T00:00:00+05:30","permalink":"https://blog.dexome.com/post/migrate-live-isp-wan-without-lockout/","title":"Moving My PPPoE WAN to VLAN 999"},{"content":"I wanted to manage my existing OPNsense configuration using OpenTofu. The firewall was already running DNS, DHCP, several VLANs, VPN connections and all the rules for my home network. Recreating everything from code was not an option.\nI used the browningluke/opnsense provider and started with one Unbound DNS setting. After importing it, I did not continue until the plan showed:\n1 No changes. Your infrastructure matches the configuration. This worked for DNS, but the next step caused a DHCP outage because one provider default removed the gateway and DNS options from client leases. This post covers the order I used after that incident and the provider limitations I found.\nImport first and make no changes There are two separate things we may want to do during this migration:\nrepresent the current system in code; clean up the system while doing it. I would not combine them.\nThe first goal is adoption. Its success criterion is boring: after import, the configuration describes the live object exactly enough that a plan proposes no change. Only after that baseline is stable should a separate change improve the object.\nThis matters most for routers and firewalls because the management path is one of the resources being changed. An incorrect web-server deployment can return a 500. An incorrect gateway, DHCP option, or anti-lockout rule can remove the path you need to repair it.\nI used four gates for every subsystem:\n1 2 3 inventory ──\u0026gt; import ──\u0026gt; zero-diff plan ──\u0026gt; one-object apply │ │ │ │ └── stop ─────┴── stop ─────┴── stop on drift ─┘ The first real apply was always deliberately small.\nflowchart TB U[Unbound DNS: 66 objects] --\u003e|zero-diff plan| K[Kea DHCP: 64 objects] K --\u003e|client lease test| O[Omada VLANs, profiles, ports and SSIDs] O --\u003e|controller no-op| C[Guest firewall canary] C --\u003e|compiled pf order| F[Per-interface filter migration] F --\u003e N[NAT and VPN resources] The adoption moved outward from lower-risk DNS objects to connectivity-critical filters and NAT, with a stop gate after every phase. Start with Unbound DNS Resolver settings and host overrides were a good first target. Sixty-six Unbound objects were numerous enough to test import automation but less dangerous than rewriting the firewall ruleset.\nThe import revealed an important category of provider behavior: fields that exist on the appliance but not in the provider schema. One host override generated a reverse record, yet the provider did not expose that switch. Importing and planning the resource produced no change, so the appliance-only field survived.\nThat was acceptable. IaC coverage does not need to be 100 percent to be useful. It does need to be honest.\nAnother DNS feature exposed the opposite problem: the provider could read a blocklist setting but failed when writing it. Rather than force ownership, I left that feature GUI-managed and documented the boundary. A provider that cannot round-trip a field does not own that field.\nThe first phase ended with dozens of objects imported and a no-op plan. The point was not the count. It was proving the API credentials, import identifiers, schema, and state storage before touching client connectivity.\nKea DHCP and the auto_collect issue Kea DHCP import covered 64 objects and looked equally clean until the first apply. Clients on THINGS and QUANTUM renewed and still received valid addresses, but they lost their default gateway and DNS server.\nThe provider exposed an auto_collect option. Its default was enabled, suggesting that the appliance would derive subnet options automatically. On this system it did not. Applying the resource removed the stored router, DNS, and NTP values.\nA simplified version of the dangerous assumption looked like this:\n1 2 3 4 resource \u0026#34;firewall_dhcp_subnet\u0026#34; \u0026#34;clients\u0026#34; { subnet = \u0026#34;10.20.0.0/24\u0026#34; auto_collect = true } The repaired declaration made every client-visible option explicit. This is a simplified version of the QUANTUM subnet, whose gateway and resolver are 10.100.30.1:\n1 2 3 4 5 6 7 8 resource \u0026#34;firewall_dhcp_subnet\u0026#34; \u0026#34;clients\u0026#34; { subnet = \u0026#34;10.100.30.0/24\u0026#34; auto_collect = false routers = [\u0026#34;10.100.30.1\u0026#34;] dns_servers = [\u0026#34;10.100.30.1\u0026#34;] ntp_servers = [\u0026#34;10.100.30.1\u0026#34;] } flowchart LR CLIENT[QUANTUM client] --\u003e|DHCP Discover| KEA[OPNsense Kea] KEA --\u003e|Offer: address only| CLIENT CLIENT --\u003e IP[Client has a 10.100.30.x address] CLIENT -. missing .-\u003e GW[Default gateway 10.100.30.1] CLIENT -. missing .-\u003e DNS[DNS server 10.100.30.1] IP --\u003e SYMPTOM[Looks connected but cannot route or resolve] The DHCP daemon stayed healthy while `auto_collect` removed the information clients needed to use their leases. The important thing here is that the provider default did not match the existing OPNsense behavior. I now set every client-visible DHCP option explicitly.\nAfter restoring the option data, I verified DHCP as a client would:\nobtain a new lease; inspect the offered router and DNS options; reach the gateway; resolve a name; cross the firewall to an external address. \u0026ldquo;The service is running\u0026rdquo; would not have caught this failure. DHCP was running perfectly while handing out incomplete leases.\nImporting Omada configuration The managed-switch controller added another translation layer. The API endpoint behind the normal reverse-proxy address redirected login requests, while the provider expected to talk directly to the controller. Connecting to the direct management origin fixed authentication.\nImports then showed several values whose controller defaults differed from the provider defaults: multicast snooping, relay booleans, and profile flags. To reach a zero-diff plan, I had to write values that the GUI had previously left implicit.\nWireless credentials were particularly important. The controller returned a non-null pre-shared key. Omitting the field in code did not mean \u0026ldquo;leave it alone\u0026rdquo;; it meant \u0026ldquo;clear it.\u0026rdquo; The secret therefore had to be supplied at runtime from an encrypted source so the plan could preserve the live network without committing the key.\nHardware controls deserve the same caution. On this controller, Power over Ethernet belonged to a port profile. Applying a profile with PoE disabled to a live access point would cut power to the device carrying the management traffic. I treated profile changes as physical operations, not harmless metadata edits.\nFirewall rules and their real order Firewall filters were the highest-risk phase because the appliance had two rule stores:\nlegacy rules created in the traditional per-interface GUI; automation rules created through the API and managed by OpenTofu. The provider could not import legacy rules because they were not the same kind of object. They had to be recreated in the automation store.\nThat raised a more important question than whether the declarations looked equivalent: where would the new rules land in the effective packet-filter order?\nFirewall evaluation is ordered. Two identical sets of rules can behave differently if a broad pass or block moves above a specific exception. The GUI\u0026rsquo;s visual order was not enough because it separated the two stores.\nI found an API endpoint that returned the compiled packet-filter rules in actual evaluation order, including labels that distinguished automation objects from legacy objects. I wrapped it in a small read-only script and made its output a mandatory gate for every interface migration.\nflowchart TB TF[OpenTofu resources] --\u003e AUTO[os-firewall Automation store] GUI[Existing GUI rules] --\u003e LEGACY[Legacy interface store] AUTO --\u003e COMPILE[OPNsense rule compiler] LEGACY --\u003e COMPILE SYSTEM[Anti-lockout and generated rules] --\u003e COMPILE COMPILE --\u003e PF[Effective pf rules in @N order] PF --\u003e CHECK[pf-rule-order.sh verification] CHECK --\u003e|Automation safely shadows legacy| REMOVE[Remove legacy twin] CHECK --\u003e|Unexpected order| STOP[Stop and repair sequence] OPNsense displayed legacy and Automation rules separately, so I queried the compiled pf order before removing any legacy rule. The sequence per interface became:\nRecreate a small set of legacy rules as automation resources. Apply them while the legacy originals remain enabled. Query the compiled ruleset. Confirm the automation rules sit in the intended order and shadow the legacy copies safely. Test traffic through that interface. Disable, then remove, the legacy copies. Plan again and confirm no unexpected drift. I started with a low-risk guest network containing only three rules. It was a canary for the ordering model. Only after its compiled order and behavior were correct did I migrate management, server, VPN, and WAN interfaces one at a time.\nExplicit sequence values were essential. Relying on every resource\u0026rsquo;s default sequence created ties and non-deterministic placement. I reserved sequence ranges per interface so both humans and the provider had one stable ordering model.\nDisabled rules can still block deletion One migration exposed another appliance quirk. A disabled legacy rule still referenced an alias, and that reference prevented OpenTofu from deleting the alias. From an operator\u0026rsquo;s perspective the rule was inactive. From the appliance\u0026rsquo;s validation perspective it still existed.\nThe fix was to remove the obsolete legacy rule, not merely disable it.\nThis is why I avoided bulk cleanup during adoption. Relationships that do not affect packet evaluation can still affect schema validation and deletion order.\nMigrating NAT separately Filter rules and NAT rules may appear together in the GUI, but they are not the same ownership boundary. Some legacy firewall rules carried an association to a generated NAT rule that the provider could not preserve. I migrated NAT in a later phase after filter behavior was stable.\nThe NAT provider also had schema gaps: some labels were unavailable, some port fields rejected aliases, and protocol values normalized differently from the appliance. These limitations did not invalidate the whole migration. They defined which details stayed appliance-managed and which needed a different expression.\nI left settings in the GUI when the provider could not safely read and write them. It is better to document that boundary than force an incomplete resource to own it.\nSecrets and state An API-driven firewall migration touches credentials in several places:\nfirewall API keys; VPN static keys and certificates; wireless pre-shared keys; remote-state access credentials. I kept secrets encrypted outside the HCL and injected them into provider or resource variables only for the command that needed them. That keeps plaintext out of source files, but it does not automatically keep secrets out of state. Provider schemas may still serialize sensitive values into the state backend.\nThe state backend therefore needs the same protection as the firewall backup: access control, encryption, and a tested recovery procedure. If the backend has no locking, only one writer can safely apply at a time.\nImport declarations are worth retaining as disaster-recovery documentation. A state loss otherwise also loses the mapping between stable resource names and opaque appliance UUIDs.\nChecks used for each resource type For every new resource family, I now ask:\nBefore import Does the provider read and write the same API representation? Which live fields are absent from the schema? Which provider defaults differ from appliance defaults? Can this resource interrupt the management path, power, DHCP, DNS, or WAN? Is there a read-only way to inspect the compiled/effective result? Before the first apply Is the plan a no-op after import? Are secret values present at runtime but absent from source? Is the first apply limited to one object or one low-risk segment? Is there an independent management path and a rollback artifact? After apply Did a real client receive the expected service, not merely a green status? Does the compiled firewall order match the intended order? Did the appliance preserve fields the provider does not expose? Does a second plan return to no changes? Final setup Not every OPNsense setting is managed by OpenTofu. Some remain in the GUI because the provider cannot represent or write them safely. What I have now is a clear list of which tool owns each resource and a repeatable process:\nimport live state; insist on zero drift; change one boundary at a time; inspect the effective system, not just the tool\u0026rsquo;s model; preserve a way back in. The main rule is to get a no-change plan after import and then apply one small change. Also verify from a real client. In the DHCP incident, the daemon was healthy and the apply succeeded, but clients received leases without a gateway or DNS server.\n","date":"2026-07-07T00:00:00+05:30","permalink":"https://blog.dexome.com/post/firewall-as-code/","title":"Managing My Existing OPNsense Setup with OpenTofu"},{"content":"I moved a few home automation workflows from n8n to Kestra. One of them listens for a Frigate MQTT event, applies a two-minute cooldown and sends a camera image through Apprise.\nThe flow YAML was not the difficult part. I had issues with the non-root container configuration, OSS authentication, the MQTT trigger type, KV expiry and old realtime triggers which continued running after the flow was disabled.\nThis post collects those fixes for Kestra 1.3.24 and 1.3.33. Some of these are version-specific, so validate them again when using a newer release.\nRunning the Kestra container as a non-root user My Kestra 1.3.24 container ran as a dedicated host UID instead of the image\u0026rsquo;s baked kestra user. The image entrypoint tried to materialize the KESTRA_CONFIGURATION environment variable at /app/confs/application.yml. That image directory was not writable by the chosen UID, so the container crash-looped before Micronaut started.\nThe fix was to render configuration declaratively, mount it read-only, and point Micronaut directly at it:\n1 2 3 host-rendered application.yml -\u0026gt; /app/confs/application.yml:ro MICRONAUT_CONFIG_FILES=/app/confs/application.yml KESTRA_CONFIGURATION unset Secret placeholders stayed in the rendered file and resolved from the runtime environment. The entrypoint\u0026rsquo;s write branch was no longer involved.\nflowchart TB N[Nix renders non-secret config] --\u003e F[Store file with env placeholders] S[SOPS runtime environment] --\u003e M[Micronaut] F --\u003e|read-only bind mount| M E[Image entrypoint config writer] -. skipped .-\u003e M A read-only bind mount bypassed the image entrypoint\u0026#39;s assumption that its baked config directory was writable. When changing the container user, check files written by the entrypoint before the application starts. The final Kestra process user alone did not explain this failure.\nPocket ID did not replace Kestra basic authentication Traefik already protected the Kestra route with Pocket ID forward auth. I expected that gate to be sufficient. Kestra OSS still presented a \u0026ldquo;create admin user\u0026rdquo; page.\nThese were independent layers:\nPocket ID decided who could reach Kestra through Traefik. Kestra OSS basic auth decided who could use its UI and API. I tried kestra.server.basic-auth.enabled: false. That property did not exist, and Kestra ignored the unknown key. Enabling Micronaut security removed the basic-auth bean but broke an OSS UI endpoint that requires the bean. In this version, an auth-less OSS UI was not a supported state.\nThe working design retained basic auth, provisioned the admin from runtime secrets, and had a Traefik response middleware set the same BASIC_AUTH cookie Kestra\u0026rsquo;s login form would set. The middleware runs after the Pocket ID gate, so users still encounter one interactive login.\nAn injected Authorization request header did not work because the SPA checked document.cookie before routing. Browser behavior, not just server behavior, was part of the auth contract.\nValidating flows with the pinned Kestra image Flow schemas and plugin properties move. I validate repository flows with the CLI from the pinned Kestra image before synchronizing them. The image\u0026rsquo;s /app/kestra launcher also had no directly usable shebang, so raw podman exec kestra /app/kestra ... returned an exec-format error. Invoking docker-entrypoint.sh flow validate ... reproduced the image\u0026rsquo;s normal shell fallback and worked.\nThis catches invalid properties and expressions, but it does not test how a long-running trigger behaves.\nUse RealtimeTrigger for MQTT events The first flow used io.kestra.plugin.mqtt.Trigger. Its name looked correct, but it polls: connect on an interval, collect messages, disconnect. Frigate\u0026rsquo;s zone events are transient and non-retained, so the flow was usually absent when the message arrived.\nio.kestra.plugin.mqtt.RealtimeTrigger holds a subscription and creates one execution per message. It also exposes the body at trigger.payload; the polling trigger has different outputs.\n1 2 3 4 5 6 triggers: - id: maindoor_person type: io.kestra.plugin.mqtt.RealtimeTrigger server: \u0026#34;{{ envs.mqtt_broker }}\u0026#34; topic: frigate/maindoor/person serdeType: STRING I put the zero-value filter in a task-level If. In Kestra 1.3.33, placing an expression condition on the realtime trigger destabilized its subscription and caused the liveness coordinator to restart it repeatedly.\nsequenceDiagram participant F as Frigate participant P as Polling trigger participant R as RealtimeTrigger P-\u003e\u003eP: disconnected between polls F--\u003e\u003eR: payload 1 Note over P: Event is missed R-\u003e\u003eR: create execution R-\u003e\u003eR: task-level If reads trigger.payload A polling subscriber samples the topic and misses edges; a realtime subscriber remains connected and filters inside the execution. Optional output fields need a default The cooldown began with a KV Get configured with errorOnMissing: false. I then checked:\n1 {{ outputs.cooldown_get.value is null }} When the key did not exist, Kestra omitted value from the output object. Pebble did not turn the absent attribute into null; it threw an IllegalVariableEvaluationException while deciding the next task.\nOptional outputs need an explicit default:\n1 {{ (outputs.cooldown_get.value ?? null) is null }} That fixed the exception and exposed the more serious cooldown problem.\nKV expiry did not work as a cooldown The flow set a KV key with ttl: PT2M and sent another alert only when the key was absent. After one notification, about 40 later events reached the check and were suppressed. The key remained readable for hours.\nEven reliable expiry would have been awkward in this version: reading an expired value could delete it and throw ResourceExpiredException, which errorOnMissing: false did not handle.\nI stopped making correctness depend on deletion. The key now stores the last alert epoch without a TTL:\n1 2 value: \u0026#34;{{ now() | timestamp }}\u0026#34; kvType: NUMBER The condition compares values:\n1 {{ (now() | timestamp) - (outputs.cooldown_get.value ?? 0) \u0026gt;= 120 }} I used a new key name because the old key contained an ISO string. The fresh key self-seeded on the first successful execution and avoided a migration-time type error.\nflowchart TB EVENT[MQTT event] --\u003e GET[Read last alert epoch] GET --\u003e AGE{now minus last is at least 120s?} AGE --\u003e|no| SKIP[Finish without notification] AGE --\u003e|yes or absent| SEND[Send through Apprise] SEND --\u003e SET[Overwrite last alert epoch] Cooldown correctness moved from an unreliable expiration side effect to explicit timestamp arithmetic. globals key was changed on the worker I moved the MQTT broker address into kestra.variables.globals.mqttBroker and referenced {{ globals.mqttBroker }}. Every realtime trigger began failing before task execution with empty trigger variables.\nMicronaut normalized map keys while reconstructing globals on the worker: mqttBroker became kebab-case. The template requested a key that no longer existed. Password secrets still worked because Kestra injected them through a different path, which made the failure look like a broker problem.\nThe fix was an environment variable:\n1 2 container: ENV_MQTT_BROKER=\u0026lt;single-sourced broker endpoint\u0026gt; flow: {{ envs.mqtt_broker }} For values used by a worker or trigger, I now test the template in that execution context. A successful UI preview or controller-side render did not prove that the worker received the same key.\nDisabled realtime flows continued to run The strangest incident arrived while replacing one realtime flow with another on Kestra 1.3.33. The old flow was marked disabled: true. It continued to receive MQTT messages and create executions.\nRemoving its YAML did not help because my namespace update was upsert-only and did not prune the running flow. Deleting the flow and its triggers row still did not finish the job. After a restart, the JDBC liveness coordinator recreated the subscription from worker_job_running.\nThe full runtime state spanned four places:\nflowchart TB FLOW[Flow definition] --\u003e TRIG[triggers row] TRIG --\u003e RUN[worker_job_running subscription] RUN --\u003e MQTT[Live MQTT subscription] MQTT --\u003e QUEUE[queues backlog] COORD[JDBC liveness coordinator] --\u003e|restarts persisted job| RUN A realtime flow exists as definition, trigger registration, queued work, and a persisted running-worker record. Retirement required deleting the flow, its trigger registration, matching queue backlog, and the worker_job_running record, then restarting Kestra. Verification meant checking that the old rows stayed absent while a retained flow on the same topic still received the next event.\nChecks I now use For a new workflow, I check these separately:\nDeployment: can the image start under its real UID and mounted config? Schema: does the exact pinned engine accept the flow? Rendering: where is each expression evaluated, and what variables exist there? State: what persists in KV, queues, registrations, and worker tables? Lifecycle: what does disable, delete, retry, restart, and reconcile actually do? The main issue was assuming that the flow YAML was the complete runtime state. Realtime subscriptions, queued executions and running worker records were stored in PostgreSQL and survived flow changes and Kestra restarts. When retiring a realtime flow, check those records and verify using a real MQTT message after the restart.\n","date":"2026-06-26T00:00:00+05:30","permalink":"https://blog.dexome.com/post/running-workflow-orchestration-production/","title":"Running Kestra for My Home Automation Workflows"},{"content":"My NixOS server heavymetal already had multiple VLANs, static IP addresses and macvlans used by containers. They were created using startup scripts and worked, but restarts and NixOS activations did not always recreate them in the correct order.\nI first moved the same setup to systemd-networkd. Later, when I needed to attach VM tap interfaces to the VLANs, I added per-VLAN bridges and finally replaced them with one VLAN-aware bridge.\nThe migration happened in three steps:\nreplace imperative networking with networkd units; move host addresses from VLAN devices onto stable per-VLAN bridges; consolidate those bridges into a single VLAN-aware trunk. I did not apply the bridge migrations live because they moved the same management IP used by SSH. Those changes were staged for boot with console access available.\nflowchart TB S[\"Scripted VLAN andmacvlan devices\"] --\u003e N[\"systemd-networkd ownsthe same topology\"] N --\u003e|Parent restart exposes child lifecycle| P[\"Per-VLAN bridgesbrvlan20 / brvlan30 / brvlan100\"] P --\u003e|Stable parents enable VM taps| T[\"VLAN-aware bridgebrtrunk\"] The network evolved in three controlled steps rather than jumping directly from scripts to the final VLAN-aware bridge. The old startup scripts The old system created VLAN and macvlan devices through a mixture of higher-level configuration and startup services. Conceptually it did this:\n1 2 3 4 5 6 ip link add link enp5s0 name enp5s0.100 type vlan id 100 ip address add 10.100.100.20/24 dev enp5s0.100 ip link set enp5s0.100 up ip link add link enp5s0.100 name tfkshim type macvlan mode bridge ip link set tfkshim up Imperative commands are easy to prototype, but they hide ownership questions:\nWhich service owns the VLAN device? What should restart when the parent disappears? Is the address ready before a dependent service binds it? What removes stale devices after a failed partial run? Does a config reload destroy children created by another service? The scripts encoded answers in execution order rather than in the topology.\nRecreate the same network with networkd I moved each concern into an explicit networkd object: VLAN .netdev units, matching .network units for addresses and routes, and dependencies for services that needed those links.\nOn NixOS, a simplified VLAN looks like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 systemd.network.netdevs.\u0026#34;10-vlan20\u0026#34; = { netdevConfig = { Name = \u0026#34;vlan20\u0026#34;; Kind = \u0026#34;vlan\u0026#34;; }; vlanConfig.Id = 20; }; systemd.network.networks.\u0026#34;40-vlan20\u0026#34; = { matchConfig.Name = \u0026#34;vlan20\u0026#34;; address = [ \u0026#34;10.20.0.10/24\u0026#34; ]; routes = [ { Gateway = \u0026#34;10.20.0.1\u0026#34;; } ]; networkConfig.RequiredForOnline = \u0026#34;routable\u0026#34;; }; The exact syntax is distribution-specific; the systemd concepts are not. A .netdev creates a virtual device. A .network matches a device and assigns addresses, routes, VLAN membership, bridge membership, and online-state requirements.\nThe important part was preserving topology exactly. This was not the moment to rename every interface or collapse five networks into one bridge. First I needed networkd to reproduce the working system.\nsystemd-resolved broke container DNS Enabling networkd on NixOS also enabled systemd-resolved through a distribution default. The host continued to resolve names, so the change initially looked healthy. But /etc/resolv.conf now pointed at the loopback stub:\n1 nameserver 127.0.0.53 Containers copied that file into their own network namespaces. From inside a container, 127.0.0.53 meant the container itself, not the host\u0026rsquo;s resolved service. External name resolution failed even though host DNS worked.\nI explicitly disabled resolved and retained a static resolver address reachable from both the host and containers. Running resolved would also have been valid if the container DNS path had been designed for it. The failure came from changing resolver architecture as an accidental side effect of changing interface management.\nThe check I added was simple:\n1 2 3 cat /etc/resolv.conf podman exec \u0026lt;container\u0026gt; cat /etc/resolv.conf podman exec \u0026lt;container\u0026gt; getent hosts example.com A host-only lookup is not enough after a network-manager migration.\nRestarting a VLAN removed its macvlan children An older deployment had already revealed a more disruptive lifecycle problem. When the service owning a VLAN netdev restarted, it deleted and recreated the parent device. Linux also deleted every macvlan child attached to that parent.\nThe container runtime\u0026rsquo;s database still believed the container was attached. The actual network namespace contained only loopback and an unrelated bridge. The reverse proxy logged that its interface had been removed and lost its virtual address.\nThe journal made the sequence visible:\n1 2 3 4 VLAN netdev service stopped parent link deleted macvlan child removed by kernel container remains running without expected interface sequenceDiagram participant Apply as NixOS activation participant Parent as enp5s0.100 netdev participant Kernel as Linux kernel participant Child as tfkshim macvlan participant Podman as Traefik container Apply-\u003e\u003eParent: Stop and delete VLAN netdev Parent-\u003e\u003eKernel: ip link del enp5s0.100 Kernel--xChild: Delete every macvlan child Child--xPodman: eth0 removed from namespace Note over Podman: Container keeps running and DB still says attached Apply-\u003e\u003eParent: Recreate VLAN netdev Note over Parent,Podman: Parent returns but child does not Deleting and recreating the VLAN parent also deleted the live macvlan child, while Podman\u0026#39;s stored attachment remained stale. At first I coupled the container-network service lifecycle to the parent netdev service: if the parent restarted, the container networks and their consumers restarted and reattached. That repaired the immediate inconsistency.\nIt also made the weakness of the topology clear. Long-lived workloads were attached directly to a device that configuration reconciliation was allowed to destroy.\nAdding one bridge per VLAN I needed the same VLANs to be shared with ThingsHQ microVM tap devices. A macvlan parent cannot serve that role cleanly, so I created always-on host bridges such as brvlan20 and brvlan100.\nThe VLAN uplink became an addressless bridge port. The host address moved to the bridge. Containers and VMs attached to the bridge rather than directly to the VLAN netdev.\nA simplified networkd definition looks like:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 systemd.network.netdevs.\u0026#34;10-br-vlan20\u0026#34; = { netdevConfig = { Name = \u0026#34;br-vlan20\u0026#34;; Kind = \u0026#34;bridge\u0026#34;; }; }; systemd.network.networks.\u0026#34;40-vlan20\u0026#34; = { matchConfig.Name = \u0026#34;vlan20\u0026#34;; networkConfig.Bridge = \u0026#34;br-vlan20\u0026#34;; linkConfig.RequiredForOnline = \u0026#34;enslaved\u0026#34;; }; systemd.network.networks.\u0026#34;40-br-vlan20\u0026#34; = { matchConfig.Name = \u0026#34;br-vlan20\u0026#34;; address = [ \u0026#34;10.20.0.10/24\u0026#34; ]; routes = [ { Gateway = \u0026#34;10.20.0.1\u0026#34;; } ]; networkConfig.RequiredForOnline = \u0026#34;routable\u0026#34;; }; This transition could not be safely applied live. A macvlan cannot be re-parented in place. Networkd could not enslave the VLAN uplink to the new bridge while the live macvlan child still depended on it, and deleting the child would remove the remote management path carried by the reverse proxy.\nInstead of a live configuration switch, I staged the new boot generation and rebooted through a console-backed maintenance path. At boot, every device was created in the new topology from an empty state. The previous generation remained selectable in the boot menu if the new management address did not come up.\nThat was not excessive caution. \u0026ldquo;Same IP, different owning device\u0026rdquo; is still a remote access migration.\nMoving to one VLAN-aware bridge Per-VLAN bridges solved the VM-sharing problem, but the host eventually needed a larger trunk topology. Five bridges and five VLAN uplinks repeated the same structure and made VM trunk attachment awkward.\nThe next design used a single bridge named brtrunk with VLAN filtering.\nflowchart LR SWITCH[Omada trunk] ==\u003e|tagged VLANs| NIC[enp5s0] NIC --\u003e BR[brtrunk VLAN-aware bridge] BR --\u003e ADMIN[brtrunk.100: host admin 10.100.100.20] BR --\u003e THINGS[brtrunk.20: Things services] BR --\u003e QUANTUM[brtrunk.30: VPN services] BR --\u003e VMS[OPNsense and ThingsHQ VM tap ports] BR --\u003e ISOLATED[VLAN 67 sync and VLAN 999 WAN transport: no host L3] In the final design, enp5s0 is a bridge port; brtrunk owns VLAN filtering, while host L3 exists only on selected VLAN interfaces. In networkd terms, the bridge enables VLAN filtering and the physical interface is enslaved as a trunk port:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 systemd.network.netdevs.\u0026#34;10-br-trunk\u0026#34; = { netdevConfig = { Name = \u0026#34;brtrunk\u0026#34;; Kind = \u0026#34;bridge\u0026#34;; }; bridgeConfig = { VLANFiltering = true; DefaultPVID = 1; }; }; systemd.network.networks.\u0026#34;40-physical-trunk\u0026#34; = { matchConfig.Name = \u0026#34;enp5s0\u0026#34;; networkConfig.Bridge = \u0026#34;brtrunk\u0026#34;; linkConfig.RequiredForOnline = \u0026#34;enslaved\u0026#34;; }; The host\u0026rsquo;s management address moved from a per-VLAN bridge onto a VLAN interface of the trunk bridge. Some transport VLANs existed only inside the bridge and had no host-layer address at all.\nAgain, I staged the configuration for the next boot rather than attempting to replace the bridge beneath an active SSH session. The migration changed the master of the physical NIC, removed several bridges, recreated VLAN interfaces, and moved the default route. A reboot was the deterministic path.\nConflicting forwarding sysctls After one deployment, host-to-container routing failed even though net.ipv4.ip_forward appeared in the configuration.\nLinux exposes closely related forwarding sysctls:\n1 2 net.ipv4.ip_forward net.ipv4.conf.all.forwarding Modules had written both aliases with conflicting values. The canonical per-family setting remained zero at runtime, leaving forwarding disabled. I made both values explicit:\n1 2 3 4 5 boot.kernel.sysctl = { \u0026#34;net.ipv4.ip_forward\u0026#34; = 1; \u0026#34;net.ipv4.conf.all.forwarding\u0026#34; = 1; \u0026#34;net.ipv6.conf.all.forwarding\u0026#34; = 1; }; Both names refer to closely related kernel settings and another module can write one after the other. So I check the runtime values after activation:\n1 2 3 sysctl net.ipv4.ip_forward sysctl net.ipv4.conf.all.forwarding sysctl net.ipv6.conf.all.forwarding The kernel is the final source of truth.\nSource-only addresses and automatic routes The host also had secondary interfaces used only to originate scans or reach macvlan workloads. Those addresses must not automatically install connected prefix routes if another interface owns the real return path.\nI set AddPrefixRoute=false on those source-only legs and kept reverse-path filtering loose where asymmetric routes were intentional. Otherwise Linux could prefer the newly connected /24, send replies out the wrong interface, and turn a harmless scan address into an SSH lockout.\nThis is a niche detail with a broad principle: every new address can also create a route. During migrations, compare the routing table, not just the address list.\nChecks after each migration After each stage I checked the system from the bottom up.\nLink ownership 1 2 3 networkctl status ip -br link ip -br address Is the physical NIC enslaved to the intended bridge? Are VLAN devices attached to the intended parent? Is each host address on the bridge/VLAN device, not the addressless port? Routes and forwarding 1 2 3 ip route ip -6 route sysctl net.ipv4.conf.all.forwarding Is there exactly one intended default route? Did source-only addresses add unwanted connected routes? Is forwarding enabled in runtime state? Namespaces Can a container resolve DNS? Does its namespace contain every expected interface? Can it reach both an internal backend and an external endpoint? Does restarting the parent/network service reattach the child cleanly? Reboot behavior Does the host return on the management address without manual intervention? Do services wait for the interface state they actually require? Does a second ordinary configuration apply leave the topology intact? Final setup systemd-networkd did not make the network itself simple, but it made device ownership and startup dependencies visible. The order I would use again is:\nreproduce the working topology declaratively; observe the lifecycle failures that the old scripts concealed; introduce stable bridge ownership; consolidate only after the dependencies are understood; stage non-live-reparentable changes for reboot with console rollback. Do not assume a declarative network change is safe to apply over SSH. Moving a management IP from a VLAN to a bridge still removes and recreates interfaces. For those changes, I used a boot-time migration and kept the previous NixOS generation available from the console.\n","date":"2026-06-18T00:00:00+05:30","permalink":"https://blog.dexome.com/post/scripted-to-systemd-networkd/","title":"Moving My NixOS Network to systemd-networkd"},{"content":"I was running Traefik as a native NixOS service. Over time, I had added proxy services to let it reach applications in Podman networks. Instead of keeping those extra proxies, I decided to run Traefik itself in Podman and attach it to the required networks.\nThe container started and its health check passed, but some routes returned 502 Bad Gateway. Requests from another VLAN timed out, and after recreating the container the ingress IP stopped responding.\nThere were three separate issues:\nloopback stopped meaning the host; a multi-homed network namespace chose the wrong return route; the macvlan MAC changed while the gateway still remembered the old one. The container network setup Native Traefik had accumulated proxy shims to reach services spread across host processes, Podman bridges, and network containers. Moving Traefik itself into Podman let it join application networks directly and removed a layer of socket forwarding.\nThe resulting Traefik container joined three networks:\npodman_network_vlan_100 for the ingress VIP 10.100.100.21; podman_rproxy at 10.89.0.2 for host and internal backends; an internet bridge at 172.28.0.12 for outbound access. flowchart TB CLIENTS[Clients and OPNsense] --\u003e|HTTPS to 10.100.100.21| VLAN[macvlan VLAN 100] VLAN --\u003e T[Traefik 3.7.5] T --\u003e|10.89.0.1 backend ports| HOST[podman_rproxy host gateway] HOST --\u003e SERVICES[Netdata, Glances, Cockpit, go2rtc, Home Assistant] T --\u003e|container DNS and updates| INET[172.28.0.0/24 Internet bridge] Containerized Traefik is multi-homed: ingress, private backends, and Internet access are separate network legs. The three networks worked, but the old Traefik configuration still assumed it was running in the host network namespace.\nHost services returned 502 Several file-provider routes still targeted backends such as:\n1 2 servers: - url: http://127.0.0.1:19999 That URL worked when Traefik was a host process. Inside the container, 127.0.0.1 referred to the Traefik container itself. Netdata, go2rtc, Cockpit, and other host services had not moved with it.\nServices on the same Podman bridge continued to work. Only host services and systemd-nspawn backends failed, while the Traefik health check remained green.\nI tested from both sides of the namespace boundary:\n1 2 3 4 5 # Works from the NixOS host curl -fsS http://10.89.0.1:8123/ # Initially failed from Traefik\u0026#39;s namespace podman exec traefik wget -qO- -T2 http://10.89.0.1:8123/ The second command showed the problem without involving DNS, TLS or OPNsense.\nThe fix had three parts:\nBind host services to the podman_rproxy gateway, 10.89.0.1, rather than relying on loopback. Allow only the required backend ports on the podman_rproxy interface in the NixOS firewall. Change Traefik\u0026rsquo;s backend URLs from 127.0.0.1 to 10.89.0.1. For Netdata, I also widened its application-level allow list from localhost to the 10.89.0.0/24 proxy subnet. Opening the host firewall alone would not have overridden the service\u0026rsquo;s own filter.\nflowchart TB subgraph NATIVE[Native Traefik] NT[Traefik process] --\u003e|127.0.0.1| NS[Host service] end subgraph CONTAINER[Containerized Traefik] CT[Traefik container] --\u003e|127.0.0.1| SELF[Traefik container loopback] CT --\u003e|10.89.0.1| HS[Host service on podman_rproxy] end NT -. moved into container .-\u003e CT The same URL points at a different machine after Traefik crosses into a container namespace. The 502s disappeared, but off-subnet clients still timed out.\nRequests from other VLANs timed out Traefik had more than one default route because netavark attached multiple networks. The route selected during one container creation was not guaranteed to be the same route selected after another.\nClients on VLAN 100 were on-link with the macvlan VIP, so replies used the connected route and worked. A client from another VLAN reached 10.100.100.21 through OPNsense, but Traefik\u0026rsquo;s reply followed the internet bridge\u0026rsquo;s default route. The request entered through the macvlan and the response tried to leave through a different gateway.\n1 2 off-subnet request: client -\u0026gt; OPNsense -\u0026gt; Traefik macvlan wrong reply: Traefik -\u0026gt; Podman Internet bridge -\u0026gt; nowhere useful Testing only from VLAN 100 made the setup look correct. I had to test from another VLAN and inspect the route inside Traefik\u0026rsquo;s network namespace.\nInside Traefik\u0026rsquo;s network namespace, ip route get \u0026lt;client-address\u0026gt; exposed the selected gateway. I installed an explicit lower-metric default route through the VLAN 100 gateway:\n1 ip route replace default via 10.100.100.1 metric 50 The internet-bridge route remained at metric 100. This was applied by the NixOS unit after Podman created the namespace, so every recreation restored the same decision.\nflowchart TB C[Client on another VLAN] --\u003e|request| FW[OPNsense] FW --\u003e|10.100.100.21| T[Traefik] T -. wrong default before fix .-\u003e I[Internet bridge gateway] I -. dropped reply .-\u003e C T --\u003e|metric 50 after fix| FW FW --\u003e|symmetric reply| C Equal default routes made off-subnet replies nondeterministic; the lower-metric VLAN route restored symmetry. After the route fix, off-subnet curls returned HTTP 200 and ip route get showed the macvlan gateway. Then a later container recreation broke ingress once more.\nThe macvlan MAC changed after recreation The macvlan attachment had a static IP but no static MAC. Podman generated a new MAC when Traefik was recreated. OPNsense and the upstream switching path still associated 10.100.100.21 with the previous MAC.\nThe container health and route table still looked correct because the stale entry was on OPNsense and the switching path, not inside the container.\nI pinned the attachment identity in the Podman network specification:\n1 \u0026#34;podman_network_vlan_100:ip=10.100.100.21,mac=02:42:0a:64:64:15\u0026#34; The locally administered 02:42 prefix avoids claiming a vendor identity. The remaining octets encode the private address, which makes the value easy to audit. More importantly, recreation no longer changes the layer-2 identity.\nChecks after the migration A reverse proxy health check proves only that Traefik can answer inside its own namespace. My final test matrix covered each boundary explicitly:\nTest What it proves Curl backend from NixOS host Backend process is listening Curl backend from Traefik container Host firewall, bind address, and app allow list work Request ingress from VLAN 100 Macvlan listener and on-link return path work Request ingress from a different VLAN Routed return path uses the correct gateway Inspect ip route get in the namespace Route choice is deterministic Recreate Traefik and repeat all tests MAC and post-start route configuration persist I also checked the gateway neighbor entry after recreation rather than waiting for a browser symptom.\nKey points Moving Traefik into a container changed more than the process manager. The native service could reach host loopback and used the host routing table. The container could use only its attached interfaces and had multiple default routes.\nFor a similar migration, check all backend bind addresses, test from inside the Traefik container, test ingress from another subnet and recreate the container once before considering it complete. A static macvlan IP should also have a stable MAC when upstream devices keep neighbor entries for it.\n","date":"2026-06-11T00:00:00+05:30","permalink":"https://blog.dexome.com/post/containerize-reverse-proxy-gotchas/","title":"Moving Traefik into a Podman Container"},{"content":"I use an Intel Arc A310 with 4 GB VRAM for Frigate, Immich and media transcoding. I wanted to use the same GPU for a private LibreChat setup, with Ollama as the model server and Netdata exposed through MCP.\nSmall models worked for normal chat. Netdata tool calling did not work reliably. The first problem was Ollama\u0026rsquo;s default 4096-token context, which removed the MCP tool definitions from the prompt. After increasing the context, the remaining problem was the capability of models which could fit on this GPU.\nMy setup Ollama 0.12.11 and later include an experimental Vulkan backend that can use Intel Arc. I passed only /dev/dri/renderD128, joined the render and video groups, and kept the API on private Podman networks. LibreChat reached Ollama by container DNS; Ollama used a separate egress bridge to pull models.\nflowchart TB USER[Browser] --\u003e CHAT[LibreChat] CHAT --\u003e|private bridge| O[Ollama and Vulkan] CHAT --\u003e|13 tool schemas| MCP[Netdata MCP] O --\u003e ARC[Intel Arc A310, 4 GB] VIDEO[Frigate, Immich, Plex, Jellyfin] --\u003e ARC O --\u003e|model pulls only| NET[Internet bridge] The model API stays private while one render node is shared with the host\u0026#39;s media and vision workloads. The Vulkan backend worked for small local models. The first issue appeared before the model generated any response.\nMCP tools were removed by context truncation LibreChat sent the system message, Netdata instructions, 13 MCP tool schemas, and the user turn. The resulting first prompt was about 11,511 tokens. Ollama\u0026rsquo;s default context was 4,096.\nThe log made the failure explicit:\n1 truncating input prompt limit=4096 prompt=11511 keep=4 new=4095 The tool definitions were near the discarded end. A model that never receives the schemas cannot issue a tool call, regardless of its instruction following. It answered in prose because prose was the only action left.\nflowchart TB P[11.5k-token prompt] --\u003e CUT{4,096-token context} CUT --\u003e KEEP[Small retained slice] CUT -. discarded .-\u003e TOOLS[Netdata instructions and 13 tool schemas] KEEP --\u003e MODEL[Model sees no callable tools] MODEL --\u003e TEXT[Plain-text answer] At the default 4k context, truncation removed the MCP schemas before the model evaluated the request. Raising OLLAMA_CONTEXT_LENGTH made structured tool attempts appear. That proved truncation was causal. It did not yet produce a usable system.\nLarger context used more VRAM At 16,384 tokens, the KV cache occupied about 1.8 GB. Only 18 of 29 model layers fit on the GPU; the other 11 spilled to CPU. Processing the first 11.5k-token prompt took 131 seconds, during which nothing streamed. LibreChat aborted before the first token.\nTurning off the 6 KB server-instructions block helped less than expected. The prompt still measured 10,189 tokens because the 13 tool schemas themselves were the dominant cost, and LibreChat could not expose only a subset of MCP tools. An 8,192-token context still truncated them.\nThe practical setting was 12,288: enough for the tool prompt plus answer headroom, with fewer layers displaced than at 16k. The first turn remained slow, roughly 80 to 100 seconds for a 3B model. Follow-up turns were fast because Ollama cached the prompt prefix; one 67-token follow-up returned in 0.8 seconds.\nflowchart TB C4[4k context] --\u003e|small KV cache| FAST[More GPU residency] C4 --\u003e|but| TRUNC[Tool schemas truncated] TRUNC -. next test .-\u003e C12[12k context] C12 --\u003e|schemas fit| VISIBLE[Tools visible] C12 --\u003e|larger KV cache| SPLIT[Some layers spill to CPU] SPLIT -. next test .-\u003e C16[16k context] C16 --\u003e|about 1.8 GB KV| SLOW[18 of 29 layers on GPU, 131s prompt evaluation] A larger context fixed schema visibility but enlarged the KV cache and forced model layers onto the CPU. The model files fitted, but the model, KV cache and the other GPU workloads did not always fit at the same time.\nModels I tested With logs confirming no truncation, I tested the models against simple Netdata questions.\nModel GPU placement First prompt Result Qwen2.5 1.5B 29/29 layers about 55s Printed a tool call as JSON text; LibreChat could not execute it Llama 3.2 3B 20-21/29 layers 105-131s Emitted a real call with schema-invalid arguments Qwen2.5 7B 13/29 layers about 218s Too slow and returned an unusable result The 3B model even failed the no-argument list_raised_alerts schema. This was not limited to one complicated metrics query. It could choose a tool and emit a tool-call shape, but not produce arguments the MCP client accepted reliably.\nLibreChat\u0026rsquo;s \u0026ldquo;Ran tool\u0026rdquo; pill was not proof of success. It appeared when dispatch began; the execution log later showed Received tool input did not match expected schema. Without checking that log, I would have mistaken an attempted call followed by hallucinated prose for real monitoring data.\nIncreasing the timeout only waited longer for the same result. It did not fix the invalid tool arguments or make the 7B model fit better.\nTesting the Intel SYCL backend Intel\u0026rsquo;s old IPEX-LLM path looked attractive because it promised optimized SYCL inference. By the time I evaluated it, the repository was archived, its bundled Ollama was old, the images used rolling tags, and the project was flagged with known security issues. I rejected it.\nThe maintained high-throughput option is upstream llama.cpp\u0026rsquo;s Intel SYCL image. I staged it beside Ollama rather than replacing the working service. It needed both renderD128 and this host\u0026rsquo;s actual card node, card1; assuming card0 prevented container creation.\nllama-server introduced its own constraints:\nthe default four parallel slots divided the usable context per request; --parallel 1 was necessary for the large single-user prompt; --jinja was required for structured tool calls; oversized prompts hard-failed unless context shifting was configured; one server process loaded one GGUF model, unlike Ollama\u0026rsquo;s model manager; the first SYCL request paid a JIT compilation cost. I kept Ollama plus Vulkan as the default. SYCL can improve throughput, but it does not make a 3B model format better arguments, nor does it make a 7B model fit in 4 GB.\nWhat works well on the A310 The A310 is useful for private chat, summarization and small experiments. Llama 3.2 3B was the best local default from my tests. It fitted well enough and could emit a real tool-call structure. Qwen2.5 1.5B was faster but printed the tool call as text.\nIt was not reliable for the 13 Netdata MCP tools. Their schemas required a large context and the models which remained usable on 4 GB VRAM could not consistently produce valid tool arguments.\nThe debugging order I use now is:\nConfirm the accelerator backend actually loaded; do not infer GPU use from container access to /dev/dri. Read the prompt token count and truncation log. Measure KV cache size and GPU layer placement at the chosen context. Separate prompt-evaluation latency from generation speed. Verify tool execution success in logs, not in UI decoration. Check whether the remaining problem is model capability rather than another timeout or context setting. I kept Ollama with Vulkan as the default because it supports model management and normal local chat worked. llama.cpp with SYCL is available for comparison, but a faster backend does not make the small model better at producing schema-valid tool calls.\n","date":"2026-06-10T00:00:00+05:30","permalink":"https://blog.dexome.com/post/self-hosting-llms-consumer-gpus/","title":"Running Local LLMs on an Intel Arc A310"},{"content":"NixOS makes it simple to change the backend used by virtualisation.oci-containers:\n1 virtualisation.oci-containers.backend = \u0026#34;podman\u0026#34;; In my setup, this changed 36 containers with custom networks, static IP addresses, shared network namespaces, health checks, GPU devices and bind mounts. Traefik, Netdata, Glances and Dozzle were also using the Docker API.\nMost Docker options worked with Podman, but a few differences stopped containers from starting. Network recreation and systemd health checks caused another set of problems after the initial migration.\nWhy I moved to Podman I wanted a runtime that fit the host\u0026rsquo;s existing systemd ownership model. Podman gave me generated podman-\u0026lt;name\u0026gt;.service units, native health actions, a rootful socket at /run/podman/podman.sock, and no long-running Docker daemon.\nI intentionally did not enable dockerCompat. Traefik, Netdata, Glances, and Dozzle were pointed at Podman\u0026rsquo;s Docker-compatible API explicitly. Portainer and the autoheal sidecar were removed; Cockpit Podman and --health-on-failure=kill took their places.\nflowchart LR NIX[NixOS OCI declarations] --\u003e UNITS[Generated systemd units] UNITS --\u003e|before| D[Docker containers] UNITS --\u003e|after| P[Podman containers] API[Traefik, Netdata, Glances, Dozzle] --\u003e|before| DS[\"/var/run/docker.sock\"] API --\u003e|after| PS[\"/run/podman/podman.sock\"] NET[Declared networks] --\u003e NAV[netavark and aardvark-dns] NAV --\u003e P The runtime changed under a declarative NixOS service graph; every explicit Docker unit and socket edge had to move with it. The migration began with mechanical changes: rename unit references from docker-* to podman-*, change supplementary group membership from docker to podman, and replace socket paths. Then the behavioral differences appeared.\n--health-on-failure needs a health check I replaced autoheal with Podman\u0026rsquo;s native unhealthy action:\n1 --health-on-failure=kill That option is valid only when the container has an explicit health command or the image contains a baked HEALTHCHECK. Docker had tolerated the old setup. Podman refused to create affected containers with exit 125:\n1 Error: cannot set on-failure action to kill without a health check Documentation and upstream Dockerfiles were not reliable enough because the pinned image was the thing actually being executed. This became the authoritative test:\n1 2 podman image inspect IMAGE \\ --format \u0026#39;{{if .Config.Healthcheck}}HAS{{else}}NONE{{end}}\u0026#39; Only Plex had the baked health check I expected. Images for Dozzle, Gluetun, qBittorrent, Stirling PDF, Gogs, MeTube, and several Immich components did not. For those I either added a real --health-cmd or removed the unhealthy action.\nTwo more parser differences stopped containers before their applications even started:\nDocker-era option Podman result Replacement --tmpfs=/transcode:uid=912,gid=912 Unknown mount option mode=1777 where appropriate --network=name=NET,ip=IP Comma split into two network names --network=NET:ip=IP These errors were visible in the generated systemd unit logs and were simple to reproduce before starting the application.\nTransient health checks caused exit code 4 Podman implements periodic health checks using transient systemd services. On the first activation, containers were pulling images and warming up while these checks began to run. Some transient podman healthcheck run \u0026lt;id\u0026gt; services returned non-zero during health_status=starting.\nswitch-to-configuration saw failed units and returned exit code 4. A minute later, the declared network unit had succeeded, every container was healthy, and systemctl --failed was empty.\nsequenceDiagram participant A as NixOS activation participant S as systemd participant C as Podman container A-\u003e\u003eS: restart generated units S-\u003e\u003eC: start container C--\u003e\u003eS: health_status=starting S-\u003e\u003eC: transient healthcheck C--\u003e\u003eS: non-zero while warming S--\u003e\u003eA: failed transient unit, exit 4 C--\u003e\u003eS: health_status=healthy Note over S,C: Final state is healthy despite activation result An activation-time health probe can fail while the final steady state becomes healthy. Exit code 4 should not be ignored, but it can be caused by a health check which has already recovered. I check the failed units, network service and current container state separately:\n1 2 3 systemctl --failed --no-pager systemctl status podman-networks.service --no-pager --full podman ps --format \u0026#39;table {{.Names}}\\t{{.Status}}\u0026#39; A transient probe that has already cleared is different from a network reconciler that is still failed.\nStopped containers are still attached to networks I wrote a declarative network reconciler so changes to subnet, gateway, or interface name would recreate a drifted Podman network. Container units were partOf the network service, so restarting the network stopped its users first.\nThe assumption was plausible and wrong. A stopped container remains associated with its network. The reconciler ran:\n1 podman network rm podman_network_rproxy Podman correctly refused:\n1 network is being used The drift path needed podman network rm -f. Force removal disconnected and removed the stopped container records; systemd recreated them after the network returned.\nNetwork recreation cleared all IPAM leases The force-removal fix exposed the next bug. Recreating the proxy bridge erased netavark\u0026rsquo;s IPAM leases. A dynamic container started first and received 10.89.0.2, the static address reserved in configuration for Traefik.\nTraefik then looped with:\n1 IPAM error: requested ip address 10.89.0.2 is already allocated The old network had worked only because its historical lease happened to keep .2 occupied by the intended container. Start order became visible after a cold reconstruction.\nI separated the static and dynamic address ranges:\n1 2 3 4 subnet = \u0026#34;10.89.0.0/24\u0026#34;; gateway = \u0026#34;10.89.0.1\u0026#34;; ipRange = \u0026#34;10.89.0.128/25\u0026#34;; leaseRange = \u0026#34;10.89.0.129-10.89.0.255\u0026#34;; Addresses .2 through .127 are available for deliberate static assignments; dynamic leases begin at .129. I verified the expanded lease range with a disposable network before trusting it in the drift comparison.\nflowchart TB R[Recreate 10.89.0.0/24 network] --\u003e EMPTY[IPAM lease table is empty] EMPTY --\u003e|without range separation| DYN[Dynamic backend starts first and takes .2] DYN --\u003e FAIL[Traefik static .2 fails] EMPTY --\u003e|dynamic pool .129-.255| HIGH[Backend receives a high address] HIGH --\u003e OK[Traefik always claims static .2] A network rebuild forgets historical leases. Reserving separate static and dynamic ranges removes start-order dependence. NixOS firewall reload broke aardvark-dns aardvark-dns listens on each netavark bridge gateway. Netavark inserted rules allowing DNS, but the NixOS firewall reload flushed those runtime-generated rules. Existing internal bridges were not recreated afterward, so their rules did not return.\nContainers began timing out on names served by the bridge gateway. OAuth2 Proxy and Immich crash-looped; containers with explicit DNS servers were unaffected.\nThe fix was to make bridge DNS part of the host firewall declaration. My bridge interfaces deliberately retain a podman prefix, so the NixOS podman+ interface wildcard can allow TCP and UDP port 53 after every reload.\nThis incident also taught me that --internal does not mean \u0026ldquo;DNS-free.\u0026rdquo; An internal bridge may need its embedded resolver for other names on that same private network even when it has no Internet route.\nChecks I would use for another migration I would split validation into four layers:\nInspect every pinned image for a baked health check instead of relying on its documentation. Create every container once to find unsupported option formats. Delete and recreate the declared networks so the test starts with empty IPAM state. Reload the NixOS firewall and check DNS from an internal bridge. Reboot and make sure the same containers and static IPs return without relying on the old start order. Podman has been working well after these fixes. The container applications did not need changes; most of the migration work was around image metadata, command line parsing, systemd health checks and netavark state.\n","date":"2026-06-08T00:00:00+05:30","permalink":"https://blog.dexome.com/post/docker-to-podman-migration-gotchas/","title":"Moving My NixOS Containers from Docker to Podman"},{"content":"I keep short incident notes for problems in my homelab. I normally write them after the service is stable and before closing the terminals used during the investigation.\nShell history can show the commands I ran, but it does not explain which output ruled out the firewall, why one repair did not work or which configuration change was the permanent fix. By the next day, those details are already harder to write down correctly.\nStart with the symptom, cause and fix I force the incident into a problem-to-outcome sentence:\nOff-subnet Traefik ingress timed out because its multi-homed container replied through the Internet bridge; a lower-metric default route through the ingress VLAN restored symmetric routing.\nThis gives me enough information to decide later whether the complete note is related to a new problem.\nA weak summary sounds like this:\nFixed Traefik networking.\nIt is short but cannot route a future investigation. It does not distinguish DNS failure, backend reachability, stale ARP, firewall policy, or asymmetric routing.\nIf I can only write “fixed Traefik networking,” I may have restored the service without finding the actual cause.\nThe format I use The useful reader is tired, under time pressure, and staring at a similar but not necessarily identical symptom. That reader needs discriminating evidence, not a polished chronology of every command.\nMy note structure is small:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 # Specific incident title ## Symptom What users and monitoring observed. ## Evidence The few checks that separated plausible causes. ## Root cause The mechanism, not only the broken component. ## Fix The exact durable change and any one-time repair. ## Verify and undo How to prove the fix, and how to back it out. flowchart TB S[Symptom] --\u003e H[Competing hypotheses] H --\u003e E[Discriminating evidence] E --\u003e R[Root cause mechanism] R --\u003e F[Durable fix] F --\u003e V[Verification and rollback] A useful incident note compresses the investigation around the evidence that changed the decision. Most of my notes are short. The important part is keeping the evidence which changed the diagnosis.\nExample from Radarr and mergerfs Radarr could not replace a movie after Copyparty renamed its directory. The directory was owned by Copyparty\u0026rsquo;s UID and the shared media group, but group write was missing.\nThe first diagnosis was correct: change the directory from 2755 to 2775. The first repair was incomplete because I changed only the btank backing path. Radarr still failed.\nThe visible path was a mergerfs union. A duplicate directory under the btwo branch still had mode 2755, and mergerfs exposed that metadata. Repairing only the branch containing the media file was not enough.\nA poor incident note would say:\n1 Radarr permissions fixed with chmod 2775. The operational note says:\n1 2 3 Copyparty-created 0755 directories blocked Radarr replacement through mergerfs. Repair every raw backing branch, never only the merged view, then enforce Copyparty chmod_d 775 and chmod_f 664 for future entries. The second note records both the existing filesystem repair and the Copyparty configuration required for new files and directories.\nFailed attempts worth keeping An incident can contain dozens of checks. Most do not deserve permanent space. I keep a failed attempt when it explains a recurring trap:\nroot privileges did not bypass Netdata\u0026rsquo;s systemd device cgroup; a stopped Podman container remained associated with its network; a healthy container did not prove off-subnet return routing; setting a flow to disabled did not stop its persisted realtime subscription; a synthetic shell respected the umask while the real s6-supervised process reset it. I omit typos and checks which did not add any information. I keep a failed attempt when it explains why a reasonable fix did not change the result.\nKeep the diagnostic sequence There is the incident timeline: what happened in production. There is also the reasoning timeline: which evidence changed the hypothesis.\nflowchart TB subgraph INCIDENT[Production timeline] I1[Change activated] --\u003e I2[Service degraded] --\u003e I3[Service restored] end subgraph REASONING[Reasoning timeline] R1[Suspect firewall] --\u003e R2[Packet arrives on ingress] R2 --\u003e R3[Route lookup selects wrong gateway] R3 --\u003e R4[Pin lower-metric ingress route] end I2 -. starts .-\u003e R1 R4 -. produces .-\u003e I3 Operational chronology and diagnostic reasoning overlap, but the note should preserve the decisions, not every event. For future debugging, packet arrives and route lookup selects wrong gateway are more useful than the exact minute when each curl command ran.\nWrite the note before cleaning up I start the note after service is stable but before deleting scratch files, closing logs, or relying on memory. The fastest workflow is:\nWrite the one-line outcome. Paste only the decisive errors and commands. Explain the mechanism in plain language. Separate temporary recovery from the permanent fix. Add verification and rollback. Remove secrets and irrelevant identifiers. Link the note from a small topic index. I can edit the wording later. Recreating the decisive command output is more difficult after the containers, logs and temporary files have changed.\nRecord the recovery and permanent fix separately Several incidents taught me to record two fixes explicitly.\nRecovery action Durable correction Clear a stale neighbor entry Pin the macvlan MAC across recreations Repair existing directories Change creation policy and inherited ACLs Restart Netdata Configure collector autodetection retry Delete a stuck KV key Replace TTL presence with timestamp comparison Manually restart a container Correct its systemd dependency or health contract This prevents a one-time recovery command from being mistaken for the permanent configuration change.\nAdd it to the topic index A folder full of excellent postmortems still fails if nobody knows what is in it. I keep a routing index grouped by service. Each note contributes its problem-to-outcome sentence.\nflowchart LR N[Short dated incident note] --\u003e I[One-line entry in topic index] Q[Future symptom] --\u003e I I --\u003e|summary matches| N N --\u003e A[Apply prior check or avoid prior mistake] The cost of a postmortem includes writing it and finding it again; a compact index reduces the second cost. I use full-text search for exact errors. The index helps when the new symptom is similar but does not contain the same error message.\nKeep old notes but mark them superseded Operational knowledge changes. A note may describe a temporary architecture, a version-specific bug, or a test later disproved in production.\nI do not silently rewrite the old conclusion. I mark it superseded and link it to the replacement; the replacement links back. That preserves why the earlier decision was reasonable while making the current instruction obvious.\nPeriodically, I scan for stale statuses, contradictory guidance, duplicated incidents, broken links, and notes that should be promoted into stable service documentation. A tiny coverage script checks that every dated note appears in the index. Humans still decide whether the summaries are true.\nWhen I create a note I write an operational note when at least one of these is true:\nthe symptom had more than one plausible cause; the winning check was non-obvious; a reasonable attempted fix made no difference; recovery and durable correction were different; state outside the visible configuration mattered; the same mistake could affect another service; rollback required knowledge not encoded in the change itself. Routine changes do not need an incident note. I create one when the investigation found something which is not obvious from the final configuration.\nOnce the fix is validated, I write the short note and add its one-line summary to the index. That is normally enough for the next investigation to start from the useful check instead of repeating the complete incident.\n","date":"2026-06-05T00:00:00+05:30","permalink":"https://blog.dexome.com/post/operational-postmortems-future-self/","title":"How I Write Incident Notes for My Homelab"},{"content":"I use AI coding agents regularly while working on my homelab repository. The chat history is useful during a task, but a new session does not automatically know what was proved in an older one.\nThis became a problem for operational details. For example, one session found the safe way to evaluate my Nix flake without copying 20 GiB of ignored model files. Another found the exact SSH and TTY sequence required for privileged diagnostics. I did not want to investigate those details again.\nI now store these notes as Markdown under docs/agent-memory/. They are part of the repository, so I can review and update them along with the configuration.\nWhat I wanted from these notes The system needed to be:\ndurable: survive chat sessions, machines, and editor profiles; reviewable: change through the same review process as code; routable: let an agent find one relevant note without reading hundreds; historical: preserve old decisions without presenting them as current; safe: record where secrets live, never their values; cheap: make adding a lesson easier than rediscovering it. I did not want one large file which every agent had to read at the start of a task. I also wanted the notes to remain visible outside one editor or agent.\nOne Markdown file for each topic Every memory is a dated Markdown file under docs/agent-memory/:\n1 2 3 2026-06-12-traefik-macvlan-asymmetric-return-route.md 2026-07-23-copyparty-rename-strips-group-write-arrs-import-denied.md 2026-08-10-netdata-2.10.3-collector-fixes.md The ISO date gives a useful filesystem order. The descriptive suffix makes the file discoverable with ordinary text search. One topic per file lets a later note supersede one conclusion without invalidating an unrelated section of a large document.\nNew notes carry queryable frontmatter:\n1 2 3 4 5 6 7 8 9 --- type: incident title: Podman network drift recreate needs rm -f description: Stopped containers remain associated with their networks. timestamp: 2026-06-12 tags: [podman, networking] resource: hosts/heavymetal/containers.nix status: resolved --- I normally write the body in this order:\nWhat was the symptom? What evidence separated it from similar failures? What was the root cause? What exact change fixed it? How was the fix verified? How can it be undone or superseded? I include a failed attempt only when it is likely to be tried again. Normal terminal exploration and command typos do not need to be stored.\nUsing a small index to find the correct note Opening every note at the start of every request would replace forgetting with context overload. The repository instead has a hand-curated index.md, grouped by service or topic. Each entry is one complete problem-to-outcome sentence:\n1 2 3 - [2026-06-12-podman-network-drift-recreate-needs-force.md](...) — Recreating a drifted Podman network requires rm -f because stopped containers remain associated with it. (_resolved_) The agent\u0026rsquo;s startup rule is simple: read the index, select the few summaries that match the current task, then open only those notes.\nflowchart TB Q[New engineering request] --\u003e I[Read compact routing index] I --\u003e S{Select matching summaries} S --\u003e N1[Open relevant incident note] S --\u003e N2[Open relevant reference note] N1 --\u003e W[Work with prior evidence] N2 --\u003e W ALL[Hundreds of unrelated notes] -. not loaded .-\u003e W The routing index keeps startup context small while preserving access to detailed operational evidence. I write these summaries by hand because a filename or heading usually does not say which fix actually worked. A script checks that every note is included in the index, but it does not generate the summary.\nNot every note needs a dated file Not every fact belongs in the dated-note catalog. I separate three classes:\nMemory class Example Lifetime Incident or migration Why macvlan replies used the wrong route Historical, may be superseded Living reference Port allocation inventory Updated in place Durable service guidance Stable operating constraint Promoted into reference documentation A port inventory would become noisy as a sequence of dated files. A completed incident should not be silently rewritten whenever understanding changes. For example, I update the port allocation inventory in place. An incident stays as a dated file because a later note may supersede it without removing the old evidence.\nMarking an old note as superseded Deleting an old note destroys the path that explains why a decision existed. Leaving it unmarked lets an agent follow obsolete instructions. The compromise is bidirectional supersession.\nThe old note says:\n1 \u0026gt; **Superseded by:** [2026-08-17-new-design.md](...) — reason. The new note says:\n1 \u0026gt; **Supersedes:** [2026-06-05-old-design.md](...). Both frontmatter records and index entries carry the same state. The old evidence remains available, but every retrieval path points toward current guidance.\nflowchart TB OLD[Old note: status superseded] --\u003e|superseded by| NEW[New note: current guidance] NEW --\u003e|supersedes| OLD INDEX[Routing index] --\u003e|current entry| NEW INDEX -. historical entry marked superseded .-\u003e OLD Supersession preserves the audit trail while making the current instruction unambiguous. I used this when a temporary design was retired and when a shell test showed that umask 0002 worked but the real container supervisor reset it. The old test was still useful, but the new note needed to be the current instruction.\nTelling the agent when to read the notes Notes do not help if the agent does not know when to read them. Repository instructions establish a retrieval protocol:\nAt the start of a request, read the routing index. Open only relevant notes. Consult living inventories before changing shared resources such as ports. Record a newly proven operational lesson in the repository. Update the index in the same change. Without these repository instructions, the notes would only be documentation which an agent might or might not find.\nflowchart TB TASK[Task begins] --\u003e ROUTE[Index and instruction routing] ROUTE --\u003e PRIOR[Relevant prior knowledge] PRIOR --\u003e ACT[Implement and validate] ACT --\u003e LESSON{New durable lesson?} LESSON --\u003e|yes| NOTE[Write dated note and index summary] NOTE --\u003e CHECK[Run coverage and consistency checks] CHECK --\u003e TASK LESSON --\u003e|no| DONE[Finish] Work produces evidence, evidence becomes a note, and future instructions route the next task back through it. Checking the index The repository includes a small dependency-free checker. It compares dated files on disk with links in the index and reports:\nnotes missing from the index; index links whose files no longer exist; notes without frontmatter. It exits nonzero for missing coverage or broken links. It does not generate summaries, decide which topic heading is best, or resolve contradictory advice. Those are semantic tasks.\nA periodic manual pass checks what code cannot reliably infer:\nstale pending or parked statuses; one-directional supersession links; newer notes that contradict older guidance; duplicate incidents without a relationship; ports or service references that drifted; lessons mature enough to move into stable documentation. The script checks the structure. I still review the summaries and decide whether one note replaces another.\nDo not store secrets or duplicate the code Repository memory must never contain secret values. A note may say that an MQTT password comes from a named SOPS secret and which service consumes it. It should not contain the password, a token copied from a log, or an unredacted credential example.\nI also avoid storing temporary chat details, large summaries of the repository and facts already clear from the code. I add a note when it preserves a useful check, an operational problem or a decision which would take time to reconstruct.\nCurrent workflow I do not use a vector database or an embedding pipeline for this. Markdown, links, frontmatter, repository instructions and a small checker are enough for my repository.\nThe workflow is:\ncapture only proven lessons; compress each lesson into a routable sentence; load details only when relevant; preserve history through supersession; keep every memory visible to the people responsible for the system. The agent still starts a new chat without the old conversation. It first reads the index, opens the notes related to the current task and continues with the checks and fixes already recorded there.\n","date":"2026-06-05T00:00:00+05:30","permalink":"https://blog.dexome.com/post/durable-memory-for-ai-coding-agent/","title":"Keeping AI Agent Notes in My Homelab Repository"},{"content":"Problem Let\u0026rsquo;s consider this scenario.\nWe have a web app where our users can login to it. For authentication, we have setup Amazon Cognito with federated sign-in for Google.\nNow, we have a need to allow the admin users as well to login. All our admin users have email with domain @some-company.com. This @some-company.com domain is managed through Google Workspace.\nAll our admin users have same access, features and capability. We do not need a fine-grain access mechanism.\nSolution Pre-requisites Understanding of Amazon Cognito Working setup of Cognito with Federated identity pool Some working understanding of OAuth2 Since all the admin have same level of access, there is no need to setup an exhaustive authorization mechanism. All we need is a special claim on JWT token to identify the user as admin.\nTo do this, we will be leveraging Cognito UserPool Group capability. When an user signs into the app for the first time, we will look at the domain part of the email and if it matches our company domain, we will add the user to the user group that we have created for admin users.\nCreate UserPool Group We will first create a group in our Cognito User Pool.\nCloudformation for creating group\n1 2 3 4 5 6 AdminUserPoolGroup: Type: AWS::Cognito::UserPoolGroup Properties: Description: Group which contains admin users. GroupName: !Ref AdminGroupName UserPoolId: !Ref UserPool Post Confirmation Lambda We will be using Lambda triggers capabilities of Cognito, especially Post Confirmation Lambda trigger.\nThe Post Confirmation Lambda is triggered only when an user is signed into the app for the first time, in our case, using the Google login button.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 const AWS = require(\u0026#39;aws-sdk\u0026#39;); exports.handler = async (event, context, callback) =\u0026gt; { const { userPoolId, userName } = event; const email = event.request.userAttributes.email; const groupName = process.env.ADMIN_GROUP_NAME; try { if (doesEmailBelongToAdminDomain(email)) { await addUserToGroup({ userPoolId, username: userName, groupName, }); } return callback(null, event); } catch (error) { return callback(error, event); } }; const doesEmailBelongToAdminDomain = (email) =\u0026gt; { const adminEmailDomainName = process.env.ADMIN_EMAIL_DOMAIN_NAME; // Split the email address so we can compare domains const address = email.split(\u0026#34;@\u0026#34;); if (adminEmailDomainName === address[1]) { return true; } return false; }; const addUserToGroup = ({ userPoolId, username, groupName, }) =\u0026gt; { const params = { GroupName: groupName, UserPoolId: userPoolId, Username: username, }; const cognitoIdp = new AWS.CognitoIdentityServiceProvider(); return cognitoIdp.adminAddUserToGroup(params).promise(); }; Token After this setup, if you login with the company domain email, you will get a token that looks like the one below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 { \u0026#34;sub\u0026#34;: \u0026#34;9893sae2-b60d-7869-ac7d-2807fec7f2ab\u0026#34;, \u0026#34;iss\u0026#34;: \u0026#34;https://cognito-idp.us-east-1.amazonaws.com/us-east-1_Oph34e1i7\u0026#34;, \u0026#34;version\u0026#34;: 2, \u0026#34;client_id\u0026#34;: \u0026#34;8968i9neasfgjq0kk864vkhb\u0026#34;, \u0026#34;cognito:groups\u0026#34;: [ \u0026#34;admin\u0026#34;, \u0026#34;userpool_231as3dgf\u0026#34; ] \u0026#34;origin_jti\u0026#34;: \u0026#34;388aff43-2d15-4640-82f7-340a16755724\u0026#34;, \u0026#34;event_id\u0026#34;: \u0026#34;e798a740-af00-4247-9321-67f2050b5144\u0026#34;, \u0026#34;token_use\u0026#34;: \u0026#34;access\u0026#34;, \u0026#34;scope\u0026#34;: \u0026#34;openid\u0026#34;, \u0026#34;auth_time\u0026#34;: 1669805733, \u0026#34;exp\u0026#34;: 1669809333, \u0026#34;iat\u0026#34;: 1669805733, \u0026#34;jti\u0026#34;: \u0026#34;c2d4f281-3f40-4cb2-a2c7-d193fefdc97c\u0026#34;, \u0026#34;username\u0026#34;: \u0026#34;9893sae2-b60d-7869-ac7d-2807fec7f2ab\u0026#34; } You will find that the cognito:groups claim will contain the name of the UserPool group you create above. This would indicate that the user belongs to admin group and you can use for authorization.\n","date":"2022-11-30T13:07:52+05:30","permalink":"https://blog.dexome.com/post/cognito-auto-admin/","title":"Make user as admin during third party (federation) sign-in Amazon Cognito"},{"content":"Sometimes you may need to separate authorization and authentication in NestJS.\nSeparation of concerns Let\u0026rsquo;s consider this scenario.\nWe want to build a system based on microservices and all of the services are built using NestJS.\nThe services are split across domains. We also have a Gateway which act as the entry point for the system and it takes care of authentication.\nAssumptions:\nBasic understanding of NestJS Understanding of how Authentication works in NestJS Authentication NestJS uses PassportJS for authentication. PassportJS has the concept of strategy which simplifies different authentication protocols/methods.\nHere, we use JWT based token validation based on OAuth2. We provide the necessary configuration to the constructor of PassportStrategy\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 import { Injectable } from \u0026#39;@nestjs/common\u0026#39;; import { ConfigService } from \u0026#39;@nestjs/config\u0026#39;; import { PassportStrategy } from \u0026#39;@nestjs/passport\u0026#39;; import { ExtractJwt, Strategy } from \u0026#39;passport-jwt\u0026#39; import { passportJwtSecret } from \u0026#39;jwks-rsa\u0026#39;; import { IdentityProviderConfig } from \u0026#39;../../configuration\u0026#39;; @Injectable() export class AuthenticationStrategy extends PassportStrategy( Strategy, \u0026#39;authn\u0026#39;, ) { constructor(configService: ConfigService) { const idpConfig = configService.get\u0026lt;IdentityProviderConfig\u0026gt;(\u0026#39;idp\u0026#39;); super({ secretOrKeyProvider: passportJwtSecret({ cache: true, rateLimit: true, jwksRequestsPerMinute: 5, jwksUri: `https://${idpConfig.url}/.well-known/jwks.json`, }), jwtFromRequest: ExtractJwt.fromAuthHeaderAsBearerToken(), ignoreExpiration: false, issuer: `https://${idpConfig.url}`, algorithms: [\u0026#39;RS256\u0026#39;], }); } public async validate(payload: any): Promise\u0026lt;any\u0026gt; { return !!payload.sub; } } Authorization In NestJS, authorization happens via Guard. Here, we create the simple guard which authorized the requestor based on a claim called scope in jwt token.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 import { CanActivate, ExecutionContext, mixin, Type } from \u0026#39;@nestjs/common\u0026#39;; import { AuthGuard } from \u0026#39;@nestjs/passport\u0026#39;; import { Scope } from \u0026#39;../scope.enum\u0026#39;; export const ScopeGuard = (scope: Scope): Type\u0026lt;CanActivate\u0026gt; =\u0026gt; { class ScopeGuardMixin extends AuthGuard(\u0026#39;authz\u0026#39;) { async canActivate(context: ExecutionContext) { await super.canActivate(context); const request = context.switchToHttp().getRequest(); const token = request.user; // decoded token return token?.scope.includes(scope); } } const mix = mixin(ScopeGuardMixin); return mix; }; You can definitely decode the jwt token within the Guard. But, I like to separate it into a separate class. This AuthorizationStrategy simply decodes token and is added into the request.user object by the use of PassportStrategy.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 import { Injectable } from \u0026#39;@nestjs/common\u0026#39;; import { ConfigService } from \u0026#39;@nestjs/config\u0026#39;; import { PassportStrategy } from \u0026#39;@nestjs/passport\u0026#39;; import { ExtractJwt, Strategy } from \u0026#39;passport-jwt\u0026#39; import { passportJwtSecret } from \u0026#39;jwks-rsa\u0026#39;; import { IdentityProviderConfig } from \u0026#39;../../configuration\u0026#39;; @Injectable() import { BadRequestException, Injectable } from \u0026#39;@nestjs/common\u0026#39;; import { PassportStrategy } from \u0026#39;@nestjs/passport\u0026#39;; import { Strategy } from \u0026#39;passport-custom\u0026#39;; import * as jwt from \u0026#39;jsonwebtoken\u0026#39;; @Injectable() export class AuthorizationStrategy extends PassportStrategy( Strategy, \u0026#39;authz\u0026#39;, ) { async validate(req: Request) { const header = req.headers[\u0026#39;authorization\u0026#39;]; const token = header.slice(7); if (!token) { throw new BadRequestException(\u0026#39;There is no access token in header\u0026#39;); } const decoded = jwt.decode(token); return decoded; // this decoded token can be accessed from request.user in Guard } } ","date":"2022-11-29T19:03:32+05:30","permalink":"https://blog.dexome.com/post/nestjs-authz-authn/","title":"Authorization and Authentication strategies for Nestjs"},{"content":"Nextcloud is a great app to be used as your private cloud. Once you have setup Nextcloud and have created user accounts in it, you can then use the phone app to configure auto-sync of your phone\u0026rsquo;s camera photos. This works well for every user individually.\nBut now, what if you want all the photos taken by your family (with multiple devices) to be available in one place so that you can have a unified gallery look.\nThe ideal tool to use in this case would be Photoprism and configure it to use Nextcloud\u0026rsquo;s data folder.\nDocker compose files I use below docker compose files to setup Nextcloud and Photoprism. Use these as pointer and modify as necessary.\nnextcloud.yml 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 version: \u0026#34;3.5\u0026#34; services: nextcloud-db: image: mariadb container_name: \u0026#34;nextcloud-db\u0026#34; restart: unless-stopped networks: - backend expose: - \u0026#34;3306\u0026#34; command: --verbose --transaction-isolation=READ-COMMITTED --binlog-format=ROW --innodb-file-per-table=1 --skip-innodb-read-only-compressed volumes: - nextcloud-db:/var/lib/mysql environment: - MYSQL_ROOT_PASSWORD=${NEXTCLOUD_ROOT_PASSWORD} - MYSQL_PASSWORD=${NEXTCLOUD_PASSWORD} - MYSQL_DATABASE=nextcloud - MYSQL_USER=nextcloud-user nextcloud: image: jhnrn/nextcloud-linux container_name: \u0026#34;nextcloud\u0026#34; restart: unless-stopped ports: - 8080:80 links: - nextcloud-db volumes: - /nextcloud/html:/var/www/html - /nextcloud/data:/var/www/html/data environment: - MYSQL_PASSWORD=${NEXTCLOUD_PASSWORD} - MYSQL_DATABASE=nextcloud - MYSQL_USER=nextcloud-user - MYSQL_HOST=nextcloud-db - NEXTCLOUD_TRUSTED_DOMAINS=${NEXTCLOUD_TRUSTED_DOMAINS} - REDIS_HOST=nextcloud-redis - NEXTCLOUD_HOSTNAME=${NEXTCLOUD_HOSTNAME} - NEXTCLOUD_ADMIN_USER=${NEXTCLOUD_ADMIN_USER} - NEXTCLOUD_ADMIN_PASSWORD=${NEXTCLOUD_ADMIN_PASSWORD} depends_on: - nextcloud-db photoprism.yml 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 version: \u0026#34;3.5\u0026#34; services: photoprism: image: photoprism/photoprism:latest container_name: \u0026#34;photoprism\u0026#34; depends_on: - photoprism-mariadb restart: unless-stopped ports: - \u0026#34;2342:2342\u0026#34; # HTTP port (host:container) environment: PHOTOPRISM_ADMIN_PASSWORD: ${PHOTOPRISM_ADMIN_PASSWORD} PHOTOPRISM_SITE_URL: ${PHOTOPRISM_SITE_URL} PHOTOPRISM_DATABASE_DRIVER: \u0026#34;mysql\u0026#34; PHOTOPRISM_DATABASE_SERVER: \u0026#34;photoprism-mariadb:3306\u0026#34; PHOTOPRISM_DATABASE_NAME: \u0026#34;photoprism\u0026#34; PHOTOPRISM_DATABASE_USER: \u0026#34;photoprism\u0026#34; PHOTOPRISM_DATABASE_PASSWORD: ${PHOTOPRISM_DATABASE_PASSWORD} PHOTOPRISM_SITE_TITLE: \u0026#34;PhotoPrism\u0026#34; PHOTOPRISM_SITE_CAPTION: \u0026#34;Browse Your Life\u0026#34; PHOTOPRISM_SITE_DESCRIPTION: \u0026#34;\u0026#34; PHOTOPRISM_SITE_AUTHOR: \u0026#34;\u0026#34; HOME: \u0026#34;/photoprism\u0026#34; working_dir: \u0026#34;/photoprism\u0026#34; volumes: - /nextcloud/data/alice/files/Photos/Camera:/photoprism/originals/alice:ro - /nextcloud/data/dennis/files/Photos/Camera:/photoprism/originals/dennis:ro labels: ofelia.enabled: \u0026#34;true\u0026#34; ofelia.job-exec.photoprism_index.schedule: \u0026#34;@every 1h\u0026#34; ofelia.job-exec.photoprism_index.command: \u0026#34;photoprism index --cleanup\u0026#34; photoprism-mariadb: restart: unless-stopped container_name: \u0026#34;photoprism-db\u0026#34; image: mariadb:10.6 security_opt: - seccomp:unconfined - apparmor:unconfined networks: - backend command: mysqld --innodb-buffer-pool-size=256M --transaction-isolation=READ-COMMITTED --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --max-connections=512 --innodb-rollback-on-timeout=OFF --innodb-lock-wait-timeout=120 volumes: - \u0026#34;photoprism-db:/var/lib/mysql\u0026#34; environment: MYSQL_ROOT_PASSWORD: ${PHOTOPRISM_ROOT_DATABASE_PASSWORD} MYSQL_DATABASE: photoprism MYSQL_USER: photoprism MYSQL_PASSWORD: ${PHOTOPRISM_DATABASE_PASSWORD} ofelia: restart: unless-stopped image: mcuadros/ofelia:latest container_name: ofelia depends_on: - photoprism command: daemon --docker volumes: - /var/run/docker.sock:/var/run/docker.sock:ro Key mentions Use the nextcloud.yml to setup Nextcloud first. You can also download the Nextcloud app from Andriod/Apple App store and setup Photo sync.\nThe key thing to note in the above docker-compose configuration is the volume section of Photoprism. You can see the Photoprism uses bind mount to the folders created by Nextcloud app. While using the photoprism.yml file, make sure the volume section is modified to point to the correct source path.\nIndex refresh This setup is fine and Photoprism would scan the folders and index the files on the first run. But any more photos synced to Nextcloud would not appear on Photoprism as there is no way for Photoprism to know about the photos added to Nextcloud. So, I use ofelia to run to schedule scan job on photoprism. This runs a scan job on Photoprism so that it reflects any new photos added, modified or deleted in Nextcloud.\nThat\u0026rsquo;s it! Now you can go into the Photoprism app and find all your photos of your family in one place.\n","date":"2022-08-04T22:32:03+05:30","permalink":"https://blog.dexome.com/post/nextcloud-photosync/","title":"Connect Photoprism with Nextcloud"},{"content":"pfSense has had difficult times with WireGuard, but that\u0026rsquo;s changing quite fast these days. Now, pfSense has a good stable package for WireGuard which can be used in home/homelab setup (I wouldn\u0026rsquo;t use it in a production environment, yet).\nLet\u0026rsquo;s put the high-level details on what we will be doing here:\nEnable (experimental) support for WireGuard in AirVPN Download config file from AirVPN Install WireGuard package in pfSense Configure WireGuard settings in pfSense 1. Enable (experimental) support for WireGuard in AirVPN Go to Airvpn Preferences and enable Access to BETA features\n2. Download config file from AirVPN Now, goto Config generator and you can see WireGuard available for selection\nThen, click Download in the bottom of the page after making your server selection.\n3. Install WireGuard package in pfSense Now log into PFSENSE. Go to System \u0026gt; Package Manager and make sure you have Wireguard installed\nIf you don\u0026rsquo;t, just click \u0026ldquo;Available Packages\u0026rdquo; and search for Wireguard, and install it.\n4. Configure WireGuard settings in pfSense Now in the top bar, go to VPN \u0026gt; Wireguard \u0026gt; Settings and make sure its enabled.\nTo configure further, you will need to uses the data present in the file downloaded in step 2.\nThe file will look like this:\nTunnels Go to \u0026ldquo;Tunnels\u0026rdquo; tab and click \u0026ldquo;Add Tunnel\u0026rdquo;.\nIn the \u0026ldquo;Tunnel Configuration\u0026rdquo; Check \u0026ldquo;Enable Tunnel\u0026rdquo;. Add a good understandable description like \u0026ldquo;AirVPN Wireguard tunnel\u0026rdquo;. Set the \u0026ldquo;Listen port\u0026rdquo; to the value present in the \u0026ldquo;Endpoint\u0026rdquo; field of the config. In my case, it is 1637. In \u0026ldquo;Interface Keys\u0026rdquo;, copy and paste the \u0026ldquo;PrivateKey\u0026rdquo; field from config and press tab key. You should see the \u0026ldquo;Public Key\u0026rdquo; text auto filled. In the \u0026ldquo;Interface Configuration\u0026rdquo; Enter \u0026ldquo;Interface Address\u0026rdquo; and the CIDR value from config\u0026rsquo;s Interface section. CIDR act as subnet mask. Read more about it here.\nSave the tunnel configuration by clicking \u0026ldquo;Save Tunnel\u0026rdquo;.\nFinal tunnel configuration should look something like this. Peers Go to \u0026ldquo;Peers\u0026rdquo; tab and click \u0026ldquo;Add Peer\u0026rdquo;.\nIn the \u0026ldquo;Peer Configuration\u0026rdquo; Check \u0026ldquo;Enable Peer\u0026rdquo; In \u0026ldquo;Tunnel\u0026rdquo;, select the tunnel which was created in previous step. Add a good understandable description in \u0026ldquo;Description\u0026rdquo;. Uncheck \u0026ldquo;Dynamic\u0026rdquo; in Dynamic Endpoint. Now two new textboxes will appear. Enter the Endpoint (in our case, it\u0026rsquo;s sg.vpn.airdns.org) and Endpoint port (1637, in our case). Keep Alve: 15 (in our case). Then copy and paste the \u0026ldquo;PublicKey\u0026rdquo; and \u0026ldquo;PresharedKey\u0026rdquo; to the respective fields. In the \u0026ldquo;Address Configuration\u0026rdquo; Enter 0.0.0.0 in Allowed ip and select \u0026ldquo;0\u0026rdquo; for CIDR. Add the description \u0026ldquo;Allow IPv4\u0026rdquo;. Save the peer configuration by clicking \u0026ldquo;Save Peer\u0026rdquo;.\nFinal peer configuration should look something like this. Getting your IP address from AirVPN Once the above steps are done, pfSense would have connected to AirVPN through WireGuard. But we wouldn\u0026rsquo;t be able to use it yet as we haven\u0026rsquo;t configured the Interface yet. Before we proceed for Interface configuration, let\u0026rsquo;s first get the IP address.\nGo to https://airvpn.org/sessions/ Make note of your VPN IPv4 address.\nNote: As far as I observed, AirVPN does not change the ip address after the first assignment. This behaviour can change in the future and I will update this guide if so. I haven't found any other way to get the IP address of the Wireguard connection.\nInterface Assignments When you created a tunnel (following the steps above), you would see a new Interface in pfSense. On top bar, go to Interfaces \u0026gt; Assignments You will see a new interface at the bottom of the list, likely named \u0026ldquo;tun_wg0\u0026rdquo;\nClick \u0026ldquo;Add\u0026rdquo; and you see it assigned to an interface. Click on the interface link to take you to the configuration page.\nEnter a Description, say \u0026ldquo;AirVPN_WireGuard\u0026rdquo; In IPv4 Configuration Type, select \u0026ldquo;Static IPv4\u0026rdquo; In IPv4 Address: (use the ip address from above step) Select 32 as CIDR value IPv4 Upstream gateway: Click \u0026ldquo;Add a new gateway\u0026rdquo; In the popup, uncheck \u0026ldquo;Default gateway\u0026rdquo; Gateway name: AirVPN_WIREGUARD_GW Gateway IPv4: Same ip address from above step. Click \u0026ldquo;Add\u0026rdquo; The final configuration should look like this. NAT settings On top bar, go to Firewall \u0026gt; NAT \u0026gt; Outbound\nSelect Hybrid Outbound NAT rule generation Click \u0026ldquo;Add\u0026rdquo; Enter following details with right local ip address that you want to have VPN access to. Click Save. Firewall Rules On top bar, go to Firewall \u0026gt; Rules \u0026gt; LAN\nYou are not limited to LAN interface. If you have configured VLANs, you can use them as well.\nSet the Gateway as AirVPN_WIREGUARD_GW to the rules which want to use VPN.\nThat\u0026rsquo;s it, now you are done!\n1 2 The WireGuard implementation in AirVPN is not stable enough. My connection drops for 15-30 seconds every now and then. I wouldn\u0026#39;t recommend you to completely switch to WireGuard yet. ","date":"2022-02-07T16:56:03+05:30","permalink":"https://blog.dexome.com/post/airvpn-pfsense/","title":"AirVPN with Wireguard in pfSense"},{"content":"Let the first post in this blog be about setting up this blog.\nPre requisites: Azure account Custom domain (optional) Some knowhow on git and GitHub I am going to guide you with few links which has a good write-up on how to achieve this. I didn\u0026rsquo;t want to replicate the steps in this post as it may go outdated.\nThis site uses Hugo to generate static site generator. It\u0026rsquo;s pretty simple and easy to people with no prior blogging or technical experience.\nYou can follow the guide here to create a simple site in your localhost.\nAfter that, you will need to deploy your application to Azure, ideally Github. Follow the article here to deploy to Azure.\nFor configuring custom domain, you will need to purchase a domain. I liked to managing my domain with Cloudflare. So, I transferred my domain to Cloudflare and I liked this promise from Cloudflare\nCloudflare Promise\nFrom the price side it’s even simpler: we promise to never charge you anything more than the wholesale price each TLD charges. That’s true the first year, and it’s true every subsequent year. If you register your domain with Cloudflare Registrar you’ll always pay the wholesale price with no markup.\nOptional things to do: Create a logo for your blog. I used this Design Studio Logo Maker\nSelect good theme for your site from here\n","date":"2022-01-07T16:01:03+05:30","permalink":"https://blog.dexome.com/post/this-blog/","title":"This Blog"}]