Containers on Dexome

Sharing Files Between Containers with Different UIDs

Mon, 17 Aug 2026 00:00:00 +0530

I have multiple containers which need to write into the same media folders. Copyparty runs as UID 924, the arr applications run as UID 911 and Frigate uses UID/GID 914 on the host. They need to create, rename and delete each other’s files without running all services as the same user.

I faced three related issues:

Copyparty ran as UID 924 and shared media with the arr stack at UID 911.
Frigate had to initialize as root in its container while retaining host ownership as UID/GID 914.
A mergerfs view combined two branches whose copies of the same directory had different modes.

Fixing only the owner or group was not enough. I needed idmapped mounts, correct directory modes, application-specific chmod settings and inherited ACLs.

Why a writable file could not be replaced

For a process to replace movies/title/video.mkv, it needs write and execute on movies/title/. Write permission on video.mkv controls modification of the file’s contents; it does not grant permission to remove that directory entry.

Three inputs decide the normal Unix permission check:

UID/GID ownership chooses the owner, group, or other class.
The inode mode or access ACL says what that class may do.
The containing directory controls create, rename, and delete.

flowchart TB
		PROC[Radarr uid 911, media group] --> CLASS{Matching class?}
		CLASS -->|directory gid media| GROUP[Use group mode or ACL]
		GROUP -->|r-x on directory| DENY[Read and traverse, cannot replace]
		GROUP -->|rwx on directory| ALLOW[Create, rename, and delete]
		FILE[Writable target file] -. does not decide unlink .-> DENY

Ownership chooses a permission class; directory permissions decide whether another service can rename or delete an entry.

So the first check was the owner and mode of the parent directory, not only the movie file.

Copyparty rename blocked a Radarr import

Radarr failed while upgrading a movie:

`1`	`Access to the path '.../Run (2002)/Run (2002) SDTV.mp4' is denied`

The destination directory told the story:

`1`	`drwxr-sr-x 924 media Run (2002)/`

UID 924 was Copyparty. GID media was correct, and the setgid bit ensured new children inherited that group. But the group had r-x, not rwx. Radarr’s UID 911 could enter the directory and read the old file but could not unlink it.

I had set UMASK=002 on the container. Copyparty’s image did not honor that for this operation and produced a 0755 directory. The permanent fix was application-level creation policy:

1
2
3

flags:
	chmod_f: 664
	chmod_d: 775

I applied that policy to the Copyparty volumes shared with the arr stack, qBittorrent, and Picard. Existing entries needed a one-time repair because creation policy is not retroactive.

Fixing both mergerfs branches

My first repair changed the affected directory under the btank backing tree. The error persisted. The visible media path was a mergerfs union of btank and btwo, and the directory existed on both branches:

1
2
3

btank branch:  drwxrwsr-x 924 media
btwo branch:   drwxr-sr-x 924 media
merged view:   drwxr-sr-x

mergerfs surfaced metadata from the stale btwo copy. With default_permissions, the kernel enforced the mode visible through the union. The branch containing the media file was not necessarily the branch supplying the directory metadata.

flowchart TB
		VIEW["/mnt/merged-media/title"] --> PICK{Metadata selected by mergerfs}
		B1["/mnt/btank/@media/title: 2775"] --> PICK
		B2["/mnt/btwo/@media/title: 2755"] --> PICK
		PICK -->|stale branch wins| MODE[Visible mode 2755]
		MODE --> DENIED[Radarr group write denied]

Repairing only the branch with the file left a stale directory mode on the other mergerfs branch.

The correct repair swept every raw backing branch, never only the merged view:

1
2

find /mnt/btank/@media/movies /mnt/btwo/@media/movies \
	-type d ! -perm -2775 -exec chmod 2775 {} +

After both branch copies were corrected, Radarr deleted the old file and completed the import.

Frigate runs as root inside the container

Frigate’s official image starts through s6-overlay and needs container root for initialization. Forcing --user 914:914 broke startup. Running it rootful with a plain bind mount would create host root:root state, which would break the native rollback path and Copyparty access.

A per-bind idmapped mount solved ownership without changing the process:

`1`	`idmap=uids=914-0-1;gids=914-0-1`

Podman’s triplet is backing ID, mapped ID, length. Through that mount, host 914:914 appears as 0:0 to the container. Frigate can initialize as root and its files still land on btrfs and ZFS as host 914:914.

flowchart LR
		F[Frigate container root 0:0] --> M[Idmapped bind mount]
		M -->|uids 914-0-1, gids 914-0-1| DISK[Backing files 914:914]
		N[Native rollback process 914:914] --> DISK
		C[Copyparty uid 924 plus group 914] --> DISK

A per-bind idmap changes the ownership view for one mount; it does not recursively mutate the backing files.

This is different from the alternatives:

Mechanism	What it changes	Why I did or did not use it
`--user 914:914`	Process identity	Broke Frigate’s root-requiring init
`:U`	Recursively changes backing ownership	Mutates data and is expensive
Whole-container user namespace	Ownership view for the container	Complicated GPU and group mappings
Per-bind idmap	Ownership view for one mount	Preserved both image and host contracts

s6 reset the process umask

In a disposable directory, --umask=0002 produced directories at 0775 and files at 0664. Copyparty, running as UID 924 with supplementary GID 914, could create, rename, and delete them.

The real Frigate processes produced 2755 directories and 0644 files. s6-overlay had reset the live process umask to 0022 after Podman launched the container. A shell test that bypassed the supervisor had proved the kernel and idmap mechanics, not the application’s final runtime behavior.

The reliable check was the live process state:

`1`	`grep '^Umask:' /proc/PID/status`

Because the image did not preserve the desired umask, I used default ACLs on the recordings tree. Newly created directories inherit group write even when the process requests a stricter base mode. Copyparty also joins supplementary GID 914, and its own creation policy emits group-writable entries when it writes into the same tree.

Final permission setup

The working setup uses:

stable host ownership through Frigate’s per-bind idmap;
shared or supplementary groups selecting the intended permission class;
setgid directories preserving group ownership;
application chmod_f and chmod_d policy where the image ignored UMASK;
inherited default ACLs where s6 reset the process umask;
repairs performed on every mergerfs backing branch.

For each shared mount, I test read, create, modify, rename, delete, mkdir and rmdir using the real service UIDs and groups. I also test files created by the actual application, not only a shell started in the same image. This was important for Frigate because s6 changed the umask after Podman started the container.

If mergerfs is involved, check and repair every backing branch. The permissions shown through the merged path may come from a different branch than the file being modified.

Moving Traefik into a Podman Container

Thu, 11 Jun 2026 00:00:00 +0530

I was running Traefik as a native NixOS service. Over time, I had added proxy services to let it reach applications in Podman networks. Instead of keeping those extra proxies, I decided to run Traefik itself in Podman and attach it to the required networks.

The container started and its health check passed, but some routes returned 502 Bad Gateway. Requests from another VLAN timed out, and after recreating the container the ingress IP stopped responding.

There were three separate issues:

loopback stopped meaning the host;
a multi-homed network namespace chose the wrong return route;
the macvlan MAC changed while the gateway still remembered the old one.

The container network setup

Native Traefik had accumulated proxy shims to reach services spread across host processes, Podman bridges, and network containers. Moving Traefik itself into Podman let it join application networks directly and removed a layer of socket forwarding.

The resulting Traefik container joined three networks:

podman_network_vlan_100 for the ingress VIP 10.100.100.21;
podman_rproxy at 10.89.0.2 for host and internal backends;
an internet bridge at 172.28.0.12 for outbound access.

flowchart TB
	CLIENTS[Clients and OPNsense] -->|HTTPS to 10.100.100.21| VLAN[macvlan VLAN 100]
	VLAN --> T[Traefik 3.7.5]
	T -->|10.89.0.1 backend ports| HOST[podman_rproxy host gateway]
	HOST --> SERVICES[Netdata, Glances, Cockpit, go2rtc, Home Assistant]
	T -->|container DNS and updates| INET[172.28.0.0/24 Internet bridge]

Containerized Traefik is multi-homed: ingress, private backends, and Internet access are separate network legs.

The three networks worked, but the old Traefik configuration still assumed it was running in the host network namespace.

Host services returned 502

Several file-provider routes still targeted backends such as:

1
2

servers:
  - url: http://127.0.0.1:19999

That URL worked when Traefik was a host process. Inside the container, 127.0.0.1 referred to the Traefik container itself. Netdata, go2rtc, Cockpit, and other host services had not moved with it.

Services on the same Podman bridge continued to work. Only host services and systemd-nspawn backends failed, while the Traefik health check remained green.

I tested from both sides of the namespace boundary:

# Works from the NixOS host
curl -fsS http://10.89.0.1:8123/

# Initially failed from Traefik's namespace
podman exec traefik wget -qO- -T2 http://10.89.0.1:8123/

The second command showed the problem without involving DNS, TLS or OPNsense.

The fix had three parts:

Bind host services to the podman_rproxy gateway, 10.89.0.1, rather than relying on loopback.
Allow only the required backend ports on the podman_rproxy interface in the NixOS firewall.
Change Traefik’s backend URLs from 127.0.0.1 to 10.89.0.1.

For Netdata, I also widened its application-level allow list from localhost to the 10.89.0.0/24 proxy subnet. Opening the host firewall alone would not have overridden the service’s own filter.

flowchart TB
	subgraph NATIVE[Native Traefik]
		NT[Traefik process] -->|127.0.0.1| NS[Host service]
	end
	subgraph CONTAINER[Containerized Traefik]
		CT[Traefik container] -->|127.0.0.1| SELF[Traefik container loopback]
		CT -->|10.89.0.1| HS[Host service on podman_rproxy]
	end
	NT -. moved into container .-> CT

The same URL points at a different machine after Traefik crosses into a container namespace.

The 502s disappeared, but off-subnet clients still timed out.

Requests from other VLANs timed out

Traefik had more than one default route because netavark attached multiple networks. The route selected during one container creation was not guaranteed to be the same route selected after another.

Clients on VLAN 100 were on-link with the macvlan VIP, so replies used the connected route and worked. A client from another VLAN reached 10.100.100.21 through OPNsense, but Traefik’s reply followed the internet bridge’s default route. The request entered through the macvlan and the response tried to leave through a different gateway.

1
2

off-subnet request: client -> OPNsense -> Traefik macvlan
wrong reply:        Traefik -> Podman Internet bridge -> nowhere useful

Testing only from VLAN 100 made the setup look correct. I had to test from another VLAN and inspect the route inside Traefik’s network namespace.

Inside Traefik’s network namespace, ip route get <client-address> exposed the selected gateway. I installed an explicit lower-metric default route through the VLAN 100 gateway:

`1`	`ip route replace default via 10.100.100.1 metric 50`

The internet-bridge route remained at metric 100. This was applied by the NixOS unit after Podman created the namespace, so every recreation restored the same decision.

flowchart TB
	C[Client on another VLAN] -->|request| FW[OPNsense]
	FW -->|10.100.100.21| T[Traefik]
	T -. wrong default before fix .-> I[Internet bridge gateway]
	I -. dropped reply .-> C
	T -->|metric 50 after fix| FW
	FW -->|symmetric reply| C

Equal default routes made off-subnet replies nondeterministic; the lower-metric VLAN route restored symmetry.

After the route fix, off-subnet curls returned HTTP 200 and ip route get showed the macvlan gateway. Then a later container recreation broke ingress once more.

The macvlan MAC changed after recreation

The macvlan attachment had a static IP but no static MAC. Podman generated a new MAC when Traefik was recreated. OPNsense and the upstream switching path still associated 10.100.100.21 with the previous MAC.

The container health and route table still looked correct because the stale entry was on OPNsense and the switching path, not inside the container.

I pinned the attachment identity in the Podman network specification:

`1`	`"podman_network_vlan_100:ip=10.100.100.21,mac=02:42:0a:64:64:15"`

The locally administered 02:42 prefix avoids claiming a vendor identity. The remaining octets encode the private address, which makes the value easy to audit. More importantly, recreation no longer changes the layer-2 identity.

Checks after the migration

A reverse proxy health check proves only that Traefik can answer inside its own namespace. My final test matrix covered each boundary explicitly:

Test	What it proves
Curl backend from NixOS host	Backend process is listening
Curl backend from Traefik container	Host firewall, bind address, and app allow list work
Request ingress from VLAN 100	Macvlan listener and on-link return path work
Request ingress from a different VLAN	Routed return path uses the correct gateway
Inspect `ip route get` in the namespace	Route choice is deterministic
Recreate Traefik and repeat all tests	MAC and post-start route configuration persist

I also checked the gateway neighbor entry after recreation rather than waiting for a browser symptom.

Key points

Moving Traefik into a container changed more than the process manager. The native service could reach host loopback and used the host routing table. The container could use only its attached interfaces and had multiple default routes.

For a similar migration, check all backend bind addresses, test from inside the Traefik container, test ingress from another subnet and recreate the container once before considering it complete. A static macvlan IP should also have a stable MAC when upstream devices keep neighbor entries for it.

Moving My NixOS Containers from Docker to Podman

Mon, 08 Jun 2026 00:00:00 +0530

NixOS makes it simple to change the backend used by virtualisation.oci-containers:

`1`	`virtualisation.oci-containers.backend = "podman";`

In my setup, this changed 36 containers with custom networks, static IP addresses, shared network namespaces, health checks, GPU devices and bind mounts. Traefik, Netdata, Glances and Dozzle were also using the Docker API.

Most Docker options worked with Podman, but a few differences stopped containers from starting. Network recreation and systemd health checks caused another set of problems after the initial migration.

Why I moved to Podman

I wanted a runtime that fit the host’s existing systemd ownership model. Podman gave me generated podman-<name>.service units, native health actions, a rootful socket at /run/podman/podman.sock, and no long-running Docker daemon.

I intentionally did not enable dockerCompat. Traefik, Netdata, Glances, and Dozzle were pointed at Podman’s Docker-compatible API explicitly. Portainer and the autoheal sidecar were removed; Cockpit Podman and --health-on-failure=kill took their places.

flowchart LR
	NIX[NixOS OCI declarations] --> UNITS[Generated systemd units]
	UNITS -->|before| D[Docker containers]
	UNITS -->|after| P[Podman containers]
	API[Traefik, Netdata, Glances, Dozzle] -->|before| DS["/var/run/docker.sock"]
	API -->|after| PS["/run/podman/podman.sock"]
	NET[Declared networks] --> NAV[netavark and aardvark-dns]
	NAV --> P

The runtime changed under a declarative NixOS service graph; every explicit Docker unit and socket edge had to move with it.

The migration began with mechanical changes: rename unit references from docker-* to podman-*, change supplementary group membership from docker to podman, and replace socket paths. Then the behavioral differences appeared.

`--health-on-failure` needs a health check

I replaced autoheal with Podman’s native unhealthy action:

`1`	`--health-on-failure=kill`

That option is valid only when the container has an explicit health command or the image contains a baked HEALTHCHECK. Docker had tolerated the old setup. Podman refused to create affected containers with exit 125:

`1`	`Error: cannot set on-failure action to kill without a health check`

Documentation and upstream Dockerfiles were not reliable enough because the pinned image was the thing actually being executed. This became the authoritative test:

1
2

podman image inspect IMAGE \
  --format '{{if .Config.Healthcheck}}HAS{{else}}NONE{{end}}'

Only Plex had the baked health check I expected. Images for Dozzle, Gluetun, qBittorrent, Stirling PDF, Gogs, MeTube, and several Immich components did not. For those I either added a real --health-cmd or removed the unhealthy action.

Two more parser differences stopped containers before their applications even started:

Docker-era option	Podman result	Replacement
`--tmpfs=/transcode:uid=912,gid=912`	Unknown mount option	`mode=1777` where appropriate
`--network=name=NET,ip=IP`	Comma split into two network names	`--network=NET:ip=IP`

These errors were visible in the generated systemd unit logs and were simple to reproduce before starting the application.

Transient health checks caused exit code 4

Podman implements periodic health checks using transient systemd services. On the first activation, containers were pulling images and warming up while these checks began to run. Some transient podman healthcheck run <id> services returned non-zero during health_status=starting.

switch-to-configuration saw failed units and returned exit code 4. A minute later, the declared network unit had succeeded, every container was healthy, and systemctl --failed was empty.

sequenceDiagram
	participant A as NixOS activation
	participant S as systemd
	participant C as Podman container
	A->>S: restart generated units
	S->>C: start container
	C-->>S: health_status=starting
	S->>C: transient healthcheck
	C-->>S: non-zero while warming
	S-->>A: failed transient unit, exit 4
	C-->>S: health_status=healthy
	Note over S,C: Final state is healthy despite activation result

An activation-time health probe can fail while the final steady state becomes healthy.

Exit code 4 should not be ignored, but it can be caused by a health check which has already recovered. I check the failed units, network service and current container state separately:

1
2
3

systemctl --failed --no-pager
systemctl status podman-networks.service --no-pager --full
podman ps --format 'table {{.Names}}\t{{.Status}}'

A transient probe that has already cleared is different from a network reconciler that is still failed.

Stopped containers are still attached to networks

I wrote a declarative network reconciler so changes to subnet, gateway, or interface name would recreate a drifted Podman network. Container units were partOf the network service, so restarting the network stopped its users first.

The assumption was plausible and wrong. A stopped container remains associated with its network. The reconciler ran:

`1`	`podman network rm podman_network_rproxy`

Podman correctly refused:

`1`	`network is being used`

The drift path needed podman network rm -f. Force removal disconnected and removed the stopped container records; systemd recreated them after the network returned.

Network recreation cleared all IPAM leases

The force-removal fix exposed the next bug. Recreating the proxy bridge erased netavark’s IPAM leases. A dynamic container started first and received 10.89.0.2, the static address reserved in configuration for Traefik.

Traefik then looped with:

`1`	`IPAM error: requested ip address 10.89.0.2 is already allocated`

The old network had worked only because its historical lease happened to keep .2 occupied by the intended container. Start order became visible after a cold reconstruction.

I separated the static and dynamic address ranges:

subnet = "10.89.0.0/24";
gateway = "10.89.0.1";
ipRange = "10.89.0.128/25";
leaseRange = "10.89.0.129-10.89.0.255";

Addresses .2 through .127 are available for deliberate static assignments; dynamic leases begin at .129. I verified the expanded lease range with a disposable network before trusting it in the drift comparison.

flowchart TB
	R[Recreate 10.89.0.0/24 network] --> EMPTY[IPAM lease table is empty]
	EMPTY -->|without range separation| DYN[Dynamic backend starts first and takes .2]
	DYN --> FAIL[Traefik static .2 fails]
	EMPTY -->|dynamic pool .129-.255| HIGH[Backend receives a high address]
	HIGH --> OK[Traefik always claims static .2]

A network rebuild forgets historical leases. Reserving separate static and dynamic ranges removes start-order dependence.

NixOS firewall reload broke aardvark-dns

aardvark-dns listens on each netavark bridge gateway. Netavark inserted rules allowing DNS, but the NixOS firewall reload flushed those runtime-generated rules. Existing internal bridges were not recreated afterward, so their rules did not return.

Containers began timing out on names served by the bridge gateway. OAuth2 Proxy and Immich crash-looped; containers with explicit DNS servers were unaffected.

The fix was to make bridge DNS part of the host firewall declaration. My bridge interfaces deliberately retain a podman prefix, so the NixOS podman+ interface wildcard can allow TCP and UDP port 53 after every reload.

This incident also taught me that --internal does not mean “DNS-free.” An internal bridge may need its embedded resolver for other names on that same private network even when it has no Internet route.

Checks I would use for another migration

I would split validation into four layers:

Inspect every pinned image for a baked health check instead of relying on its documentation.
Create every container once to find unsupported option formats.
Delete and recreate the declared networks so the test starts with empty IPAM state.
Reload the NixOS firewall and check DNS from an internal bridge.
Reboot and make sure the same containers and static IPs return without relying on the old start order.

Podman has been working well after these fixes. The container applications did not need changes; most of the migration work was around image metadata, command line parsing, systemd health checks and netavark state.