Linux on Dexome

Sharing Files Between Containers with Different UIDs

Mon, 17 Aug 2026 00:00:00 +0530

I have multiple containers which need to write into the same media folders. Copyparty runs as UID 924, the arr applications run as UID 911 and Frigate uses UID/GID 914 on the host. They need to create, rename and delete each other’s files without running all services as the same user.

I faced three related issues:

Copyparty ran as UID 924 and shared media with the arr stack at UID 911.
Frigate had to initialize as root in its container while retaining host ownership as UID/GID 914.
A mergerfs view combined two branches whose copies of the same directory had different modes.

Fixing only the owner or group was not enough. I needed idmapped mounts, correct directory modes, application-specific chmod settings and inherited ACLs.

Why a writable file could not be replaced

For a process to replace movies/title/video.mkv, it needs write and execute on movies/title/. Write permission on video.mkv controls modification of the file’s contents; it does not grant permission to remove that directory entry.

Three inputs decide the normal Unix permission check:

UID/GID ownership chooses the owner, group, or other class.
The inode mode or access ACL says what that class may do.
The containing directory controls create, rename, and delete.

flowchart TB
		PROC[Radarr uid 911, media group] --> CLASS{Matching class?}
		CLASS -->|directory gid media| GROUP[Use group mode or ACL]
		GROUP -->|r-x on directory| DENY[Read and traverse, cannot replace]
		GROUP -->|rwx on directory| ALLOW[Create, rename, and delete]
		FILE[Writable target file] -. does not decide unlink .-> DENY

Ownership chooses a permission class; directory permissions decide whether another service can rename or delete an entry.

So the first check was the owner and mode of the parent directory, not only the movie file.

Copyparty rename blocked a Radarr import

Radarr failed while upgrading a movie:

`1`	`Access to the path '.../Run (2002)/Run (2002) SDTV.mp4' is denied`

The destination directory told the story:

`1`	`drwxr-sr-x 924 media Run (2002)/`

UID 924 was Copyparty. GID media was correct, and the setgid bit ensured new children inherited that group. But the group had r-x, not rwx. Radarr’s UID 911 could enter the directory and read the old file but could not unlink it.

I had set UMASK=002 on the container. Copyparty’s image did not honor that for this operation and produced a 0755 directory. The permanent fix was application-level creation policy:

1
2
3

flags:
	chmod_f: 664
	chmod_d: 775

I applied that policy to the Copyparty volumes shared with the arr stack, qBittorrent, and Picard. Existing entries needed a one-time repair because creation policy is not retroactive.

Fixing both mergerfs branches

My first repair changed the affected directory under the btank backing tree. The error persisted. The visible media path was a mergerfs union of btank and btwo, and the directory existed on both branches:

1
2
3

btank branch:  drwxrwsr-x 924 media
btwo branch:   drwxr-sr-x 924 media
merged view:   drwxr-sr-x

mergerfs surfaced metadata from the stale btwo copy. With default_permissions, the kernel enforced the mode visible through the union. The branch containing the media file was not necessarily the branch supplying the directory metadata.

flowchart TB
		VIEW["/mnt/merged-media/title"] --> PICK{Metadata selected by mergerfs}
		B1["/mnt/btank/@media/title: 2775"] --> PICK
		B2["/mnt/btwo/@media/title: 2755"] --> PICK
		PICK -->|stale branch wins| MODE[Visible mode 2755]
		MODE --> DENIED[Radarr group write denied]

Repairing only the branch with the file left a stale directory mode on the other mergerfs branch.

The correct repair swept every raw backing branch, never only the merged view:

1
2

find /mnt/btank/@media/movies /mnt/btwo/@media/movies \
	-type d ! -perm -2775 -exec chmod 2775 {} +

After both branch copies were corrected, Radarr deleted the old file and completed the import.

Frigate runs as root inside the container

Frigate’s official image starts through s6-overlay and needs container root for initialization. Forcing --user 914:914 broke startup. Running it rootful with a plain bind mount would create host root:root state, which would break the native rollback path and Copyparty access.

A per-bind idmapped mount solved ownership without changing the process:

`1`	`idmap=uids=914-0-1;gids=914-0-1`

Podman’s triplet is backing ID, mapped ID, length. Through that mount, host 914:914 appears as 0:0 to the container. Frigate can initialize as root and its files still land on btrfs and ZFS as host 914:914.

flowchart LR
		F[Frigate container root 0:0] --> M[Idmapped bind mount]
		M -->|uids 914-0-1, gids 914-0-1| DISK[Backing files 914:914]
		N[Native rollback process 914:914] --> DISK
		C[Copyparty uid 924 plus group 914] --> DISK

A per-bind idmap changes the ownership view for one mount; it does not recursively mutate the backing files.

This is different from the alternatives:

Mechanism	What it changes	Why I did or did not use it
`--user 914:914`	Process identity	Broke Frigate’s root-requiring init
`:U`	Recursively changes backing ownership	Mutates data and is expensive
Whole-container user namespace	Ownership view for the container	Complicated GPU and group mappings
Per-bind idmap	Ownership view for one mount	Preserved both image and host contracts

s6 reset the process umask

In a disposable directory, --umask=0002 produced directories at 0775 and files at 0664. Copyparty, running as UID 924 with supplementary GID 914, could create, rename, and delete them.

The real Frigate processes produced 2755 directories and 0644 files. s6-overlay had reset the live process umask to 0022 after Podman launched the container. A shell test that bypassed the supervisor had proved the kernel and idmap mechanics, not the application’s final runtime behavior.

The reliable check was the live process state:

`1`	`grep '^Umask:' /proc/PID/status`

Because the image did not preserve the desired umask, I used default ACLs on the recordings tree. Newly created directories inherit group write even when the process requests a stricter base mode. Copyparty also joins supplementary GID 914, and its own creation policy emits group-writable entries when it writes into the same tree.

Final permission setup

The working setup uses:

stable host ownership through Frigate’s per-bind idmap;
shared or supplementary groups selecting the intended permission class;
setgid directories preserving group ownership;
application chmod_f and chmod_d policy where the image ignored UMASK;
inherited default ACLs where s6 reset the process umask;
repairs performed on every mergerfs backing branch.

For each shared mount, I test read, create, modify, rename, delete, mkdir and rmdir using the real service UIDs and groups. I also test files created by the actual application, not only a shell started in the same image. This was important for Frigate because s6 changed the umask after Podman started the container.

If mergerfs is involved, check and repair every backing branch. The permissions shown through the merged path may come from a different branch than the file being modified.

Moving My NixOS Network to systemd-networkd

Thu, 18 Jun 2026 00:00:00 +0530

My NixOS server heavymetal already had multiple VLANs, static IP addresses and macvlans used by containers. They were created using startup scripts and worked, but restarts and NixOS activations did not always recreate them in the correct order.

I first moved the same setup to systemd-networkd. Later, when I needed to attach VM tap interfaces to the VLANs, I added per-VLAN bridges and finally replaced them with one VLAN-aware bridge.

The migration happened in three steps:

replace imperative networking with networkd units;
move host addresses from VLAN devices onto stable per-VLAN bridges;
consolidate those bridges into a single VLAN-aware trunk.

I did not apply the bridge migrations live because they moved the same management IP used by SSH. Those changes were staged for boot with console access available.

flowchart TB
	S["Scripted VLAN and
macvlan devices"] --> N["systemd-networkd owns
the same topology"]
	N -->|Parent restart exposes child lifecycle| P["Per-VLAN bridges
brvlan20 / brvlan30 / brvlan100"]
	P -->|Stable parents enable VM taps| T["VLAN-aware bridge
brtrunk"]

The network evolved in three controlled steps rather than jumping directly from scripts to the final VLAN-aware bridge.

The old startup scripts

The old system created VLAN and macvlan devices through a mixture of higher-level configuration and startup services. Conceptually it did this:

ip link add link enp5s0 name enp5s0.100 type vlan id 100
ip address add 10.100.100.20/24 dev enp5s0.100
ip link set enp5s0.100 up

ip link add link enp5s0.100 name tfkshim type macvlan mode bridge
ip link set tfkshim up

Imperative commands are easy to prototype, but they hide ownership questions:

Which service owns the VLAN device?
What should restart when the parent disappears?
Is the address ready before a dependent service binds it?
What removes stale devices after a failed partial run?
Does a config reload destroy children created by another service?

The scripts encoded answers in execution order rather than in the topology.

Recreate the same network with networkd

I moved each concern into an explicit networkd object: VLAN .netdev units, matching .network units for addresses and routes, and dependencies for services that needed those links.

On NixOS, a simplified VLAN looks like this:

systemd.network.netdevs."10-vlan20" = {
	netdevConfig = {
		Name = "vlan20";
		Kind = "vlan";
	};
	vlanConfig.Id = 20;
};

systemd.network.networks."40-vlan20" = {
	matchConfig.Name = "vlan20";
	address = [ "10.20.0.10/24" ];
	routes = [ { Gateway = "10.20.0.1"; } ];
	networkConfig.RequiredForOnline = "routable";
};

The exact syntax is distribution-specific; the systemd concepts are not. A .netdev creates a virtual device. A .network matches a device and assigns addresses, routes, VLAN membership, bridge membership, and online-state requirements.

The important part was preserving topology exactly. This was not the moment to rename every interface or collapse five networks into one bridge. First I needed networkd to reproduce the working system.

systemd-resolved broke container DNS

Enabling networkd on NixOS also enabled systemd-resolved through a distribution default. The host continued to resolve names, so the change initially looked healthy. But /etc/resolv.conf now pointed at the loopback stub:

`1`	`nameserver 127.0.0.53`

Containers copied that file into their own network namespaces. From inside a container, 127.0.0.53 meant the container itself, not the host’s resolved service. External name resolution failed even though host DNS worked.

I explicitly disabled resolved and retained a static resolver address reachable from both the host and containers. Running resolved would also have been valid if the container DNS path had been designed for it. The failure came from changing resolver architecture as an accidental side effect of changing interface management.

The check I added was simple:

1
2
3

cat /etc/resolv.conf
podman exec <container> cat /etc/resolv.conf
podman exec <container> getent hosts example.com

A host-only lookup is not enough after a network-manager migration.

Restarting a VLAN removed its macvlan children

An older deployment had already revealed a more disruptive lifecycle problem. When the service owning a VLAN netdev restarted, it deleted and recreated the parent device. Linux also deleted every macvlan child attached to that parent.

The container runtime’s database still believed the container was attached. The actual network namespace contained only loopback and an unrelated bridge. The reverse proxy logged that its interface had been removed and lost its virtual address.

The journal made the sequence visible:

VLAN netdev service stopped
parent link deleted
macvlan child removed by kernel
container remains running without expected interface

sequenceDiagram
	participant Apply as NixOS activation
	participant Parent as enp5s0.100 netdev
	participant Kernel as Linux kernel
	participant Child as tfkshim macvlan
	participant Podman as Traefik container

	Apply->>Parent: Stop and delete VLAN netdev
	Parent->>Kernel: ip link del enp5s0.100
	Kernel--xChild: Delete every macvlan child
	Child--xPodman: eth0 removed from namespace
	Note over Podman: Container keeps running and DB still says attached
	Apply->>Parent: Recreate VLAN netdev
	Note over Parent,Podman: Parent returns but child does not

Deleting and recreating the VLAN parent also deleted the live macvlan child, while Podman's stored attachment remained stale.

At first I coupled the container-network service lifecycle to the parent netdev service: if the parent restarted, the container networks and their consumers restarted and reattached. That repaired the immediate inconsistency.

It also made the weakness of the topology clear. Long-lived workloads were attached directly to a device that configuration reconciliation was allowed to destroy.

Adding one bridge per VLAN

I needed the same VLANs to be shared with ThingsHQ microVM tap devices. A macvlan parent cannot serve that role cleanly, so I created always-on host bridges such as brvlan20 and brvlan100.

The VLAN uplink became an addressless bridge port. The host address moved to the bridge. Containers and VMs attached to the bridge rather than directly to the VLAN netdev.

A simplified networkd definition looks like:

systemd.network.netdevs."10-br-vlan20" = {
	netdevConfig = {
		Name = "br-vlan20";
		Kind = "bridge";
	};
};

systemd.network.networks."40-vlan20" = {
	matchConfig.Name = "vlan20";
	networkConfig.Bridge = "br-vlan20";
	linkConfig.RequiredForOnline = "enslaved";
};

systemd.network.networks."40-br-vlan20" = {
	matchConfig.Name = "br-vlan20";
	address = [ "10.20.0.10/24" ];
	routes = [ { Gateway = "10.20.0.1"; } ];
	networkConfig.RequiredForOnline = "routable";
};

This transition could not be safely applied live. A macvlan cannot be re-parented in place. Networkd could not enslave the VLAN uplink to the new bridge while the live macvlan child still depended on it, and deleting the child would remove the remote management path carried by the reverse proxy.

Instead of a live configuration switch, I staged the new boot generation and rebooted through a console-backed maintenance path. At boot, every device was created in the new topology from an empty state. The previous generation remained selectable in the boot menu if the new management address did not come up.

That was not excessive caution. “Same IP, different owning device” is still a remote access migration.

Moving to one VLAN-aware bridge

Per-VLAN bridges solved the VM-sharing problem, but the host eventually needed a larger trunk topology. Five bridges and five VLAN uplinks repeated the same structure and made VM trunk attachment awkward.

The next design used a single bridge named brtrunk with VLAN filtering.

flowchart LR
	SWITCH[Omada trunk] ==>|tagged VLANs| NIC[enp5s0]
	NIC --> BR[brtrunk VLAN-aware bridge]
	BR --> ADMIN[brtrunk.100: host admin 10.100.100.20]
	BR --> THINGS[brtrunk.20: Things services]
	BR --> QUANTUM[brtrunk.30: VPN services]
	BR --> VMS[OPNsense and ThingsHQ VM tap ports]
	BR --> ISOLATED[VLAN 67 sync and VLAN 999 WAN transport: no host L3]

In the final design, enp5s0 is a bridge port; brtrunk owns VLAN filtering, while host L3 exists only on selected VLAN interfaces.

In networkd terms, the bridge enables VLAN filtering and the physical interface is enslaved as a trunk port:

systemd.network.netdevs."10-br-trunk" = {
	netdevConfig = {
		Name = "brtrunk";
		Kind = "bridge";
	};
	bridgeConfig = {
		VLANFiltering = true;
		DefaultPVID = 1;
	};
};

systemd.network.networks."40-physical-trunk" = {
	matchConfig.Name = "enp5s0";
	networkConfig.Bridge = "brtrunk";
	linkConfig.RequiredForOnline = "enslaved";
};

The host’s management address moved from a per-VLAN bridge onto a VLAN interface of the trunk bridge. Some transport VLANs existed only inside the bridge and had no host-layer address at all.

Again, I staged the configuration for the next boot rather than attempting to replace the bridge beneath an active SSH session. The migration changed the master of the physical NIC, removed several bridges, recreated VLAN interfaces, and moved the default route. A reboot was the deterministic path.

Conflicting forwarding sysctls

After one deployment, host-to-container routing failed even though net.ipv4.ip_forward appeared in the configuration.

Linux exposes closely related forwarding sysctls:

1
2

net.ipv4.ip_forward
net.ipv4.conf.all.forwarding

Modules had written both aliases with conflicting values. The canonical per-family setting remained zero at runtime, leaving forwarding disabled. I made both values explicit:

boot.kernel.sysctl = {
	"net.ipv4.ip_forward" = 1;
	"net.ipv4.conf.all.forwarding" = 1;
	"net.ipv6.conf.all.forwarding" = 1;
};

Both names refer to closely related kernel settings and another module can write one after the other. So I check the runtime values after activation:

1
2
3

sysctl net.ipv4.ip_forward
sysctl net.ipv4.conf.all.forwarding
sysctl net.ipv6.conf.all.forwarding

The kernel is the final source of truth.

Source-only addresses and automatic routes

The host also had secondary interfaces used only to originate scans or reach macvlan workloads. Those addresses must not automatically install connected prefix routes if another interface owns the real return path.

I set AddPrefixRoute=false on those source-only legs and kept reverse-path filtering loose where asymmetric routes were intentional. Otherwise Linux could prefer the newly connected /24, send replies out the wrong interface, and turn a harmless scan address into an SSH lockout.

This is a niche detail with a broad principle: every new address can also create a route. During migrations, compare the routing table, not just the address list.

Checks after each migration

After each stage I checked the system from the bottom up.

Link ownership

1
2
3

networkctl status
ip -br link
ip -br address

Is the physical NIC enslaved to the intended bridge?
Are VLAN devices attached to the intended parent?
Is each host address on the bridge/VLAN device, not the addressless port?

Routes and forwarding

1
2
3

ip route
ip -6 route
sysctl net.ipv4.conf.all.forwarding

Is there exactly one intended default route?
Did source-only addresses add unwanted connected routes?
Is forwarding enabled in runtime state?

Namespaces

Can a container resolve DNS?
Does its namespace contain every expected interface?
Can it reach both an internal backend and an external endpoint?
Does restarting the parent/network service reattach the child cleanly?

Reboot behavior

Does the host return on the management address without manual intervention?
Do services wait for the interface state they actually require?
Does a second ordinary configuration apply leave the topology intact?

Final setup

systemd-networkd did not make the network itself simple, but it made device ownership and startup dependencies visible. The order I would use again is:

reproduce the working topology declaratively;
observe the lifecycle failures that the old scripts concealed;
introduce stable bridge ownership;
consolidate only after the dependencies are understood;
stage non-live-reparentable changes for reboot with console rollback.

Do not assume a declarative network change is safe to apply over SSH. Moving a management IP from a VLAN to a bridge still removes and recreates interfaces. For those changes, I used a boot-time migration and kept the previous NixOS generation available from the console.

Moving My NixOS Containers from Docker to Podman

Mon, 08 Jun 2026 00:00:00 +0530

NixOS makes it simple to change the backend used by virtualisation.oci-containers:

`1`	`virtualisation.oci-containers.backend = "podman";`

In my setup, this changed 36 containers with custom networks, static IP addresses, shared network namespaces, health checks, GPU devices and bind mounts. Traefik, Netdata, Glances and Dozzle were also using the Docker API.

Most Docker options worked with Podman, but a few differences stopped containers from starting. Network recreation and systemd health checks caused another set of problems after the initial migration.

Why I moved to Podman

I wanted a runtime that fit the host’s existing systemd ownership model. Podman gave me generated podman-<name>.service units, native health actions, a rootful socket at /run/podman/podman.sock, and no long-running Docker daemon.

I intentionally did not enable dockerCompat. Traefik, Netdata, Glances, and Dozzle were pointed at Podman’s Docker-compatible API explicitly. Portainer and the autoheal sidecar were removed; Cockpit Podman and --health-on-failure=kill took their places.

flowchart LR
	NIX[NixOS OCI declarations] --> UNITS[Generated systemd units]
	UNITS -->|before| D[Docker containers]
	UNITS -->|after| P[Podman containers]
	API[Traefik, Netdata, Glances, Dozzle] -->|before| DS["/var/run/docker.sock"]
	API -->|after| PS["/run/podman/podman.sock"]
	NET[Declared networks] --> NAV[netavark and aardvark-dns]
	NAV --> P

The runtime changed under a declarative NixOS service graph; every explicit Docker unit and socket edge had to move with it.

The migration began with mechanical changes: rename unit references from docker-* to podman-*, change supplementary group membership from docker to podman, and replace socket paths. Then the behavioral differences appeared.

`--health-on-failure` needs a health check

I replaced autoheal with Podman’s native unhealthy action:

`1`	`--health-on-failure=kill`

That option is valid only when the container has an explicit health command or the image contains a baked HEALTHCHECK. Docker had tolerated the old setup. Podman refused to create affected containers with exit 125:

`1`	`Error: cannot set on-failure action to kill without a health check`

Documentation and upstream Dockerfiles were not reliable enough because the pinned image was the thing actually being executed. This became the authoritative test:

1
2

podman image inspect IMAGE \
  --format '{{if .Config.Healthcheck}}HAS{{else}}NONE{{end}}'

Only Plex had the baked health check I expected. Images for Dozzle, Gluetun, qBittorrent, Stirling PDF, Gogs, MeTube, and several Immich components did not. For those I either added a real --health-cmd or removed the unhealthy action.

Two more parser differences stopped containers before their applications even started:

Docker-era option	Podman result	Replacement
`--tmpfs=/transcode:uid=912,gid=912`	Unknown mount option	`mode=1777` where appropriate
`--network=name=NET,ip=IP`	Comma split into two network names	`--network=NET:ip=IP`

These errors were visible in the generated systemd unit logs and were simple to reproduce before starting the application.

Transient health checks caused exit code 4

Podman implements periodic health checks using transient systemd services. On the first activation, containers were pulling images and warming up while these checks began to run. Some transient podman healthcheck run <id> services returned non-zero during health_status=starting.

switch-to-configuration saw failed units and returned exit code 4. A minute later, the declared network unit had succeeded, every container was healthy, and systemctl --failed was empty.

sequenceDiagram
	participant A as NixOS activation
	participant S as systemd
	participant C as Podman container
	A->>S: restart generated units
	S->>C: start container
	C-->>S: health_status=starting
	S->>C: transient healthcheck
	C-->>S: non-zero while warming
	S-->>A: failed transient unit, exit 4
	C-->>S: health_status=healthy
	Note over S,C: Final state is healthy despite activation result

An activation-time health probe can fail while the final steady state becomes healthy.

Exit code 4 should not be ignored, but it can be caused by a health check which has already recovered. I check the failed units, network service and current container state separately:

1
2
3

systemctl --failed --no-pager
systemctl status podman-networks.service --no-pager --full
podman ps --format 'table {{.Names}}\t{{.Status}}'

A transient probe that has already cleared is different from a network reconciler that is still failed.

Stopped containers are still attached to networks

I wrote a declarative network reconciler so changes to subnet, gateway, or interface name would recreate a drifted Podman network. Container units were partOf the network service, so restarting the network stopped its users first.

The assumption was plausible and wrong. A stopped container remains associated with its network. The reconciler ran:

`1`	`podman network rm podman_network_rproxy`

Podman correctly refused:

`1`	`network is being used`

The drift path needed podman network rm -f. Force removal disconnected and removed the stopped container records; systemd recreated them after the network returned.

Network recreation cleared all IPAM leases

The force-removal fix exposed the next bug. Recreating the proxy bridge erased netavark’s IPAM leases. A dynamic container started first and received 10.89.0.2, the static address reserved in configuration for Traefik.

Traefik then looped with:

`1`	`IPAM error: requested ip address 10.89.0.2 is already allocated`

The old network had worked only because its historical lease happened to keep .2 occupied by the intended container. Start order became visible after a cold reconstruction.

I separated the static and dynamic address ranges:

subnet = "10.89.0.0/24";
gateway = "10.89.0.1";
ipRange = "10.89.0.128/25";
leaseRange = "10.89.0.129-10.89.0.255";

Addresses .2 through .127 are available for deliberate static assignments; dynamic leases begin at .129. I verified the expanded lease range with a disposable network before trusting it in the drift comparison.

flowchart TB
	R[Recreate 10.89.0.0/24 network] --> EMPTY[IPAM lease table is empty]
	EMPTY -->|without range separation| DYN[Dynamic backend starts first and takes .2]
	DYN --> FAIL[Traefik static .2 fails]
	EMPTY -->|dynamic pool .129-.255| HIGH[Backend receives a high address]
	HIGH --> OK[Traefik always claims static .2]

A network rebuild forgets historical leases. Reserving separate static and dynamic ranges removes start-order dependence.

NixOS firewall reload broke aardvark-dns

aardvark-dns listens on each netavark bridge gateway. Netavark inserted rules allowing DNS, but the NixOS firewall reload flushed those runtime-generated rules. Existing internal bridges were not recreated afterward, so their rules did not return.

Containers began timing out on names served by the bridge gateway. OAuth2 Proxy and Immich crash-looped; containers with explicit DNS servers were unaffected.

The fix was to make bridge DNS part of the host firewall declaration. My bridge interfaces deliberately retain a podman prefix, so the NixOS podman+ interface wildcard can allow TCP and UDP port 53 after every reload.

This incident also taught me that --internal does not mean “DNS-free.” An internal bridge may need its embedded resolver for other names on that same private network even when it has no Internet route.

Checks I would use for another migration

I would split validation into four layers:

Inspect every pinned image for a baked health check instead of relying on its documentation.
Create every container once to find unsupported option formats.
Delete and recreate the declared networks so the test starts with empty IPAM state.
Reload the NixOS firewall and check DNS from an internal bridge.
Reboot and make sure the same containers and static IPs return without relying on the old start order.

Podman has been working well after these fixes. The container applications did not need changes; most of the migration work was around image metadata, command line parsing, systemd health checks and netavark state.