Networking on Dexome

Setting up OPNsense HA with CARP and pfsync

Tue, 04 Aug 2026 00:00:00 +0530

I run two OPNsense VMs, one on metalbox and another on heavymetal. I wanted the second firewall to take over the gateway, DNS, DHCP and existing connections when the first host was unavailable.

CARP itself was the easy part. The difficult parts were the services around it. When I first booted the backup firewall, the network reached around 160,000 packets per second and became unusable. Later, CARP moved the IP correctly but DNS stopped and existing TCP connections were lost.

Below is how I configured the full setup and the issues I found while testing it.

The HA setup

The finished design runs two OPNsense VMs on separate NixOS hypervisors. Node A runs on metalbox; node B runs on heavymetal. Clients keep using the familiar .1 gateways on each internal VLAN, now implemented as CARP VIPs. The nodes have real .6 and .7 management addresses, and a dedicated VLAN 67 sync link at 10.100.67.6/28 and 10.100.67.7/28.

flowchart TB
	CLIENTS[LAN, THINGS, QUANTUM and SERVERS clients] --> VIPS[CARP gateway VIPs ending in .1]
	INTERNET[PPPoE uplink] --> WANVIP[WAN CARP VIP 192.168.1.5]

	subgraph M[metalbox NixOS hypervisor]
		A[OPNsense A - preferred MASTER]
		AREAL[Real addresses ending in .6]
		A --- AREAL
	end

	subgraph H[heavymetal NixOS hypervisor]
		B[OPNsense B - BACKUP]
		BREAL[Real addresses ending in .7]
		B --- BREAL
	end

	VIPS --> A
	VIPS -. failover .-> B
	WANVIP --> A
	WANVIP -. failover .-> B
	A ==>|VLAN 67: XMLRPC, pfsync, Kea HA| B

The completed OPNsense HA topology separates client VIPs from node management and the VLAN 67 control plane.

There are four separate parts in this setup:

CARP moves gateway and service IP addresses between nodes.
Configuration sync keeps rules and services aligned.
State sync copies the firewall state table so established connections live.
Application-level HA makes services such as DHCP and mDNS behave correctly.

I configured and tested them one at a time.

Configure CARP on the first node

Before introducing a backup, I converted the existing firewall into a single CARP master. Its old gateway addresses became virtual IPs, while the firewall received separate real addresses for management. The WAN source-NAT rule also changed to use the WAN virtual IP, so outbound traffic would retain the same source after a failover.

This stage sounds redundant: why configure failover with only one node? Because it isolates the address migration from the redundancy problem. I could prove that clients still reached their familiar gateways, outbound NAT used the expected address, and management remained available before another system was allowed to advertise anything.

It also established a useful naming pattern:

one stable name for the firewall service, resolving to the virtual IP;
one node-specific name per firewall, resolving to its real address.

When HA itself is broken, the node-specific addresses are the way back in.

Booting the backup caused an mDNS storm

The second node started from a copy of the first node’s configuration. Its CARP advertisement priority was lower, so it should have stayed in BACKUP. Within roughly 30 to 60 seconds, however, the network flooded.

The timing made a split-brain theory persuasive. I checked the usual suspects:

shared CARP passwords matched;
virtual host IDs matched;
the master and backup had the intended advertisement skew;
multicast advertisements arrived on every VLAN;
the backup remained silent in BACKUP instead of advertising as another master.

These checks showed CARP was working as expected. I then captured the traffic causing the packet storm.

The breakthrough was to stop looking only at CARP packets and inspect the storm itself. Almost all of it was UDP port 5353: multicast DNS. Both node MAC addresses were flooding at similar rates.

The copied configuration had enabled an mDNS repeater on both firewalls. Each reflector received packets repeated by the other and reflected them again across the same interfaces. A small amount of multicast became an exponential loop.

sequenceDiagram
	participant Device as mDNS device
	participant A as OPNsense A reflector
	participant B as OPNsense B reflector
	participant LAN as Other VLANs

	Device->>A: Multicast query on UDP 5353
	A->>LAN: Reflect query
	LAN->>B: Reflected query arrives
	B->>LAN: Reflect it again
	LAN->>A: Re-reflected query arrives
	loop Exponential amplification
		A->>LAN: Reflect B's copy
		LAN->>B: Deliver A's copy
		B->>LAN: Reflect A's copy
		LAN->>A: Deliver B's copy
	end

OPNsense B inherited the active mDNS repeater configuration. Each node reflected the other's reflected packets until the LAN reached roughly 160,000 packets per second.

The firewall software had the correct feature for this situation: enable the repeater only while the node is CARP master. I configured the service identically on both nodes but enabled its CARP-aware failover mode. The backup retained the configuration without running the reflector until promotion.

The same check is needed for any service that broadcasts or reflects traffic. Copying the configuration to the backup can make both instances active at the same time.

Dedicated network between the firewalls

I added VLAN 67 between the firewalls for control traffic. It had no client gateway and no virtual IP. OPNsense A used 10.100.67.6/28, OPNsense B used 10.100.67.7/28, and a tightly scoped firewall rule allowed traffic only within that sync subnet.

The lack of a default pass rule on a newly assigned firewall interface was an early trap. Both addresses existed, but all layer-3 traffic was silently dropped until the sync-network rule was installed. Link state and correct addresses do not prove that the control plane can communicate.

The dedicated link carried config sync, pfsync, and DHCP peer communication. It also kept that traffic away from client VLANs and gave packet captures a much cleaner place to answer “did the peers actually talk?”

flowchart LR
	A[OPNsense A 10.100.67.6] ==>|XMLRPC: configuration A to B| B[OPNsense B 10.100.67.7]
	A <-->|pfsync: firewall states| B
	A <-->|Kea HA: leases and peer health| B

	CARP[CARP advertisements on client VLANs] -.-> A
	CARP -.-> B

CARP is only the address-ownership layer; three separate protocols cross VLAN 67 to preserve configuration, leases, and live connections.

Problems with configuration sync

The config-sync page looked straightforward: peer address, username, password, and a list of areas to synchronize. It failed for five independent reasons.

Management ports were different

One node exposed its GUI directly on the default HTTPS port. The other listened on a non-default local port behind a reverse proxy. Config sync contacted the peer’s real address, not the public proxy name, so both web services needed a reachable and explicitly matching port.

GUI was not listening on the sync interface

The backup’s web server listened only on its LAN interface. A TCP connection over the dedicated sync address therefore had nowhere to land. Adding the sync interface to the GUI’s listen scope fixed the transport without widening client access.

Certificate verification failed

The peers used a self-signed management certificate over a private, dedicated L2 link. Certificate verification failed before credentials were considered. In this topology I disabled peer verification for the sync call. A better option, when supported, is to give each node a certificate chaining to an internal CA.

The sync user needed another privilege

The user could open the HA configuration page but could not call the XMLRPC library. “High Availability” GUI access and XMLRPC execution were separate permissions. Granting only the narrowly required library privilege fixed the API call without turning the account into an administrator.

Changes were not pushed automatically

The largest conceptual surprise was that saving a configuration did not necessarily push it to the backup. The GUI’s explicit “synchronize all” action worked, but ordinary changes and infrastructure-as-code applies did not invoke it.

I added a post-apply hook that calls the firewall’s sync endpoint. GUI-only edits still require the operator to trigger synchronization, so the operating rule is simple: one node is authoritative; the backup is not a second place to edit.

The reusable lesson is to test config sync as an action, not as a checkbox. Change a harmless object on the primary, trigger the documented sync path, and prove it appears on the backup.

Kea DHCP hot standby

CARP moving the gateway does not automatically make the backup DHCP server safe. Two independent DHCP servers with copied configuration can both answer clients, while a master-only DHCP service can leave an availability gap during promotion.

I used Kea’s hot-standby mode over the sync network. The primary serves leases under normal conditions; the standby receives lease updates and takes over after it decides the partner is down.

The configuration failed when both peers used the same server name. Kea requires this-server-name to match one of the declared peer names. Explicitly writing the primary’s name into synchronized configuration made both nodes identify as the primary.

I gave the firewalls distinct hostnames, declared those as peer names, and left this-server-name empty so each node derives its identity locally. The node identity cannot be copied from primary to backup.

Hot standby also has a detection interval. In this case the standby waited about a minute before entering partner-down mode. Existing clients with leases stayed online; a new or renewing client might wait. That is not a broken failover, but it is part of the service-level objective and should be measured rather than assumed.

Unbound stopped answering after failover

With CARP working, the virtual gateway moved cleanly to the backup. Clients could route, but DNS queries to the gateway timed out.

The resolver had been configured to bind selected interfaces. On the backup, virtual IPs do not exist while it is in BACKUP. When the resolver configuration was regenerated, the absent virtual address disappeared from the generated listen set. CARP later promoted the node and created the address, but the resolver was not listening there.

The fix was to let the resolver listen on all local interfaces using its automatic interface behavior. Firewall policy, rather than a brittle application bind list, controlled which networks could reach port 53.

This is a broader HA pattern: applications should not permanently compile a list of addresses whose presence changes with ownership. Either bind wildcard/automatic and enforce access in the firewall, or use a service hook that reacts correctly to every promotion and demotion.

Keeping connections with pfsync

At this point a cable pull promoted the backup, moved the virtual IPs, and kept DNS and DHCP available. Yet established TCP sessions still died.

That was expected in retrospect. A stateful firewall tracks sequence numbers, NAT translations, timeouts, and policy decisions in a state table. The backup cannot infer that state from a newly arrived midstream packet.

I enabled pfsync on both nodes using explicit unicast peer addresses on the sync network. Configuring only one direction was not sufficient; each node needed to send and receive state because either could be active after maintenance or a second failure.

The acceptance test was deliberately mundane: start a long-lived transfer, pull the active firewall’s network cable, and watch whether the transfer continues. After bidirectional state sync was working, it did.

That test proved something a successful ping never could.

How I tested the failover

I ended with a matrix instead of a single “HA works” result:

Test	What it proves
Primary enters maintenance mode	Graceful CARP demotion and backup promotion
Primary loses its cable	Failure detection without operator help
Fail back to the original node	Prior master can safely resume ownership
DNS query during promotion	Resolver listens correctly on the new owner
New DHCP request after peer timeout	Standby has leases and enters partner-down
Long TCP transfer during cable pull	State and NAT translations were synchronized
Config change on primary	Explicit sync path updates the backup
Boot both nodes from cold	Broadcast services do not duplicate or loop

Running the matrix in both directions exposed assumptions hidden by testing only the preferred primary.

Final checks

Moving the virtual IP was only one of the checks. The setup was complete when:

there is exactly one active owner for each singleton or broadcast service;
both nodes carry compatible configuration without becoming competing writers;
DHCP and DNS continue according to measured failover timing;
established stateful connections survive a hard loss;
operators can reach either node through a real, node-specific address;
the behavior has been proven by removing the active node, not inferred from a green status page.

The main issue I found was assuming that copying a service configuration also made the service HA-aware. The mDNS repeater had to run only on the CARP master, Kea needed its own peer state and Unbound needed to listen on an address which appears only after promotion.

Finally, test with a real cable pull and a long-running connection. A successful ping after pressing the CARP maintenance button does not verify pfsync or the hard-failure path.

Moving My PPPoE WAN to VLAN 999

Thu, 30 Jul 2026 00:00:00 +0530

I wanted to move the WAN connection from a dedicated port on OPNsense to VLAN 999 through my Omada switch. This would allow both OPNsense nodes to reach the same ONT connection and was needed for the HA setup.

The PPPoE username, password and ONT were not changing. I had already configured the VLAN and expected the Internet to be down only for the time needed to move the cable and connect again.

It did not work that way. The first connection failed because I had left the old PPPoE session active at the ISP. Once I fixed that, the connection dropped again because the Omada switch was sending loop-detection packets towards the ONT.

This post covers both issues and the order I now use for a PPPoE cutover.

The setup

The old topology was direct. The Syrotech GPON ONT connected to the bare-metal OPNsense firewall’s igc0 interface. The new design moved that Ethernet segment onto VLAN 999 through a TP-Link Omada TL-SG2008P, then carried the tag over a trunk to OPNsense as igc2_vlan999.

flowchart TB
	BEFORE["Before
Syrotech ONT to OPNsense igc0 to PPPoE"]
	AFTER["After
Syrotech ONT to Omada TL-SG2008P to VLAN 999 to OPNsense igc2_vlan999 to PPPoE"]
	BEFORE -->|Move WAN onto the managed-switch trunk| AFTER

The WAN migration inserted an Omada switch and VLAN 999 between the Syrotech ONT and OPNsense.

The VLAN setup looked correct. A test device on the same path could reach the management IP of the ONT and packet captures showed the expected VLAN traffic. So I pulled the cable from the old port, moved it and enabled the new interface.

The connection never completed.

Packet captures showed PADI discovery frames leaving the firewall, but the expected PADO response did not come back. At other points the logs showed LCP negotiation repeating with new magic numbers, never reaching a usable session. It looked close enough to working to keep sending me down the wrong paths.

Checking the VLAN first

VLANs are an obvious suspect because they can fail silently. An access port can be untagged in the wrong VLAN, a trunk can omit the tag, or a native VLAN can consume traffic that was supposed to remain tagged.

I tested the transport independently of PPPoE. A client attached to the same logical path could reach the optical terminal’s management network. Captures on the firewall showed the expected Ethernet discovery frames on the expected VLAN. That did not prove every switch behavior was correct, but it proved that this was not simply a missing tag.

Checking the WAN MAC

Many providers bind service to a router or ONT MAC address, so MAC locking was the next theory. I tried preserving the previous WAN MAC and inspected the provider-facing frames. It was a reasonable theory, but it did not explain the behavior.

Some ISPs lock the connection to a MAC address, so this was still worth checking. There are three separate things which can cause a new router or port to fail:

Does the provider authenticate only with a username and password?
Does it also remember the client MAC?
Does it permit more than one simultaneous PPPoE session on the line?

All three look similar from OPNsense. In my case, the third one was the problem.

Closing the old PPPoE session

I had moved the cable while PPPoE was still active. That is not a clean session shutdown.

A graceful PPPoE disconnect sends a PADT (PPPoE Active Discovery Terminate), or terminates the link through PPP before removing the carrier. A cable pull sends neither. From the ISP access concentrator’s point of view, the old session can remain alive until its own timeout expires.

The provider allowed one session on the line. My new firewall interface was not replacing the old session; it was asking for a second one while the first still existed upstream.

That explained why waiting sometimes appeared to fix the problem. A long power-off interval gave the stale session time to age out. It also explained why rechecking credentials and VLAN tags accomplished nothing. The obstacle was not in my current configuration at all.

The reliable cutover was:

Leave the old WAN connected.
Disconnect or disable PPPoE on the old interface.
Verify that a PADT leaves the old interface.
Move the cable or change the VLAN assignment.
Enable PPPoE on the new interface.

sequenceDiagram
	participant Old as OPNsense igc0
	participant BRAS as ISP access concentrator
	participant New as OPNsense igc2_vlan999

	Old->>BRAS: PPPoE session established
	Note over Old,BRAS: Cable pulled without LCP Terminate or PADT
	New->>BRAS: PADI from VLAN 999
	BRAS--xNew: One-session policy blocks new session
	Note over BRAS: Old session remains until timeout
	Old->>BRAS: PADT during controlled retry
	BRAS-->>Old: Old session removed
	New->>BRAS: PADI then PADR
	BRAS-->>New: PADO then PADS
	Note over New,BRAS: New PPPoE session succeeds

The cable pull left the old session alive at the ISP; a PADT-first cutover explicitly closed it before the new interface dialled.

On a BSD-based firewall, a capture like this is enough to observe the discovery traffic:

`1`	`tcpdump -ni <wan-interface> -e ether proto 0x8863`

The exact command used to disconnect varies by platform. I prefer the firewall’s normal interface disconnect action because it follows the same control path the system uses in production. The capture is there to prove the shutdown emitted a termination frame before any cable moved.

With the old session closed first, the new interface authenticated normally.

The connection came up on VLAN 999 after this. But it did not stay up.

Omada loop detection on the ONT port

The PPPoE session later failed again, this time after it had already been stable. The pattern was different: discovery could be blocked outright, or an established session would disappear several minutes later.

The managed switch between the firewall and optical terminal had loop-detection features enabled. It injected control traffic into the provider-facing bridged segment: first spanning-tree-style frames, then vendor loopback-detection probes. On an ordinary LAN those features can be useful. On a transparent WAN transport, the ISP side is not an ordinary LAN and did not appreciate the extra frames.

The fix was not to disable protection across the whole switch. I disabled loopback control only on the port connected to the optical terminal. The WAN VLAN remained isolated and tagged exactly as before; only the switch’s active probing stopped.

flowchart TD
	MOVE[Move ONT cable from igc0 to VLAN 999] --> IMMEDIATE{When does it fail?}
	IMMEDIATE -->|Immediately| STALE[Old PPPoE session still held by ISP]
	STALE --> PADT[Disconnect old WAN and verify PADT]
	PADT --> UP[PPPoE connects on igc2_vlan999]
	IMMEDIATE -->|Minutes after connecting| PROBES[Omada loopback-control probes enter WAN segment]
	PROBES --> DROP[Established PPPoE session drops]
	DROP --> PORT[Disable Loopback Control on ONT-facing port]
	PORT --> STABLE[Session remains stable through soak and reboot]

Two independent faults occupied the same WAN path and failed on different timelines.

After that per-port change, the PPPoE session stayed up through the observation window and survived reboots of the relevant equipment.

Initially I thought the earlier fix was incomplete, but this was a separate problem on the same path:

the cutover failed immediately because the old PPPoE session was never closed;
the repaired connection later failed because the switch injected control traffic into the WAN bridge.

The migration steps I use now

Below is the order I now use when moving a live PPPoE WAN between ports, VLANs or firewall nodes.

Before the maintenance window

Record the working parent interface, VLAN ID, MTU, credentials source, and WAN MAC.
Confirm whether the provider documents MAC binding or single-session limits.
Preconfigure the destination interface without enabling it.
Verify the VLAN at layer 2 independently of PPPoE when possible.
Review the ISP-facing switch port for spanning tree, loop detection, LLDP, discovery, or other active control protocols.
Keep a management path that does not depend on the WAN being migrated.

During the cutover

old PPPoE connected
	│
	▼
disconnect old PPPoE cleanly
	│
	▼
observe PADT / termination
	│
	▼
move cable or VLAN ownership
	│
	▼
enable new PPPoE interface
	│
	▼
verify discovery, LCP, IPCP, route and DNS

Do not clone the MAC reflexively. Preserve it only when you know the provider requires it or when a controlled test demonstrates that it matters. Unnecessary MAC duplication is especially hazardous if old and new devices can be online at the same time.

After the session comes up

Do more than check that a public IP appeared:

inspect the PPP logs for successful LCP and IPCP negotiation;
confirm the default route uses the new interface;
test DNS through the normal client path;
verify the negotiated MTU with don’t-fragment pings;
keep the session under observation long enough to catch periodic switch probes;
reboot the switch or firewall if reboot survival is part of the acceptance criteria.

The distinction between “connected once” and “operationally complete” saved me from calling the migration finished before the loop-detection failure surfaced.

Key things to check

The immediate failure and the delayed failure had different causes. If PPPoE does not connect after moving a cable, first check whether the old session was closed properly. If it connects and drops later, check what the switch is sending on the WAN VLAN.

Also, a managed switch is not always transparent. Disable STP, loop detection or similar active features on the ONT-facing port unless you know the ISP path can handle them.

Most importantly, disconnect PPPoE before moving the cable and verify the PADT in a packet capture. Waiting for the ISP timeout also worked, but it made a simple cutover take much longer than needed.

Managing My Existing OPNsense Setup with OpenTofu

Tue, 07 Jul 2026 00:00:00 +0530

I wanted to manage my existing OPNsense configuration using OpenTofu. The firewall was already running DNS, DHCP, several VLANs, VPN connections and all the rules for my home network. Recreating everything from code was not an option.

I used the browningluke/opnsense provider and started with one Unbound DNS setting. After importing it, I did not continue until the plan showed:

`1`	`No changes. Your infrastructure matches the configuration.`

This worked for DNS, but the next step caused a DHCP outage because one provider default removed the gateway and DNS options from client leases. This post covers the order I used after that incident and the provider limitations I found.

Import first and make no changes

There are two separate things we may want to do during this migration:

represent the current system in code;
clean up the system while doing it.

I would not combine them.

The first goal is adoption. Its success criterion is boring: after import, the configuration describes the live object exactly enough that a plan proposes no change. Only after that baseline is stable should a separate change improve the object.

This matters most for routers and firewalls because the management path is one of the resources being changed. An incorrect web-server deployment can return a 500. An incorrect gateway, DHCP option, or anti-lockout rule can remove the path you need to repair it.

I used four gates for every subsystem:

1
2
3

inventory ──> import ──> zero-diff plan ──> one-object apply
		│             │             │                  │
		└── stop ─────┴── stop ─────┴── stop on drift ─┘

The first real apply was always deliberately small.

flowchart TB
	U[Unbound DNS: 66 objects] -->|zero-diff plan| K[Kea DHCP: 64 objects]
	K -->|client lease test| O[Omada VLANs, profiles, ports and SSIDs]
	O -->|controller no-op| C[Guest firewall canary]
	C -->|compiled pf order| F[Per-interface filter migration]
	F --> N[NAT and VPN resources]

The adoption moved outward from lower-risk DNS objects to connectivity-critical filters and NAT, with a stop gate after every phase.

Start with Unbound DNS

Resolver settings and host overrides were a good first target. Sixty-six Unbound objects were numerous enough to test import automation but less dangerous than rewriting the firewall ruleset.

The import revealed an important category of provider behavior: fields that exist on the appliance but not in the provider schema. One host override generated a reverse record, yet the provider did not expose that switch. Importing and planning the resource produced no change, so the appliance-only field survived.

That was acceptable. IaC coverage does not need to be 100 percent to be useful. It does need to be honest.

Another DNS feature exposed the opposite problem: the provider could read a blocklist setting but failed when writing it. Rather than force ownership, I left that feature GUI-managed and documented the boundary. A provider that cannot round-trip a field does not own that field.

The first phase ended with dozens of objects imported and a no-op plan. The point was not the count. It was proving the API credentials, import identifiers, schema, and state storage before touching client connectivity.

Kea DHCP and the `auto_collect` issue

Kea DHCP import covered 64 objects and looked equally clean until the first apply. Clients on THINGS and QUANTUM renewed and still received valid addresses, but they lost their default gateway and DNS server.

The provider exposed an auto_collect option. Its default was enabled, suggesting that the appliance would derive subnet options automatically. On this system it did not. Applying the resource removed the stored router, DNS, and NTP values.

A simplified version of the dangerous assumption looked like this:

resource "firewall_dhcp_subnet" "clients" {
	subnet       = "10.20.0.0/24"
	auto_collect = true
}

The repaired declaration made every client-visible option explicit. This is a simplified version of the QUANTUM subnet, whose gateway and resolver are 10.100.30.1:

resource "firewall_dhcp_subnet" "clients" {
	subnet       = "10.100.30.0/24"
	auto_collect = false

	routers     = ["10.100.30.1"]
	dns_servers = ["10.100.30.1"]
	ntp_servers = ["10.100.30.1"]
}

flowchart LR
		CLIENT[QUANTUM client] -->|DHCP Discover| KEA[OPNsense Kea]
		KEA -->|Offer: address only| CLIENT
		CLIENT --> IP[Client has a 10.100.30.x address]
		CLIENT -. missing .-> GW[Default gateway 10.100.30.1]
		CLIENT -. missing .-> DNS[DNS server 10.100.30.1]
		IP --> SYMPTOM[Looks connected but cannot route or resolve]

The DHCP daemon stayed healthy while `auto_collect` removed the information clients needed to use their leases.

The important thing here is that the provider default did not match the existing OPNsense behavior. I now set every client-visible DHCP option explicitly.

After restoring the option data, I verified DHCP as a client would:

obtain a new lease;
inspect the offered router and DNS options;
reach the gateway;
resolve a name;
cross the firewall to an external address.

“The service is running” would not have caught this failure. DHCP was running perfectly while handing out incomplete leases.

Importing Omada configuration

The managed-switch controller added another translation layer. The API endpoint behind the normal reverse-proxy address redirected login requests, while the provider expected to talk directly to the controller. Connecting to the direct management origin fixed authentication.

Imports then showed several values whose controller defaults differed from the provider defaults: multicast snooping, relay booleans, and profile flags. To reach a zero-diff plan, I had to write values that the GUI had previously left implicit.

Wireless credentials were particularly important. The controller returned a non-null pre-shared key. Omitting the field in code did not mean “leave it alone”; it meant “clear it.” The secret therefore had to be supplied at runtime from an encrypted source so the plan could preserve the live network without committing the key.

Hardware controls deserve the same caution. On this controller, Power over Ethernet belonged to a port profile. Applying a profile with PoE disabled to a live access point would cut power to the device carrying the management traffic. I treated profile changes as physical operations, not harmless metadata edits.

Firewall rules and their real order

Firewall filters were the highest-risk phase because the appliance had two rule stores:

legacy rules created in the traditional per-interface GUI;
automation rules created through the API and managed by OpenTofu.

The provider could not import legacy rules because they were not the same kind of object. They had to be recreated in the automation store.

That raised a more important question than whether the declarations looked equivalent: where would the new rules land in the effective packet-filter order?

Firewall evaluation is ordered. Two identical sets of rules can behave differently if a broad pass or block moves above a specific exception. The GUI’s visual order was not enough because it separated the two stores.

I found an API endpoint that returned the compiled packet-filter rules in actual evaluation order, including labels that distinguished automation objects from legacy objects. I wrapped it in a small read-only script and made its output a mandatory gate for every interface migration.

flowchart TB
	TF[OpenTofu resources] --> AUTO[os-firewall Automation store]
	GUI[Existing GUI rules] --> LEGACY[Legacy interface store]
	AUTO --> COMPILE[OPNsense rule compiler]
	LEGACY --> COMPILE
	SYSTEM[Anti-lockout and generated rules] --> COMPILE
	COMPILE --> PF[Effective pf rules in @N order]
	PF --> CHECK[pf-rule-order.sh verification]
	CHECK -->|Automation safely shadows legacy| REMOVE[Remove legacy twin]
	CHECK -->|Unexpected order| STOP[Stop and repair sequence]

OPNsense displayed legacy and Automation rules separately, so I queried the compiled pf order before removing any legacy rule.

The sequence per interface became:

Recreate a small set of legacy rules as automation resources.
Apply them while the legacy originals remain enabled.
Query the compiled ruleset.
Confirm the automation rules sit in the intended order and shadow the legacy copies safely.
Test traffic through that interface.
Disable, then remove, the legacy copies.
Plan again and confirm no unexpected drift.

I started with a low-risk guest network containing only three rules. It was a canary for the ordering model. Only after its compiled order and behavior were correct did I migrate management, server, VPN, and WAN interfaces one at a time.

Explicit sequence values were essential. Relying on every resource’s default sequence created ties and non-deterministic placement. I reserved sequence ranges per interface so both humans and the provider had one stable ordering model.

Disabled rules can still block deletion

One migration exposed another appliance quirk. A disabled legacy rule still referenced an alias, and that reference prevented OpenTofu from deleting the alias. From an operator’s perspective the rule was inactive. From the appliance’s validation perspective it still existed.

The fix was to remove the obsolete legacy rule, not merely disable it.

This is why I avoided bulk cleanup during adoption. Relationships that do not affect packet evaluation can still affect schema validation and deletion order.

Migrating NAT separately

Filter rules and NAT rules may appear together in the GUI, but they are not the same ownership boundary. Some legacy firewall rules carried an association to a generated NAT rule that the provider could not preserve. I migrated NAT in a later phase after filter behavior was stable.

The NAT provider also had schema gaps: some labels were unavailable, some port fields rejected aliases, and protocol values normalized differently from the appliance. These limitations did not invalidate the whole migration. They defined which details stayed appliance-managed and which needed a different expression.

I left settings in the GUI when the provider could not safely read and write them. It is better to document that boundary than force an incomplete resource to own it.

Secrets and state

An API-driven firewall migration touches credentials in several places:

firewall API keys;
VPN static keys and certificates;
wireless pre-shared keys;
remote-state access credentials.

I kept secrets encrypted outside the HCL and injected them into provider or resource variables only for the command that needed them. That keeps plaintext out of source files, but it does not automatically keep secrets out of state. Provider schemas may still serialize sensitive values into the state backend.

The state backend therefore needs the same protection as the firewall backup: access control, encryption, and a tested recovery procedure. If the backend has no locking, only one writer can safely apply at a time.

Import declarations are worth retaining as disaster-recovery documentation. A state loss otherwise also loses the mapping between stable resource names and opaque appliance UUIDs.

Checks used for each resource type

For every new resource family, I now ask:

Before import

Does the provider read and write the same API representation?
Which live fields are absent from the schema?
Which provider defaults differ from appliance defaults?
Can this resource interrupt the management path, power, DHCP, DNS, or WAN?
Is there a read-only way to inspect the compiled/effective result?

Before the first apply

Is the plan a no-op after import?
Are secret values present at runtime but absent from source?
Is the first apply limited to one object or one low-risk segment?
Is there an independent management path and a rollback artifact?

After apply

Did a real client receive the expected service, not merely a green status?
Does the compiled firewall order match the intended order?
Did the appliance preserve fields the provider does not expose?
Does a second plan return to no changes?

Final setup

Not every OPNsense setting is managed by OpenTofu. Some remain in the GUI because the provider cannot represent or write them safely. What I have now is a clear list of which tool owns each resource and a repeatable process:

import live state;
insist on zero drift;
change one boundary at a time;
inspect the effective system, not just the tool’s model;
preserve a way back in.

The main rule is to get a no-change plan after import and then apply one small change. Also verify from a real client. In the DHCP incident, the daemon was healthy and the apply succeeded, but clients received leases without a gateway or DNS server.

Moving My NixOS Network to systemd-networkd

Thu, 18 Jun 2026 00:00:00 +0530

My NixOS server heavymetal already had multiple VLANs, static IP addresses and macvlans used by containers. They were created using startup scripts and worked, but restarts and NixOS activations did not always recreate them in the correct order.

I first moved the same setup to systemd-networkd. Later, when I needed to attach VM tap interfaces to the VLANs, I added per-VLAN bridges and finally replaced them with one VLAN-aware bridge.

The migration happened in three steps:

replace imperative networking with networkd units;
move host addresses from VLAN devices onto stable per-VLAN bridges;
consolidate those bridges into a single VLAN-aware trunk.

I did not apply the bridge migrations live because they moved the same management IP used by SSH. Those changes were staged for boot with console access available.

flowchart TB
	S["Scripted VLAN and
macvlan devices"] --> N["systemd-networkd owns
the same topology"]
	N -->|Parent restart exposes child lifecycle| P["Per-VLAN bridges
brvlan20 / brvlan30 / brvlan100"]
	P -->|Stable parents enable VM taps| T["VLAN-aware bridge
brtrunk"]

The network evolved in three controlled steps rather than jumping directly from scripts to the final VLAN-aware bridge.

The old startup scripts

The old system created VLAN and macvlan devices through a mixture of higher-level configuration and startup services. Conceptually it did this:

ip link add link enp5s0 name enp5s0.100 type vlan id 100
ip address add 10.100.100.20/24 dev enp5s0.100
ip link set enp5s0.100 up

ip link add link enp5s0.100 name tfkshim type macvlan mode bridge
ip link set tfkshim up

Imperative commands are easy to prototype, but they hide ownership questions:

Which service owns the VLAN device?
What should restart when the parent disappears?
Is the address ready before a dependent service binds it?
What removes stale devices after a failed partial run?
Does a config reload destroy children created by another service?

The scripts encoded answers in execution order rather than in the topology.

Recreate the same network with networkd

I moved each concern into an explicit networkd object: VLAN .netdev units, matching .network units for addresses and routes, and dependencies for services that needed those links.

On NixOS, a simplified VLAN looks like this:

systemd.network.netdevs."10-vlan20" = {
	netdevConfig = {
		Name = "vlan20";
		Kind = "vlan";
	};
	vlanConfig.Id = 20;
};

systemd.network.networks."40-vlan20" = {
	matchConfig.Name = "vlan20";
	address = [ "10.20.0.10/24" ];
	routes = [ { Gateway = "10.20.0.1"; } ];
	networkConfig.RequiredForOnline = "routable";
};

The exact syntax is distribution-specific; the systemd concepts are not. A .netdev creates a virtual device. A .network matches a device and assigns addresses, routes, VLAN membership, bridge membership, and online-state requirements.

The important part was preserving topology exactly. This was not the moment to rename every interface or collapse five networks into one bridge. First I needed networkd to reproduce the working system.

systemd-resolved broke container DNS

Enabling networkd on NixOS also enabled systemd-resolved through a distribution default. The host continued to resolve names, so the change initially looked healthy. But /etc/resolv.conf now pointed at the loopback stub:

`1`	`nameserver 127.0.0.53`

Containers copied that file into their own network namespaces. From inside a container, 127.0.0.53 meant the container itself, not the host’s resolved service. External name resolution failed even though host DNS worked.

I explicitly disabled resolved and retained a static resolver address reachable from both the host and containers. Running resolved would also have been valid if the container DNS path had been designed for it. The failure came from changing resolver architecture as an accidental side effect of changing interface management.

The check I added was simple:

1
2
3

cat /etc/resolv.conf
podman exec <container> cat /etc/resolv.conf
podman exec <container> getent hosts example.com

A host-only lookup is not enough after a network-manager migration.

Restarting a VLAN removed its macvlan children

An older deployment had already revealed a more disruptive lifecycle problem. When the service owning a VLAN netdev restarted, it deleted and recreated the parent device. Linux also deleted every macvlan child attached to that parent.

The container runtime’s database still believed the container was attached. The actual network namespace contained only loopback and an unrelated bridge. The reverse proxy logged that its interface had been removed and lost its virtual address.

The journal made the sequence visible:

VLAN netdev service stopped
parent link deleted
macvlan child removed by kernel
container remains running without expected interface

sequenceDiagram
	participant Apply as NixOS activation
	participant Parent as enp5s0.100 netdev
	participant Kernel as Linux kernel
	participant Child as tfkshim macvlan
	participant Podman as Traefik container

	Apply->>Parent: Stop and delete VLAN netdev
	Parent->>Kernel: ip link del enp5s0.100
	Kernel--xChild: Delete every macvlan child
	Child--xPodman: eth0 removed from namespace
	Note over Podman: Container keeps running and DB still says attached
	Apply->>Parent: Recreate VLAN netdev
	Note over Parent,Podman: Parent returns but child does not

Deleting and recreating the VLAN parent also deleted the live macvlan child, while Podman's stored attachment remained stale.

At first I coupled the container-network service lifecycle to the parent netdev service: if the parent restarted, the container networks and their consumers restarted and reattached. That repaired the immediate inconsistency.

It also made the weakness of the topology clear. Long-lived workloads were attached directly to a device that configuration reconciliation was allowed to destroy.

Adding one bridge per VLAN

I needed the same VLANs to be shared with ThingsHQ microVM tap devices. A macvlan parent cannot serve that role cleanly, so I created always-on host bridges such as brvlan20 and brvlan100.

The VLAN uplink became an addressless bridge port. The host address moved to the bridge. Containers and VMs attached to the bridge rather than directly to the VLAN netdev.

A simplified networkd definition looks like:

systemd.network.netdevs."10-br-vlan20" = {
	netdevConfig = {
		Name = "br-vlan20";
		Kind = "bridge";
	};
};

systemd.network.networks."40-vlan20" = {
	matchConfig.Name = "vlan20";
	networkConfig.Bridge = "br-vlan20";
	linkConfig.RequiredForOnline = "enslaved";
};

systemd.network.networks."40-br-vlan20" = {
	matchConfig.Name = "br-vlan20";
	address = [ "10.20.0.10/24" ];
	routes = [ { Gateway = "10.20.0.1"; } ];
	networkConfig.RequiredForOnline = "routable";
};

This transition could not be safely applied live. A macvlan cannot be re-parented in place. Networkd could not enslave the VLAN uplink to the new bridge while the live macvlan child still depended on it, and deleting the child would remove the remote management path carried by the reverse proxy.

Instead of a live configuration switch, I staged the new boot generation and rebooted through a console-backed maintenance path. At boot, every device was created in the new topology from an empty state. The previous generation remained selectable in the boot menu if the new management address did not come up.

That was not excessive caution. “Same IP, different owning device” is still a remote access migration.

Moving to one VLAN-aware bridge

Per-VLAN bridges solved the VM-sharing problem, but the host eventually needed a larger trunk topology. Five bridges and five VLAN uplinks repeated the same structure and made VM trunk attachment awkward.

The next design used a single bridge named brtrunk with VLAN filtering.

flowchart LR
	SWITCH[Omada trunk] ==>|tagged VLANs| NIC[enp5s0]
	NIC --> BR[brtrunk VLAN-aware bridge]
	BR --> ADMIN[brtrunk.100: host admin 10.100.100.20]
	BR --> THINGS[brtrunk.20: Things services]
	BR --> QUANTUM[brtrunk.30: VPN services]
	BR --> VMS[OPNsense and ThingsHQ VM tap ports]
	BR --> ISOLATED[VLAN 67 sync and VLAN 999 WAN transport: no host L3]

In the final design, enp5s0 is a bridge port; brtrunk owns VLAN filtering, while host L3 exists only on selected VLAN interfaces.

In networkd terms, the bridge enables VLAN filtering and the physical interface is enslaved as a trunk port:

systemd.network.netdevs."10-br-trunk" = {
	netdevConfig = {
		Name = "brtrunk";
		Kind = "bridge";
	};
	bridgeConfig = {
		VLANFiltering = true;
		DefaultPVID = 1;
	};
};

systemd.network.networks."40-physical-trunk" = {
	matchConfig.Name = "enp5s0";
	networkConfig.Bridge = "brtrunk";
	linkConfig.RequiredForOnline = "enslaved";
};

The host’s management address moved from a per-VLAN bridge onto a VLAN interface of the trunk bridge. Some transport VLANs existed only inside the bridge and had no host-layer address at all.

Again, I staged the configuration for the next boot rather than attempting to replace the bridge beneath an active SSH session. The migration changed the master of the physical NIC, removed several bridges, recreated VLAN interfaces, and moved the default route. A reboot was the deterministic path.

Conflicting forwarding sysctls

After one deployment, host-to-container routing failed even though net.ipv4.ip_forward appeared in the configuration.

Linux exposes closely related forwarding sysctls:

1
2

net.ipv4.ip_forward
net.ipv4.conf.all.forwarding

Modules had written both aliases with conflicting values. The canonical per-family setting remained zero at runtime, leaving forwarding disabled. I made both values explicit:

boot.kernel.sysctl = {
	"net.ipv4.ip_forward" = 1;
	"net.ipv4.conf.all.forwarding" = 1;
	"net.ipv6.conf.all.forwarding" = 1;
};

Both names refer to closely related kernel settings and another module can write one after the other. So I check the runtime values after activation:

1
2
3

sysctl net.ipv4.ip_forward
sysctl net.ipv4.conf.all.forwarding
sysctl net.ipv6.conf.all.forwarding

The kernel is the final source of truth.

Source-only addresses and automatic routes

The host also had secondary interfaces used only to originate scans or reach macvlan workloads. Those addresses must not automatically install connected prefix routes if another interface owns the real return path.

I set AddPrefixRoute=false on those source-only legs and kept reverse-path filtering loose where asymmetric routes were intentional. Otherwise Linux could prefer the newly connected /24, send replies out the wrong interface, and turn a harmless scan address into an SSH lockout.

This is a niche detail with a broad principle: every new address can also create a route. During migrations, compare the routing table, not just the address list.

Checks after each migration

After each stage I checked the system from the bottom up.

Link ownership

1
2
3

networkctl status
ip -br link
ip -br address

Is the physical NIC enslaved to the intended bridge?
Are VLAN devices attached to the intended parent?
Is each host address on the bridge/VLAN device, not the addressless port?

Routes and forwarding

1
2
3

ip route
ip -6 route
sysctl net.ipv4.conf.all.forwarding

Is there exactly one intended default route?
Did source-only addresses add unwanted connected routes?
Is forwarding enabled in runtime state?

Namespaces

Can a container resolve DNS?
Does its namespace contain every expected interface?
Can it reach both an internal backend and an external endpoint?
Does restarting the parent/network service reattach the child cleanly?

Reboot behavior

Does the host return on the management address without manual intervention?
Do services wait for the interface state they actually require?
Does a second ordinary configuration apply leave the topology intact?

Final setup

systemd-networkd did not make the network itself simple, but it made device ownership and startup dependencies visible. The order I would use again is:

reproduce the working topology declaratively;
observe the lifecycle failures that the old scripts concealed;
introduce stable bridge ownership;
consolidate only after the dependencies are understood;
stage non-live-reparentable changes for reboot with console rollback.

Do not assume a declarative network change is safe to apply over SSH. Moving a management IP from a VLAN to a bridge still removes and recreates interfaces. For those changes, I used a boot-time migration and kept the previous NixOS generation available from the console.

Moving Traefik into a Podman Container

Thu, 11 Jun 2026 00:00:00 +0530

I was running Traefik as a native NixOS service. Over time, I had added proxy services to let it reach applications in Podman networks. Instead of keeping those extra proxies, I decided to run Traefik itself in Podman and attach it to the required networks.

The container started and its health check passed, but some routes returned 502 Bad Gateway. Requests from another VLAN timed out, and after recreating the container the ingress IP stopped responding.

There were three separate issues:

loopback stopped meaning the host;
a multi-homed network namespace chose the wrong return route;
the macvlan MAC changed while the gateway still remembered the old one.

The container network setup

Native Traefik had accumulated proxy shims to reach services spread across host processes, Podman bridges, and network containers. Moving Traefik itself into Podman let it join application networks directly and removed a layer of socket forwarding.

The resulting Traefik container joined three networks:

podman_network_vlan_100 for the ingress VIP 10.100.100.21;
podman_rproxy at 10.89.0.2 for host and internal backends;
an internet bridge at 172.28.0.12 for outbound access.

flowchart TB
	CLIENTS[Clients and OPNsense] -->|HTTPS to 10.100.100.21| VLAN[macvlan VLAN 100]
	VLAN --> T[Traefik 3.7.5]
	T -->|10.89.0.1 backend ports| HOST[podman_rproxy host gateway]
	HOST --> SERVICES[Netdata, Glances, Cockpit, go2rtc, Home Assistant]
	T -->|container DNS and updates| INET[172.28.0.0/24 Internet bridge]

Containerized Traefik is multi-homed: ingress, private backends, and Internet access are separate network legs.

The three networks worked, but the old Traefik configuration still assumed it was running in the host network namespace.

Host services returned 502

Several file-provider routes still targeted backends such as:

1
2

servers:
  - url: http://127.0.0.1:19999

That URL worked when Traefik was a host process. Inside the container, 127.0.0.1 referred to the Traefik container itself. Netdata, go2rtc, Cockpit, and other host services had not moved with it.

Services on the same Podman bridge continued to work. Only host services and systemd-nspawn backends failed, while the Traefik health check remained green.

I tested from both sides of the namespace boundary:

# Works from the NixOS host
curl -fsS http://10.89.0.1:8123/

# Initially failed from Traefik's namespace
podman exec traefik wget -qO- -T2 http://10.89.0.1:8123/

The second command showed the problem without involving DNS, TLS or OPNsense.

The fix had three parts:

Bind host services to the podman_rproxy gateway, 10.89.0.1, rather than relying on loopback.
Allow only the required backend ports on the podman_rproxy interface in the NixOS firewall.
Change Traefik’s backend URLs from 127.0.0.1 to 10.89.0.1.

For Netdata, I also widened its application-level allow list from localhost to the 10.89.0.0/24 proxy subnet. Opening the host firewall alone would not have overridden the service’s own filter.

flowchart TB
	subgraph NATIVE[Native Traefik]
		NT[Traefik process] -->|127.0.0.1| NS[Host service]
	end
	subgraph CONTAINER[Containerized Traefik]
		CT[Traefik container] -->|127.0.0.1| SELF[Traefik container loopback]
		CT -->|10.89.0.1| HS[Host service on podman_rproxy]
	end
	NT -. moved into container .-> CT

The same URL points at a different machine after Traefik crosses into a container namespace.

The 502s disappeared, but off-subnet clients still timed out.

Requests from other VLANs timed out

Traefik had more than one default route because netavark attached multiple networks. The route selected during one container creation was not guaranteed to be the same route selected after another.

Clients on VLAN 100 were on-link with the macvlan VIP, so replies used the connected route and worked. A client from another VLAN reached 10.100.100.21 through OPNsense, but Traefik’s reply followed the internet bridge’s default route. The request entered through the macvlan and the response tried to leave through a different gateway.

1
2

off-subnet request: client -> OPNsense -> Traefik macvlan
wrong reply:        Traefik -> Podman Internet bridge -> nowhere useful

Testing only from VLAN 100 made the setup look correct. I had to test from another VLAN and inspect the route inside Traefik’s network namespace.

Inside Traefik’s network namespace, ip route get <client-address> exposed the selected gateway. I installed an explicit lower-metric default route through the VLAN 100 gateway:

`1`	`ip route replace default via 10.100.100.1 metric 50`

The internet-bridge route remained at metric 100. This was applied by the NixOS unit after Podman created the namespace, so every recreation restored the same decision.

flowchart TB
	C[Client on another VLAN] -->|request| FW[OPNsense]
	FW -->|10.100.100.21| T[Traefik]
	T -. wrong default before fix .-> I[Internet bridge gateway]
	I -. dropped reply .-> C
	T -->|metric 50 after fix| FW
	FW -->|symmetric reply| C

Equal default routes made off-subnet replies nondeterministic; the lower-metric VLAN route restored symmetry.

After the route fix, off-subnet curls returned HTTP 200 and ip route get showed the macvlan gateway. Then a later container recreation broke ingress once more.

The macvlan MAC changed after recreation

The macvlan attachment had a static IP but no static MAC. Podman generated a new MAC when Traefik was recreated. OPNsense and the upstream switching path still associated 10.100.100.21 with the previous MAC.

The container health and route table still looked correct because the stale entry was on OPNsense and the switching path, not inside the container.

I pinned the attachment identity in the Podman network specification:

`1`	`"podman_network_vlan_100:ip=10.100.100.21,mac=02:42:0a:64:64:15"`

The locally administered 02:42 prefix avoids claiming a vendor identity. The remaining octets encode the private address, which makes the value easy to audit. More importantly, recreation no longer changes the layer-2 identity.

Checks after the migration

A reverse proxy health check proves only that Traefik can answer inside its own namespace. My final test matrix covered each boundary explicitly:

Test	What it proves
Curl backend from NixOS host	Backend process is listening
Curl backend from Traefik container	Host firewall, bind address, and app allow list work
Request ingress from VLAN 100	Macvlan listener and on-link return path work
Request ingress from a different VLAN	Routed return path uses the correct gateway
Inspect `ip route get` in the namespace	Route choice is deterministic
Recreate Traefik and repeat all tests	MAC and post-start route configuration persist

I also checked the gateway neighbor entry after recreation rather than waiting for a browser symptom.

Key points

Moving Traefik into a container changed more than the process manager. The native service could reach host loopback and used the host routing table. The container could use only its attached interfaces and had multiple default routes.

For a similar migration, check all backend bind addresses, test from inside the Traefik container, test ingress from another subnet and recreate the container once before considering it complete. A static macvlan IP should also have a stable MAC when upstream devices keep neighbor entries for it.

Networking on Dexome

Setting up OPNsense HA with CARP and pfsync

The HA setup

Configure CARP on the first node

Booting the backup caused an mDNS storm

Dedicated network between the firewalls

Problems with configuration sync

Management ports were different

GUI was not listening on the sync interface

Certificate verification failed

The sync user needed another privilege

Changes were not pushed automatically

Kea DHCP hot standby

Unbound stopped answering after failover

Keeping connections with pfsync

How I tested the failover

Final checks

Moving My PPPoE WAN to VLAN 999

The setup

Checking the VLAN first

Checking the WAN MAC

Closing the old PPPoE session

Omada loop detection on the ONT port

The migration steps I use now

Before the maintenance window

During the cutover

After the session comes up

Key things to check

Managing My Existing OPNsense Setup with OpenTofu

Import first and make no changes

Start with Unbound DNS

Kea DHCP and the auto_collect issue

Importing Omada configuration

Firewall rules and their real order

Disabled rules can still block deletion

Migrating NAT separately

Secrets and state

Checks used for each resource type

Before import

Before the first apply

After apply

Final setup

Moving My NixOS Network to systemd-networkd

The old startup scripts

Recreate the same network with networkd

systemd-resolved broke container DNS

Restarting a VLAN removed its macvlan children

Adding one bridge per VLAN

Moving to one VLAN-aware bridge

Conflicting forwarding sysctls

Source-only addresses and automatic routes

Checks after each migration

Link ownership

Routes and forwarding

Namespaces

Reboot behavior

Final setup

Moving Traefik into a Podman Container

The container network setup

Host services returned 502

Requests from other VLANs timed out

The macvlan MAC changed after recreation

Checks after the migration

Key points

Kea DHCP and the `auto_collect` issue