Firewall on Dexome

Setting up OPNsense HA with CARP and pfsync

Tue, 04 Aug 2026 00:00:00 +0530

I run two OPNsense VMs, one on metalbox and another on heavymetal. I wanted the second firewall to take over the gateway, DNS, DHCP and existing connections when the first host was unavailable.

CARP itself was the easy part. The difficult parts were the services around it. When I first booted the backup firewall, the network reached around 160,000 packets per second and became unusable. Later, CARP moved the IP correctly but DNS stopped and existing TCP connections were lost.

Below is how I configured the full setup and the issues I found while testing it.

The HA setup

The finished design runs two OPNsense VMs on separate NixOS hypervisors. Node A runs on metalbox; node B runs on heavymetal. Clients keep using the familiar .1 gateways on each internal VLAN, now implemented as CARP VIPs. The nodes have real .6 and .7 management addresses, and a dedicated VLAN 67 sync link at 10.100.67.6/28 and 10.100.67.7/28.

flowchart TB
	CLIENTS[LAN, THINGS, QUANTUM and SERVERS clients] --> VIPS[CARP gateway VIPs ending in .1]
	INTERNET[PPPoE uplink] --> WANVIP[WAN CARP VIP 192.168.1.5]

	subgraph M[metalbox NixOS hypervisor]
		A[OPNsense A - preferred MASTER]
		AREAL[Real addresses ending in .6]
		A --- AREAL
	end

	subgraph H[heavymetal NixOS hypervisor]
		B[OPNsense B - BACKUP]
		BREAL[Real addresses ending in .7]
		B --- BREAL
	end

	VIPS --> A
	VIPS -. failover .-> B
	WANVIP --> A
	WANVIP -. failover .-> B
	A ==>|VLAN 67: XMLRPC, pfsync, Kea HA| B

The completed OPNsense HA topology separates client VIPs from node management and the VLAN 67 control plane.

There are four separate parts in this setup:

CARP moves gateway and service IP addresses between nodes.
Configuration sync keeps rules and services aligned.
State sync copies the firewall state table so established connections live.
Application-level HA makes services such as DHCP and mDNS behave correctly.

I configured and tested them one at a time.

Configure CARP on the first node

Before introducing a backup, I converted the existing firewall into a single CARP master. Its old gateway addresses became virtual IPs, while the firewall received separate real addresses for management. The WAN source-NAT rule also changed to use the WAN virtual IP, so outbound traffic would retain the same source after a failover.

This stage sounds redundant: why configure failover with only one node? Because it isolates the address migration from the redundancy problem. I could prove that clients still reached their familiar gateways, outbound NAT used the expected address, and management remained available before another system was allowed to advertise anything.

It also established a useful naming pattern:

one stable name for the firewall service, resolving to the virtual IP;
one node-specific name per firewall, resolving to its real address.

When HA itself is broken, the node-specific addresses are the way back in.

Booting the backup caused an mDNS storm

The second node started from a copy of the first node’s configuration. Its CARP advertisement priority was lower, so it should have stayed in BACKUP. Within roughly 30 to 60 seconds, however, the network flooded.

The timing made a split-brain theory persuasive. I checked the usual suspects:

shared CARP passwords matched;
virtual host IDs matched;
the master and backup had the intended advertisement skew;
multicast advertisements arrived on every VLAN;
the backup remained silent in BACKUP instead of advertising as another master.

These checks showed CARP was working as expected. I then captured the traffic causing the packet storm.

The breakthrough was to stop looking only at CARP packets and inspect the storm itself. Almost all of it was UDP port 5353: multicast DNS. Both node MAC addresses were flooding at similar rates.

The copied configuration had enabled an mDNS repeater on both firewalls. Each reflector received packets repeated by the other and reflected them again across the same interfaces. A small amount of multicast became an exponential loop.

sequenceDiagram
	participant Device as mDNS device
	participant A as OPNsense A reflector
	participant B as OPNsense B reflector
	participant LAN as Other VLANs

	Device->>A: Multicast query on UDP 5353
	A->>LAN: Reflect query
	LAN->>B: Reflected query arrives
	B->>LAN: Reflect it again
	LAN->>A: Re-reflected query arrives
	loop Exponential amplification
		A->>LAN: Reflect B's copy
		LAN->>B: Deliver A's copy
		B->>LAN: Reflect A's copy
		LAN->>A: Deliver B's copy
	end

OPNsense B inherited the active mDNS repeater configuration. Each node reflected the other's reflected packets until the LAN reached roughly 160,000 packets per second.

The firewall software had the correct feature for this situation: enable the repeater only while the node is CARP master. I configured the service identically on both nodes but enabled its CARP-aware failover mode. The backup retained the configuration without running the reflector until promotion.

The same check is needed for any service that broadcasts or reflects traffic. Copying the configuration to the backup can make both instances active at the same time.

Dedicated network between the firewalls

I added VLAN 67 between the firewalls for control traffic. It had no client gateway and no virtual IP. OPNsense A used 10.100.67.6/28, OPNsense B used 10.100.67.7/28, and a tightly scoped firewall rule allowed traffic only within that sync subnet.

The lack of a default pass rule on a newly assigned firewall interface was an early trap. Both addresses existed, but all layer-3 traffic was silently dropped until the sync-network rule was installed. Link state and correct addresses do not prove that the control plane can communicate.

The dedicated link carried config sync, pfsync, and DHCP peer communication. It also kept that traffic away from client VLANs and gave packet captures a much cleaner place to answer “did the peers actually talk?”

flowchart LR
	A[OPNsense A 10.100.67.6] ==>|XMLRPC: configuration A to B| B[OPNsense B 10.100.67.7]
	A <-->|pfsync: firewall states| B
	A <-->|Kea HA: leases and peer health| B

	CARP[CARP advertisements on client VLANs] -.-> A
	CARP -.-> B

CARP is only the address-ownership layer; three separate protocols cross VLAN 67 to preserve configuration, leases, and live connections.

Problems with configuration sync

The config-sync page looked straightforward: peer address, username, password, and a list of areas to synchronize. It failed for five independent reasons.

Management ports were different

One node exposed its GUI directly on the default HTTPS port. The other listened on a non-default local port behind a reverse proxy. Config sync contacted the peer’s real address, not the public proxy name, so both web services needed a reachable and explicitly matching port.

GUI was not listening on the sync interface

The backup’s web server listened only on its LAN interface. A TCP connection over the dedicated sync address therefore had nowhere to land. Adding the sync interface to the GUI’s listen scope fixed the transport without widening client access.

Certificate verification failed

The peers used a self-signed management certificate over a private, dedicated L2 link. Certificate verification failed before credentials were considered. In this topology I disabled peer verification for the sync call. A better option, when supported, is to give each node a certificate chaining to an internal CA.

The sync user needed another privilege

The user could open the HA configuration page but could not call the XMLRPC library. “High Availability” GUI access and XMLRPC execution were separate permissions. Granting only the narrowly required library privilege fixed the API call without turning the account into an administrator.

Changes were not pushed automatically

The largest conceptual surprise was that saving a configuration did not necessarily push it to the backup. The GUI’s explicit “synchronize all” action worked, but ordinary changes and infrastructure-as-code applies did not invoke it.

I added a post-apply hook that calls the firewall’s sync endpoint. GUI-only edits still require the operator to trigger synchronization, so the operating rule is simple: one node is authoritative; the backup is not a second place to edit.

The reusable lesson is to test config sync as an action, not as a checkbox. Change a harmless object on the primary, trigger the documented sync path, and prove it appears on the backup.

Kea DHCP hot standby

CARP moving the gateway does not automatically make the backup DHCP server safe. Two independent DHCP servers with copied configuration can both answer clients, while a master-only DHCP service can leave an availability gap during promotion.

I used Kea’s hot-standby mode over the sync network. The primary serves leases under normal conditions; the standby receives lease updates and takes over after it decides the partner is down.

The configuration failed when both peers used the same server name. Kea requires this-server-name to match one of the declared peer names. Explicitly writing the primary’s name into synchronized configuration made both nodes identify as the primary.

I gave the firewalls distinct hostnames, declared those as peer names, and left this-server-name empty so each node derives its identity locally. The node identity cannot be copied from primary to backup.

Hot standby also has a detection interval. In this case the standby waited about a minute before entering partner-down mode. Existing clients with leases stayed online; a new or renewing client might wait. That is not a broken failover, but it is part of the service-level objective and should be measured rather than assumed.

Unbound stopped answering after failover

With CARP working, the virtual gateway moved cleanly to the backup. Clients could route, but DNS queries to the gateway timed out.

The resolver had been configured to bind selected interfaces. On the backup, virtual IPs do not exist while it is in BACKUP. When the resolver configuration was regenerated, the absent virtual address disappeared from the generated listen set. CARP later promoted the node and created the address, but the resolver was not listening there.

The fix was to let the resolver listen on all local interfaces using its automatic interface behavior. Firewall policy, rather than a brittle application bind list, controlled which networks could reach port 53.

This is a broader HA pattern: applications should not permanently compile a list of addresses whose presence changes with ownership. Either bind wildcard/automatic and enforce access in the firewall, or use a service hook that reacts correctly to every promotion and demotion.

Keeping connections with pfsync

At this point a cable pull promoted the backup, moved the virtual IPs, and kept DNS and DHCP available. Yet established TCP sessions still died.

That was expected in retrospect. A stateful firewall tracks sequence numbers, NAT translations, timeouts, and policy decisions in a state table. The backup cannot infer that state from a newly arrived midstream packet.

I enabled pfsync on both nodes using explicit unicast peer addresses on the sync network. Configuring only one direction was not sufficient; each node needed to send and receive state because either could be active after maintenance or a second failure.

The acceptance test was deliberately mundane: start a long-lived transfer, pull the active firewall’s network cable, and watch whether the transfer continues. After bidirectional state sync was working, it did.

That test proved something a successful ping never could.

How I tested the failover

I ended with a matrix instead of a single “HA works” result:

Test	What it proves
Primary enters maintenance mode	Graceful CARP demotion and backup promotion
Primary loses its cable	Failure detection without operator help
Fail back to the original node	Prior master can safely resume ownership
DNS query during promotion	Resolver listens correctly on the new owner
New DHCP request after peer timeout	Standby has leases and enters partner-down
Long TCP transfer during cable pull	State and NAT translations were synchronized
Config change on primary	Explicit sync path updates the backup
Boot both nodes from cold	Broadcast services do not duplicate or loop

Running the matrix in both directions exposed assumptions hidden by testing only the preferred primary.

Final checks

Moving the virtual IP was only one of the checks. The setup was complete when:

there is exactly one active owner for each singleton or broadcast service;
both nodes carry compatible configuration without becoming competing writers;
DHCP and DNS continue according to measured failover timing;
established stateful connections survive a hard loss;
operators can reach either node through a real, node-specific address;
the behavior has been proven by removing the active node, not inferred from a green status page.

The main issue I found was assuming that copying a service configuration also made the service HA-aware. The mDNS repeater had to run only on the CARP master, Kea needed its own peer state and Unbound needed to listen on an address which appears only after promotion.

Finally, test with a real cable pull and a long-running connection. A successful ping after pressing the CARP maintenance button does not verify pfsync or the hard-failure path.

Managing My Existing OPNsense Setup with OpenTofu

Tue, 07 Jul 2026 00:00:00 +0530

I wanted to manage my existing OPNsense configuration using OpenTofu. The firewall was already running DNS, DHCP, several VLANs, VPN connections and all the rules for my home network. Recreating everything from code was not an option.

I used the browningluke/opnsense provider and started with one Unbound DNS setting. After importing it, I did not continue until the plan showed:

`1`	`No changes. Your infrastructure matches the configuration.`

This worked for DNS, but the next step caused a DHCP outage because one provider default removed the gateway and DNS options from client leases. This post covers the order I used after that incident and the provider limitations I found.

Import first and make no changes

There are two separate things we may want to do during this migration:

represent the current system in code;
clean up the system while doing it.

I would not combine them.

The first goal is adoption. Its success criterion is boring: after import, the configuration describes the live object exactly enough that a plan proposes no change. Only after that baseline is stable should a separate change improve the object.

This matters most for routers and firewalls because the management path is one of the resources being changed. An incorrect web-server deployment can return a 500. An incorrect gateway, DHCP option, or anti-lockout rule can remove the path you need to repair it.

I used four gates for every subsystem:

1
2
3

inventory ──> import ──> zero-diff plan ──> one-object apply
		│             │             │                  │
		└── stop ─────┴── stop ─────┴── stop on drift ─┘

The first real apply was always deliberately small.

flowchart TB
	U[Unbound DNS: 66 objects] -->|zero-diff plan| K[Kea DHCP: 64 objects]
	K -->|client lease test| O[Omada VLANs, profiles, ports and SSIDs]
	O -->|controller no-op| C[Guest firewall canary]
	C -->|compiled pf order| F[Per-interface filter migration]
	F --> N[NAT and VPN resources]

The adoption moved outward from lower-risk DNS objects to connectivity-critical filters and NAT, with a stop gate after every phase.

Start with Unbound DNS

Resolver settings and host overrides were a good first target. Sixty-six Unbound objects were numerous enough to test import automation but less dangerous than rewriting the firewall ruleset.

The import revealed an important category of provider behavior: fields that exist on the appliance but not in the provider schema. One host override generated a reverse record, yet the provider did not expose that switch. Importing and planning the resource produced no change, so the appliance-only field survived.

That was acceptable. IaC coverage does not need to be 100 percent to be useful. It does need to be honest.

Another DNS feature exposed the opposite problem: the provider could read a blocklist setting but failed when writing it. Rather than force ownership, I left that feature GUI-managed and documented the boundary. A provider that cannot round-trip a field does not own that field.

The first phase ended with dozens of objects imported and a no-op plan. The point was not the count. It was proving the API credentials, import identifiers, schema, and state storage before touching client connectivity.

Kea DHCP and the `auto_collect` issue

Kea DHCP import covered 64 objects and looked equally clean until the first apply. Clients on THINGS and QUANTUM renewed and still received valid addresses, but they lost their default gateway and DNS server.

The provider exposed an auto_collect option. Its default was enabled, suggesting that the appliance would derive subnet options automatically. On this system it did not. Applying the resource removed the stored router, DNS, and NTP values.

A simplified version of the dangerous assumption looked like this:

resource "firewall_dhcp_subnet" "clients" {
	subnet       = "10.20.0.0/24"
	auto_collect = true
}

The repaired declaration made every client-visible option explicit. This is a simplified version of the QUANTUM subnet, whose gateway and resolver are 10.100.30.1:

resource "firewall_dhcp_subnet" "clients" {
	subnet       = "10.100.30.0/24"
	auto_collect = false

	routers     = ["10.100.30.1"]
	dns_servers = ["10.100.30.1"]
	ntp_servers = ["10.100.30.1"]
}

flowchart LR
		CLIENT[QUANTUM client] -->|DHCP Discover| KEA[OPNsense Kea]
		KEA -->|Offer: address only| CLIENT
		CLIENT --> IP[Client has a 10.100.30.x address]
		CLIENT -. missing .-> GW[Default gateway 10.100.30.1]
		CLIENT -. missing .-> DNS[DNS server 10.100.30.1]
		IP --> SYMPTOM[Looks connected but cannot route or resolve]

The DHCP daemon stayed healthy while `auto_collect` removed the information clients needed to use their leases.

The important thing here is that the provider default did not match the existing OPNsense behavior. I now set every client-visible DHCP option explicitly.

After restoring the option data, I verified DHCP as a client would:

obtain a new lease;
inspect the offered router and DNS options;
reach the gateway;
resolve a name;
cross the firewall to an external address.

“The service is running” would not have caught this failure. DHCP was running perfectly while handing out incomplete leases.

Importing Omada configuration

The managed-switch controller added another translation layer. The API endpoint behind the normal reverse-proxy address redirected login requests, while the provider expected to talk directly to the controller. Connecting to the direct management origin fixed authentication.

Imports then showed several values whose controller defaults differed from the provider defaults: multicast snooping, relay booleans, and profile flags. To reach a zero-diff plan, I had to write values that the GUI had previously left implicit.

Wireless credentials were particularly important. The controller returned a non-null pre-shared key. Omitting the field in code did not mean “leave it alone”; it meant “clear it.” The secret therefore had to be supplied at runtime from an encrypted source so the plan could preserve the live network without committing the key.

Hardware controls deserve the same caution. On this controller, Power over Ethernet belonged to a port profile. Applying a profile with PoE disabled to a live access point would cut power to the device carrying the management traffic. I treated profile changes as physical operations, not harmless metadata edits.

Firewall rules and their real order

Firewall filters were the highest-risk phase because the appliance had two rule stores:

legacy rules created in the traditional per-interface GUI;
automation rules created through the API and managed by OpenTofu.

The provider could not import legacy rules because they were not the same kind of object. They had to be recreated in the automation store.

That raised a more important question than whether the declarations looked equivalent: where would the new rules land in the effective packet-filter order?

Firewall evaluation is ordered. Two identical sets of rules can behave differently if a broad pass or block moves above a specific exception. The GUI’s visual order was not enough because it separated the two stores.

I found an API endpoint that returned the compiled packet-filter rules in actual evaluation order, including labels that distinguished automation objects from legacy objects. I wrapped it in a small read-only script and made its output a mandatory gate for every interface migration.

flowchart TB
	TF[OpenTofu resources] --> AUTO[os-firewall Automation store]
	GUI[Existing GUI rules] --> LEGACY[Legacy interface store]
	AUTO --> COMPILE[OPNsense rule compiler]
	LEGACY --> COMPILE
	SYSTEM[Anti-lockout and generated rules] --> COMPILE
	COMPILE --> PF[Effective pf rules in @N order]
	PF --> CHECK[pf-rule-order.sh verification]
	CHECK -->|Automation safely shadows legacy| REMOVE[Remove legacy twin]
	CHECK -->|Unexpected order| STOP[Stop and repair sequence]

OPNsense displayed legacy and Automation rules separately, so I queried the compiled pf order before removing any legacy rule.

The sequence per interface became:

Recreate a small set of legacy rules as automation resources.
Apply them while the legacy originals remain enabled.
Query the compiled ruleset.
Confirm the automation rules sit in the intended order and shadow the legacy copies safely.
Test traffic through that interface.
Disable, then remove, the legacy copies.
Plan again and confirm no unexpected drift.

I started with a low-risk guest network containing only three rules. It was a canary for the ordering model. Only after its compiled order and behavior were correct did I migrate management, server, VPN, and WAN interfaces one at a time.

Explicit sequence values were essential. Relying on every resource’s default sequence created ties and non-deterministic placement. I reserved sequence ranges per interface so both humans and the provider had one stable ordering model.

Disabled rules can still block deletion

One migration exposed another appliance quirk. A disabled legacy rule still referenced an alias, and that reference prevented OpenTofu from deleting the alias. From an operator’s perspective the rule was inactive. From the appliance’s validation perspective it still existed.

The fix was to remove the obsolete legacy rule, not merely disable it.

This is why I avoided bulk cleanup during adoption. Relationships that do not affect packet evaluation can still affect schema validation and deletion order.

Migrating NAT separately

Filter rules and NAT rules may appear together in the GUI, but they are not the same ownership boundary. Some legacy firewall rules carried an association to a generated NAT rule that the provider could not preserve. I migrated NAT in a later phase after filter behavior was stable.

The NAT provider also had schema gaps: some labels were unavailable, some port fields rejected aliases, and protocol values normalized differently from the appliance. These limitations did not invalidate the whole migration. They defined which details stayed appliance-managed and which needed a different expression.

I left settings in the GUI when the provider could not safely read and write them. It is better to document that boundary than force an incomplete resource to own it.

Secrets and state

An API-driven firewall migration touches credentials in several places:

firewall API keys;
VPN static keys and certificates;
wireless pre-shared keys;
remote-state access credentials.

I kept secrets encrypted outside the HCL and injected them into provider or resource variables only for the command that needed them. That keeps plaintext out of source files, but it does not automatically keep secrets out of state. Provider schemas may still serialize sensitive values into the state backend.

The state backend therefore needs the same protection as the firewall backup: access control, encryption, and a tested recovery procedure. If the backend has no locking, only one writer can safely apply at a time.

Import declarations are worth retaining as disaster-recovery documentation. A state loss otherwise also loses the mapping between stable resource names and opaque appliance UUIDs.

Checks used for each resource type

For every new resource family, I now ask:

Before import

Does the provider read and write the same API representation?
Which live fields are absent from the schema?
Which provider defaults differ from appliance defaults?
Can this resource interrupt the management path, power, DHCP, DNS, or WAN?
Is there a read-only way to inspect the compiled/effective result?

Before the first apply

Is the plan a no-op after import?
Are secret values present at runtime but absent from source?
Is the first apply limited to one object or one low-risk segment?
Is there an independent management path and a rollback artifact?

After apply

Did a real client receive the expected service, not merely a green status?
Does the compiled firewall order match the intended order?
Did the appliance preserve fields the provider does not expose?
Does a second plan return to no changes?

Final setup

Not every OPNsense setting is managed by OpenTofu. Some remain in the GUI because the provider cannot represent or write them safely. What I have now is a clear list of which tool owns each resource and a repeatable process:

import live state;
insist on zero drift;
change one boundary at a time;
inspect the effective system, not just the tool’s model;
preserve a way back in.

The main rule is to get a no-change plan after import and then apply one small change. Also verify from a real client. In the DHCP incident, the daemon was healthy and the apply succeeded, but clients received leases without a gateway or DNS server.