mwan3 is OpenWrt's multi-WAN policy routing framework. It classifies packets using firewall marks, then uses ip rule entries to route marked packets through per-interface routing tables. The nftables port replaces all iptables/ipset usage with nftables equivalents while keeping the ip rule/route management largely unchanged. mwan3rtmon is ported to a ucode implementation.

Key Design Decisions

Own standalone table - mwan3 lives in table inet mwan3, completely independent of table inet fw4. fw4 reload, restart, or reconfiguration has no effect on mwan3 rules.
Static skeleton + dynamic rules - mwan3-skeleton.nft defines empty sets and chains loaded by nft -f at service start using the atomic table-replace idiom. All rules are added dynamically by shell scripts since they depend on the configurable MMX_MASK.
Hook priority mangle + 1 - mwan3_prerouting and mwan3_output register at priority -149, placing them after fw4's mangle chains and any other packages registering at -150. Mark operations use masked OR-immediate setter chains via vmap-dispatch so they are non-destructive with respect to bits owned by other packages regardless of execution order.
Non-destructive mark save/restore - Connmark save and restore are masked to mwan3's own bit-range (MMX_MASK) and never touch bits owned by other packages. The kernel rejects compound two-source bitwise expressions; mwan3 synthesises masked save/restore through a vmap-dispatch technique built from per-mark OR-immediate setter chains. See Section 2 - Connmark Operations.
inet family - Chains handle both IPv4 and IPv6 in a single pass. Sets remain type-specific (separate v4/v6 sets) since nftables requires a single address type per set.
Atomic non-destructive reload - reload_service rebuilds the entire mwan3 ruleset in a single nft -f batch while the old ruleset serves traffic, committing atomically with zero downtime window. See Section 11.
User-defined nft sets - config ipset sections in /etc/config/mwan3 create named nft sets in table inet mwan3 supporting inline entries, file loading, and dnsmasq domain population. See Section 12.

Component Map

                                                UCI Config (/etc/config/mwan3)
                                                              |
                            +---------------------------------+---------------------------------+
                            |                                 |                                 |
                       init.d/mwan3                      mwan3track                         mwan3rtmon
                    (service lifecycle)                (health probes)                 (route replication)
                            |                                 |                                 |
       +--------------------+--------------------+            |                                 |
       |                    |                    |            |                                 |
       v                    v                    v            |                                 |
    common.sh           mwan3.sh        mwan3-skeleton.nft    |                                 |
    (helpers)           (engine)        (static skeleton)     |                                 |
       |                    |                    |            |                                 |
       v                    v                    |            |                                 |
 mwan3-get-addr.uc       mwan3ct                 |            |                                 |
       |          mwan3-manage-rules.uc          |            |                                 |
       |          mwan3-list-routes.uc           |            |                                 |
       |       mwan3-create-iface-route.uc       |            |                                 |
       |                    |                    |            |                                 |
       v                    v                    v            v                                 v
   nft / netlink    netlink / conntrack         nft     writes STATUS files                nft / netlink
                                                      to /var/run/mwan3track
  

        Hotplug:       25-mwan3                      --calls-->   mwan3.sh functions
        Hotplug user:  26-mwan3-user                 --calls-->   /etc/mwan3.user
        CLI:           /usr/sbin/mwan3               --calls-->   mwan3.sh functions
        RPC:           rpcd/ucode/mwan3              --calls-->   nft -j (JSON output)
        dnsmasq:       mwan3_write_dnsmasq_fragments --writes-->  confdir nftset fragments
        mwan3rtmon:                                  --uses-->    ucode-mod-rtnl (netlink) + nft
        mwan3ct:                                     --uses-->    libnetfilter_conntrack + libmnl
        mwan3.sh:                                    --calls-->   mwan3ct (conntrack flush)
        mwan3.sh:                                    --calls-->   mwan3-manage-rules.uc (ip rule management)
        mwan3.sh:                                    --calls-->   mwan3-list-routes.uc (route enumeration)
        mwan3.sh:                                    --calls-->   mwan3-create-iface-route.uc (route install)
        mwan3-get-addr.uc:                           --uses-->    ucode-mod-rtnl (netlink)
        mwan3-manage-rules.uc:                       --uses-->    ucode-mod-rtnl (netlink)
        mwan3-list-routes.uc:                        --uses-->    ucode-mod-rtnl (netlink)
        mwan3-create-iface-route.uc:                 --uses-->    ucode-mod-rtnl (netlink)

2. The Mark Bitmask System

mwan3 uses a configurable bitmask (MMX_MASK, default 0x3F00) within the 32-bit packet mark to encode routing decisions. The mask determines how many interfaces can be supported and which mark values are reserved.

Mark Layout (default 0x3F00)

Bits	Mask	Owner	Purpose
0-7	`0x000000FF`	free	-
8-13	`0x00003F00`	mwan3	interface/policy marks
14-15	`0x0000C000`	free	-
16-23	`0x00FF0000`	pbr	policy routing marks
24-31	`0xFF000000`	free	-

Value	Meaning	With 0x3F00
0	Unmarked (needs classification)	`0x0000`
1 .. N	Interface marks (N = max interfaces)	`0x0100` .. depends on mask
mmdefault-2	MM_BLACKHOLE	Routes to blackhole
mmdefault-1	MM_UNREACHABLE	Routes to unreachable
mmdefault (all bits set)	MMX_DEFAULT (= MMX_MASK)	`0x3F00` = use default routing

Bit Spreading: `mwan3_id2mask()`

Interface IDs (sequential integers 1, 2, 3...) are mapped onto the mask bits using mwan3_id2mask(). This "spreads" the ID's bits into only the positions where the mask has a 1-bit. For example, with mask 0x3F00:

Interface 1 (binary 000001) -> 0x0100   (bit 8 set)
Interface 2 (binary 000010) -> 0x0200   (bit 9 set)
Interface 3 (binary 000011) -> 0x0300   (bits 8+9)
Interface 5 (binary 000101) -> 0x0500   (bits 8+10)

nftables Mark Manipulation

The iptables operation -j MARK --set-xmark VALUE/MASK means mark = (mark & ~MASK) | VALUE. In nftables this becomes:

meta mark set meta mark & COMPLEMENT | VALUE

where COMPLEMENT = ~MASK & 0xFFFFFFFF. The helper mwan3_nft_mark_expr() generates this expression.

Warning

Operator syntax: Always use the & and | symbols, not the and/or keywords. The nft parser treats keywords ambiguously after expressions like meta mark set ct mark - it cannot tell if and starts a new match or a bitwise operation. Symbols are unambiguous.

Connmark Operations

mwan3's connmark save and restore are scoped to its own bit-range (MMX_MASK) so that mark bits owned by other packages are never disturbed in either direction.

The natural nftables expression for a masked restore would be:

meta mark set (meta mark & ~MMX_MASK) | (ct mark & MMX_MASK)

The kernel rejects this with "Operation not supported": an nft set-statement can reference at most one runtime source register on its right-hand side.

vmap-dispatch save/restore

mwan3 synthesises the masked-restore and masked-save from primitives the kernel does allow: OR-ing a literal immediate into a single register (restore), or masking and OR-ing in a single expression (save).

meta mark set meta mark | <constant>                   # restore: only sets bits, never clears
ct mark set ct mark & MMX_MASK_COMPLEMENT | <constant>  # save: atomic clear+set in one expression

The runtime source value is bridged to the constant immediate via a verdict map (vmap) that dispatches on the masked source bits into per-mark setter chains:

# Restore: copy mwan3 bits from ct mark into meta mark, non-destructively
meta mark & MMX_MASK == 0 ct mark & MMX_MASK vmap {
    0x0100 : jump mwan3_or_meta_0x100,
    0x0200 : jump mwan3_or_meta_0x200,
    ...
    0x3f00 : jump mwan3_or_meta_0x3f00
}

chain mwan3_or_meta_0x0100 { meta mark set meta mark | 0x0100 ; return }
chain mwan3_or_meta_0x0200 { meta mark set meta mark | 0x0200 ; return }
...

# Save: atomic clear+set via vmap dispatch
meta mark & MMX_MASK vmap {
    0x0100 : jump mwan3_or_ct_0x0100,
    ...
}

chain mwan3_or_ct_0x0100 { ct mark set ct mark & MMX_MASK_COMPLEMENT | 0x0100 ; return }
...

Properties:

Restore is purely additive (meta mark | <imm>); bits in meta mark not owned by mwan3 are preserved across restore.
Save atomically clears mwan3's own bits and sets the new value in a single nft expression (ct mark & MMX_MASK_COMPLEMENT | <imm>, where MMX_MASK_COMPLEMENT = ~MMX_MASK & 0xFFFFFFFF). The ct mark goes directly from old-value to new-value with no intermediate state visible to other CPUs. Bits owned by other packages survive the save unchanged.
The dispatch tables are bounded: with the default MMX_MASK = 0x3F00, there are 63 possible non-zero mwan3 mark values, so 63 setter chains per direction (126 total). All chains are 2-statement skeletons (a mark-set expression + return).
The technique is order-independent: whether mwan3's hook fires before or after another package's hook in the prerouting stack, both packages' mark bits arrive at the routing decision intact.

Why this works where a direct expression does not: the kernel constraint is on the expression, not the control flow. The kernel will not let one rule combine two register sources, but it will happily let a vmap dispatch on a runtime register value into a chain whose body uses a literal immediate. The dispatch chain materialises at runtime exactly the value you wanted to OR, baked into a constant when the mwan3 init scripts emit the rule.

Cost: chain count, not packet-path overhead. Each packet traverses one extra vmap lookup (O(log) in the kernel's set lookup) and one extra jump/return per save and per restore. The 126 setter chains add no fast-path cost - they are visited via constant-time dispatch, and most never fire on any given packet.

Note

Compound two-source bitwise set expressions are rejected by the kernel. vmap-dispatch is built from primitives that are permitted and does not require any new kernel capability.

3. table inet mwan3 Architecture

mwan3 operates in its own standalone nftables table, table inet mwan3. This table is completely independent of fw4: fw4 reloads, restarts, and reconfiguration do not affect it. The table is created at service start from mwan3-skeleton.nft using the atomic table-replace idiom, and torn down at service stop.

Static Objects (from mwan3-skeleton.nft)

Object	Type	Purpose
`mwan3_connected_v4`	set (ipv4_addr, interval, auto-merge)	Directly connected IPv4 networks
`mwan3_connected_v6`	set (ipv6_addr, interval, auto-merge)	Directly connected IPv6 networks
`mwan3_custom_v4`	set (ipv4_addr, interval, auto-merge)	Networks from routing tables in UCI `globals.rt_table_lookup`
`mwan3_custom_v6`	set (ipv6_addr, interval, auto-merge)	Networks from routing tables in UCI `globals.rt_table_lookup`
`mwan3_dynamic_v4`	set (ipv4_addr, interval, auto-merge)	IPv4 CIDRs from UCI `globals.bypass_network`
`mwan3_dynamic_v6`	set (ipv6_addr, interval, auto-merge)	IPv6 CIDRs from UCI `globals.bypass_network`
`mwan3_prerouting`	chain (filter, prerouting, mangle+1)	Entry point for forwarded/incoming traffic
`mwan3_output`	chain (route, output, mangle+1)	Entry point for locally-originated traffic
`mwan3_postrouting`	chain (nat, postrouting, srcnat-1)	Opt-in IPv6 SNAT for router-originated rerouted traffic - see §4
`mwan3_ifaces_in`	chain (regular)	Dispatches to per-interface chains
`mwan3_rules`	chain (regular)	User-defined classification rules
`mwan3_connected`	chain (regular)	Marks traffic to connected networks as default
`mwan3_custom`	chain (regular)	Marks traffic to custom-table networks as default
`mwan3_dynamic`	chain (regular)	Marks traffic to dynamic networks as default

Note

auto-merge flag: All sets include the auto-merge flag in addition to interval. This allows nftables to merge overlapping elements (e.g., a host address and a containing CIDR) automatically, preventing insertion failures. The nft add set command is idempotent for creation but does not update flags on existing sets - mwan3_ensure_nft_framework() deletes and recreates the six internal sets at startup to guarantee the flag is present.

Dynamic Objects (created at runtime, in table inet mwan3)

Object Pattern	Type	Created By
`mwan3_iface_in_<name>`	chain	`mwan3_create_iface_nft()`
`mwan3_policy_<name>`	chain	`mwan3_create_policies_nft()`
`mwan3_rule_<name>`	chain	`mwan3_set_user_nft_rule()` (sticky rules only)
`mwan3_or_meta_<mark>`	chain (×63 with default `MMX_MASK`)	`mwan3_build_or_chains_nft()` - non-destructive restore setter chains
`mwan3_or_ct_<mark>`	chain (×63 with default `MMX_MASK`)	`mwan3_build_or_chains_nft()` - non-destructive save setter chains
`mwan3_sticky_v4_<rule>_<id>`	set (ipv4_addr, timeout)	`mwan3_set_user_nft_rule()` - one set per policy member (id = interface id)
`mwan3_sticky_v6_<rule>_<id>`	set (ipv6_addr, timeout)	`mwan3_set_user_nft_rule()` - one set per policy member (id = interface id)
`<name>` (user-defined)	set (ipv4_addr or ipv6_addr, interval, auto-merge)	`mwan3_render_config_ipsets()` from `config ipset` UCI sections

Note

Why type route for output? The output chain uses type route (not type filter) because changing a packet's mark on locally-originated traffic must trigger a routing re-lookup. This matches fw4's own mangle_output chain type.

Note

fw4 reload is a non-event for mwan3. fw4 reload rewrites only table inet fw4. table inet mwan3 is untouched. No recovery, detection, or rebuild is needed on fw4 reload.

4. Packet Flow Through Chains

The same logical flow applies to both mwan3_prerouting and mwan3_output, with one difference: prerouting includes an IPv6 RA bypass at the top.

Packet enters mwan3_prerouting (or mwan3_output)
  |
  |-- [prerouting only] ICMPv6 RA/NS/NA/redirect? --> ACCEPT (bypass)
  |
  |-- mark & MMX_MASK == 0?
  |     |
  |     +-- YES: Restore mwan3 bits from conntrack via vmap-dispatch
  |     |        (ct mark & MMX_MASK -> jump mwan3_or_meta_<mark>;
  |     |         non-destructive - preserves non-mwan3 bits in meta mark)
  |     |
  |     +-- Still mark == 0?
  |           |
  |           +-- jump mwan3_ifaces_in
  |           |     Per-interface chains check source address:
  |           |       - src in connected/custom/dynamic? -> mark = MMX_DEFAULT
  |           |       - otherwise -> mark = interface mark
  |           |
  |           +-- [prerouting only] Still mark == 0? fib daddr type local? --> RETURN
  |           |
  |           +-- Still mark == 0?
  |           |     jump mwan3_custom    (dst in custom sets? -> MMX_DEFAULT)
  |           |     jump mwan3_connected (dst in connected?   -> MMX_DEFAULT)
  |           |     jump mwan3_dynamic   (dst in dynamic?     -> MMX_DEFAULT)
  |           |
  |           +-- Still mark == 0?
  |                 jump mwan3_rules     (user classification rules)
  |                   -> jump to policy chain / set mark directly
  |
  |-- Save mwan3 bits to conntrack via vmap-dispatch:
  |     meta mark & MMX_MASK -> jump mwan3_or_ct_<mark>
  |     (atomic clear+set in one expression; preserves non-mwan3 bits in ct mark)
  |
  |-- mark & MMX_MASK != MMX_DEFAULT?
  |     (Traffic that got a specific interface mark, not "default")
  |     Re-check against custom/connected/dynamic destinations
  |     This allows connected-destination traffic to be overridden
  |     back to default routing even if it was marked by user rules
  |
  +-- ACCEPT (policy accept; packet continues to routing decision)

The Three Bypass Set Groups

mwan3 maintains three parallel groups of destination-bypass sets. Each group has a v4 and v6 set, a corresponding regular chain (jumped from both mwan3_prerouting and mwan3_output), and matching rules in every mwan3_iface_in_* chain. They differ only in how their sets are populated:

Set group	Populated by	Source
`mwan3_connected_v4/v6`	`mwan3rtmon` (continuous) and `mwan3_set_connected_sets()` at startup	CIDR routes in the kernel's main routing table
`mwan3_custom_v4/v6`	`mwan3_set_custom_sets()` at startup	Prefixes from routing tables listed in UCI `globals.rt_table_lookup`
`mwan3_dynamic_v4/v6`	`mwan3_set_dynamic_sets()` at startup; also writable at runtime via `nft add element`	UCI `globals.bypass_network` CIDR list

Each group appears in two distinct contexts with different match directions:

Destination match - mwan3_connected / mwan3_custom / mwan3_dynamic chains: These chains are jumped from mwan3_prerouting and mwan3_output before the user rules chain. Rules match ip daddr @set / ip6 daddr @set. If the packet's destination falls in one of the sets, the packet is stamped MMX_DEFAULT (mwan3 mark bits cleared to zero) and returned - the user rules chain is never reached. Traffic destined for directly-connected, custom-table, or explicitly listed bypass networks is not policy-routed.

Source match - mwan3_iface_in_* chains: Each per-interface chain includes source-match rules against all three set groups (ip saddr @set, scoped to the interface's address family). If an inbound WAN packet's source address is in any of the sets, it is stamped MMX_DEFAULT. Traffic arriving on a WAN interface from a connected or bypassed address does not get the WAN's interface mark, so replies to it use the main routing table rather than the WAN-specific policy route.

Routes wide enough to be a component of a default route are never swept into the connected or custom sets. A destination prefix of /2 or shorter for IPv4, or /3 or shorter for IPv6, is treated as default-equivalent and excluded by every producer that fills those sets: the startup seeding in mwan3_set_connected_sets() and mwan3_set_custom_sets(), and each of mwan3rtmon's three sweeps. Each route is judged on its own, so a client that has installed only half of a 0.0.0.0/1 plus 128.0.0.0/1 pair is excluded just the same. The 224.0.0.0/3 element of mwan3_connected_v4 is unaffected: it is added directly rather than swept from a route, and the IPv4 threshold stops at /2.

Without this exclusion a single such route would silently disable mwan3 for its whole address family. The bypass chains stamp MMX_DEFAULT before the user rules and again after them, so a covering route in the connected set would override rule steering, sticky pinning and a fail-closed last_resort policy used as a kill switch, and the source-match rules in the mwan3_iface_in_* chains would strip inbound reply-path classification from every WAN at once. No genuine network route is as wide as these thresholds. A deliberately broad bypass is still available, but it has to be asked for explicitly through globals.bypass_network, which populates the mwan3_dynamic sets and is never swept from the routing table.

Per-Interface Chain Detail

Each mwan3_iface_in_<name> chain handles packets arriving on a specific WAN device:

iifname and meta nfproto match - only processes packets arriving on the correct physical device and address family. The meta nfproto ipv4/meta nfproto ipv6 guard is critical when two mwan3 interfaces share the same physical device (e.g. dual-stack PPPoE): without it, an IPv4 interface's catchall would misclassify incoming IPv6 packets.
Source in bypass sets → MMX_DEFAULT - if the arriving packet's source is in mwan3_connected_v4/v6, mwan3_custom_v4/v6, or mwan3_dynamic_v4/v6, the packet is marked MMX_DEFAULT. This connection will use the main routing table for replies rather than being pinned to this WAN's policy route.
Otherwise → interface mark - the packet is stamped with the interface's unique fwmark. The conntrack save step in the calling prerouting/output chain then records this mark so subsequent packets in the same connection have their mark restored from ct mark.
Address-family-scoped catchall - the rule that marks unmatched packets with the interface fwmark carries a meta nfproto guard matching the interface's configured family, completing the dual-stack isolation begun in step 1.

Router-Originated Traffic and Source Address Rewriting

When the router itself originates a packet (via mwan3_output), the kernel binds the source address at sendto() time using the initial, unmarked route lookup. mwan3's mark is not set at that point, so the kernel picks the source address corresponding to whichever WAN the unmarked default route points to - call it WAN-A. Later in the egress path, mwan3_output sets a mark and the kernel performs a re-lookup (because the chain is type route) that may move the outgoing interface to WAN-B. The re-lookup updates oif but does not rewrite the source address - that was already committed. The packet leaves WAN-B carrying WAN-A's source address and is dropped upstream by BCP38 or uRPF filtering.

mwan3track is unaffected by this problem: it sets SO_BINDTODEVICE at socket creation, which forces the correct source address at bind time before any of this occurs.

IPv4: fw4's srcnat_wan masquerade rule applies to all outgoing traffic including locally-originated packets. When a rerouted packet reaches the srcnat hook, masquerade picks the primary IP of the actual outgoing interface and rewrites the source address correctly. No per-interface SNAT rule is needed from mwan3 - fw4 handles it automatically.

IPv6: fw4 does not masquerade IPv6 by default, so there is no automatic safety net. mwan3 provides an opt-in per-interface UCI option snat6 to address this. Blanket NAT66 is deliberately not the default for several reasons: RFC 6724 source-address selection with SADR routing tables can solve the problem without NAT in correctly-configured dual-stack deployments; NAT66 is harmful in ULA+delegated-PA topologies and breaks address-embedding protocols; some upstreams (tunnel brokers, fixed-address WireGuard endpoints) require a specific source address; and the IPv6 community treats address translation as an explicit opt-in.

The snat6 option accepts three values:

Value	Meaning
unset / `0`	No IPv6 SNAT (default)
`1`	SNAT to the interface's primary global address, looked up via `mwan3_get_src_ip`
`<v6 addr>`	SNAT to the literal address - used for fixed-source pinning from a delegated prefix

When snat6 is set, mwan3 installs a rule in the mwan3_postrouting base chain:

oifname "<dev>" meta nfproto ipv6
    meta mark & MMX_MASK == <iface_mark>
    fib saddr type local
    ip6 saddr != <iface_src_ip>
    snat to <iface_src_ip>

The fib saddr type local guard limits the rule to router-originated traffic. Stale rules are cleaned up by tag (mwan3_snat_<iface>) in mwan3_create_iface_nft() and mwan3_delete_iface_nft(). The literal-address form is not validated against the egress interface - some deployments deliberately pin a source from a delegated prefix not directly configured on the device.

snat6 only addresses the router-originated rerouted case. It does not extend mwan3's IPv6 capability to forwarded LAN traffic, SADR integration, or NPTv6 prefix translation.

The LuCI control for snat6 is described in Section 15.1.2.

5. IP Rules and Lookup Tables

mwan3 uses Linux policy routing as the actual packet-steering mechanism. The nftables chains (Section 4) write a firewall mark onto each packet; the kernel's ip rule database then selects the right routing table based on that mark. The nftables side and the policy-routing side are independent: nftables writes marks, the kernel routes according to them.

5.1 Per-Interface Routing Tables

Each mwan3 config interface section is assigned a sequential integer ID by mwan3_update_iface_to_table(). Interface sections are enumerated in UCI declaration order; the first enabled interface is ID 1, the second is ID 2, and so on. The ID is both the routing table number and the input to mwan3_id2mask() that generates the interface's mark value (see Section 2). Routing tables 1 through MWAN3_INTERFACE_MAX (derived from MMX_MASK) are reserved for mwan3 and should not be used for other purposes.

Each per-interface routing table is built by mwan3_create_iface_route() from two sources:

CIDR routes from the main routing table (host routes are excluded)
Routes from any additional tables listed in UCI globals.rt_table_lookup

The MWAN3_ROUTE_LINE_EXP sed expression strips transient attributes (linkdown, offload, expiry timestamps, error codes) before copying routes. Routes whose device matches the mwan3 interface's network device keep their original dev clause; the interface's default gateway (default via ... dev ...) is the critical entry that directs all policy-routed traffic to the correct WAN path.

Every mwan3-managed interface must carry a literal default route out of its own device in its own per-interface table: 0.0.0.0/0 for an IPv4 interface, ::/0 for an IPv6 one. This is a requirement rather than a recommendation. mwan3 is a multi-WAN manager and its interfaces are by definition whole-internet paths that policies choose between, so an interface that cannot act as a default route cannot do the job mwan3 exists to give it. The requirement is capability, not use: the interface must be able to carry the whole internet even if no rule ever steers everything to it. For a WireGuard interface this means allowed_ips '0.0.0.0/0' (and ::/0 for an IPv6 interface) together with route_allowed_ips '1', which installs a literal default route on the interface.

An interface whose own table holds no literal default route out of its own device is taken offline, deliberately. The route monitor evaluates each tracked interface's table and writes the verdict to the interface's ROUTE_STATUS file, logging the loss at warn level as <interface> default route down. The tracker reads that file once per round and treats down as a failed round, so the ordinary score and hysteresis carry the interface offline, and back online when the route returns. An interface that reads offline with that message in the log is not misbehaving; it is this requirement being enforced, and the remedy is to give the interface a default route.

Some VPN clients install routes that cover the whole internet without being a default route, and the shape is often the provider's choice rather than yours. OpenVPN's redirect-gateway def1 installs 0.0.0.0/1 and 128.0.0.0/1 in place of a default, split-tunnel WireGuard configurations do the same, and some IPv6 providers push the global-unicast prefix 2000::/3. None of these is a literal default route, so none of them satisfies the requirement. Where the client configuration is under your control, prefer plain redirect-gateway to redirect-gateway def1, which installs a genuine default. Where it is not, such as a server-pushed def1 pair or a server-pushed 2000::/3, add a literal default route to the interface yourself as a static route in the interface's configuration; the pushed routes are left untouched and the interface then satisfies the route-state check.

5.2 Four ip Rule Tiers Per Interface

mwan3_create_iface_rules() installs the Tier 1, Tier 2 and Tier 4 ip rule entries for each mwan3 interface. For IPv4 interfaces these are installed via ip rule; for IPv6 via ip -6 rule, with Tier 4 installed for IPv4 interfaces only. All three are installed when the interface comes up (ifup or connected hotplug event) and removed when it goes down (ifdown).

Tier 3 (the per-WAN unreachable backstop) has a separate lifecycle managed by mwan3_set_member_backstops(). One Tier 3 rule is installed for every configured interface at service start and on every reload, and removed only when the service stops. The Tier 3 rule is therefore present for all configured WANs, including those that are currently offline or disabled, for as long as the service is running.

Tier 1 - iif lookup (priority id + iif_rule_base, default id + 1000):

ip rule add pref <id+1000> iif <device> lookup <id>

Matches packets arriving on this interface's network device, regardless of mark. Routes them by looking up the interface's own routing table. This handles traffic that arrived on a WAN interface and needs to exit the router via the same WAN - the interface's table contains the WAN's default gateway. Without this rule, reply packets from the router (such as responses to health-check probes from the WAN's remote host) would reach the routing decision without a mark, fall through to the main table, and potentially exit via a different WAN.

Tier 2 - fwmark lookup (priority id + fwmark_rule_base, default id + 2000):

ip rule add pref <id+2000> fwmark <mark>/<MMX_MASK> lookup <id>

The primary steering rule. Matches packets whose mark, after masking with MMX_MASK, equals this interface's mark value (computed by mwan3_id2mask(id, MMX_MASK)). Routes them through the interface's per-interface routing table, which contains the WAN's default gateway. This is the rule that causes policy-classified traffic to exit via the correct WAN.

Tier 3 - fwmark unreachable (priority id + unreachable_rule_base, default id + 3000):

ip rule add pref <id+3000> fwmark <mark>/<MMX_MASK> unreachable

Matches the same fwmark as Tier 2 but returns ICMP unreachable instead of routing. Sits below Tier 2 in priority (higher priority number = lower precedence). Serves as a safety net: if the Tier 2 lookup finds an empty or incomplete routing table (e.g. the default route is transiently absent during route installation), the packet receives an explicit unreachable response instead of silently falling through to the main routing table and potentially leaking out a different WAN.

Tier 4 - source address lookup (priority id + unreachable_rule_base + MWAN3_INTERFACE_MAX + 1, default id + 3061, IPv4 only):

ip rule add pref <id+3061> from <address> lookup <id>

Matches router-originated traffic by its source address rather than by mark or by arrival device. mwan3_create_iface_rules() delegates to mwan3-manage-rules.uc add-src, which dumps the interface's addresses and installs one rule per global-scope IPv4 address carried on the device, all at the same priority, so a WAN holding several addresses gets several rules.

The tier exists because a sender that performs its own route lookup outside netfilter's view never acquires an mwan3 mark, so Tier 2 cannot steer it. Its packets follow the main table out of the default WAN even though the source address they carry belongs to a different WAN, and are dropped upstream by BCP38 filtering or masqueraded to the wrong address. WireGuard is the common case: a handshake arriving on a non-default WAN is answered from that WAN's address but leaves by the default WAN, so the handshake never completes. Any locally originated flow that no user rule classifies shares the gap. With the source rule in place the route lookup resolves back to the arrival WAN's own table and the reply keeps a source address that is valid on the path it takes.

The tier sits below Tier 3 in priority, so marked traffic still matches Tier 2 or the Tier 3 backstop first and policy verdicts and the fail-closed backstops behave exactly as they did without it. The rules are inert for forwarded traffic, whose source is a downstream address, and for a single-WAN setup.

5.3 Global Policy Rules

mwan3_set_general_rules() installs two rules that are not tied to any specific interface. These handle the mark values reserved for policy last_resort actions:

Blackhole rule (priority MM_BLACKHOLE + fwmark_rule_base):

ip rule add pref <priority> fwmark <MMX_BLACKHOLE>/<MMX_MASK> blackhole

Silently drops packets marked with MMX_BLACKHOLE. Applied when a policy chain marks traffic with this value via a last_resort blackhole policy option.

Unreachable rule (priority MM_UNREACHABLE + fwmark_rule_base):

ip rule add pref <priority> fwmark <MMX_UNREACHABLE>/<MMX_MASK> unreachable

Returns ICMP unreachable for packets marked with MMX_UNREACHABLE. This is the default last_resort for policies with no available members.

With MMX_MASK = 0x3F00 (6 mask bits): MM_BLACKHOLE = 61, priority 61 + 2000 = 2061; MM_UNREACHABLE = 62, priority 62 + 2000 = 2062. Both fall within the fwmark tier, above all per-interface Tier 2 entries.

5.4 MMX_DEFAULT Fall-Through

Packets marked with MMX_DEFAULT (= MMX_MASK, all mask bits set) do not match any per-interface fwmark rule or the global blackhole/unreachable rules. They fall through to the kernel's standard policy database, which includes the main routing table at priority 32766. Traffic stamped MMX_DEFAULT is routed normally via the main table without any mwan3 policy steering. This is the intended path for traffic destined for directly-connected, custom-table, or dynamically bypassed networks (see the bypass set groups in Section 4).

5.5 Configurable Rule Base Priorities

The three rule-base offsets are configurable in UCI config globals:

UCI option	Default	Priority range used (60 interfaces, default MMX_MASK)
`iif_rule_base`	1000	1001 - 1060 (one per interface)
`fwmark_rule_base`	2000	2001 - 2062 (per-interface + 2 global)
`unreachable_rule_base`	3000	3001 - 3060 (one per interface)
none, derived	3061	3062 - 3121 (Tier 4, one or more per IPv4 interface)

The Tier 4 source band has no UCI option of its own. It is derived as unreachable_rule_base + MWAN3_INTERFACE_MAX + 1 + id, so it moves with unreachable_rule_base and always sits immediately above the unreachable band, separated from it by one unused priority (3061 with the defaults).

Two ordering constraints must hold at startup:

iif_rule_base + MWAN3_INTERFACE_MAX < fwmark_rule_base
fwmark_rule_base + MWAN3_INTERFACE_MAX + 1 < unreachable_rule_base

The + 1 in the second constraint accounts for the global blackhole and unreachable entries at the top of the fwmark tier (MM_BLACKHOLE + fwmark_rule_base and MM_UNREACHABLE + fwmark_rule_base). If either constraint is violated, mwan3_init logs a warning and reverts all three to 1000/2000/3000.

mwan3_delete_iface_rules() removes the Tier 1, Tier 2 and Tier 4 rules by delegating to mwan3-manage-rules.uc delete-iface, which locates each rule by its priority offset and table ID and removes it via RTM_DELRULE; a Tier 4 rule is identified by carrying a source address at the source-band priority, so every address rule the interface holds goes in the one pass. This approach remains correct if the bases or MMX_MASK are changed between service restarts. The Tier 3 unreachable backstop is not deleted here; it persists until service stop.

5.6 Example: Live ip rule Output

Three WANs are configured on this router: eth1 (id 1) and eth2 (id 2) are online, while a third interface (id 3) is configured but currently offline.

0:      from all lookup local
1001:   from all iif eth1 lookup 1
1002:   from all iif eth2 lookup 2
2001:   from all fwmark 0x100/0x3f00 lookup 1
2002:   from all fwmark 0x200/0x3f00 lookup 2
2061:   from all fwmark 0x3d00/0x3f00 blackhole
2062:   from all fwmark 0x3e00/0x3f00 unreachable
3001:   from all fwmark 0x100/0x3f00 unreachable
3002:   from all fwmark 0x200/0x3f00 unreachable
3003:   from all fwmark 0x300/0x3f00 unreachable
3062:   from 192.0.2.2 lookup 1
3063:   from 198.51.100.2 lookup 2
32766:  from all lookup main
32767:  from all lookup default

The two entries at 3062 and 3063 are the Tier 4 source rules, one per global-scope IPv4 address on each online WAN's device. Router-originated traffic that carries no mwan3 mark, such as a WireGuard reply, reaches them with the WAN address it was sourced from and is routed back out that WAN's own table instead of following the main table out of the default WAN.

Interface id 3 shows the effect of the permanent Tier 3 backstops. It is offline, so it has no Tier 1 (iif) rule, no Tier 2 (fwmark 0x300/0x3f00 lookup 3) rule and no Tier 4 source rule; those are installed only while the interface is up, so there is no 1003, 2003 or 3064 entry. Its Tier 3 rule at priority 3003 is still present, because the backstops are installed for every configured WAN at service start and removed only at service stop. Traffic still marked 0x300 for the offline WAN therefore matches 3003 and is rejected as unreachable instead of falling through to the main table at priority 32766 and leaking out the default route. The backstop matches on the fwmark rather than on a device, which is why it can remain in place while the interface has no device bound.

The corresponding per-interface routing table (ip route show table 1) contains the WAN gateway and a copy of connected routes:

default via 192.0.2.1 dev eth1
192.0.2.0/24 dev eth1 proto kernel scope link src 192.0.2.2
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1

The priority gap between 2002 and 2061 is intentional: per-interface Tier 2 entries occupy priorities 2001-2060 (with the default 60-interface capacity) and the global entries occupy 2061-2062 at the top of the fwmark tier.

6. File Reference

6.1 `lib/mwan3/mwan3-skeleton.nft` [static]

The static nftables skeleton file. Loaded by start_service() via nft -f /lib/mwan3/mwan3-skeleton.nft before any dynamic rule installation. Uses the canonical atomic table-replace idiom:

table inet mwan3
delete table inet mwan3
table inet mwan3 { ... }

The three statements execute as one atomic nft -f transaction: the first ensures the table exists so the delete can succeed, the second wipes it, and the third recreates it fresh. This is idempotent: safe to run whether the table already exists or not.

Defines 6 named sets (all empty, flags interval and auto-merge) and 8 skeleton chains (all empty). No rules are present - all rules are added dynamically because they depend on the configurable MMX_MASK value.

The hook chains:

mwan3_prerouting - type filter at priority mangle + 1
mwan3_output - type route at priority mangle + 1 (type route is required so mark mutations trigger a routing re-lookup for locally-originated traffic)
mwan3_postrouting - type nat at priority srcnat - 1. Opt-in IPv6 SNAT chain. See §4.

6.2 `lib/mwan3/common.sh`

Shared helper library sourced by all mwan3 shell scripts. Provides:

Tool variables: $IP4, $IP6, $NFT
IPv6 detection: Checks /proc/sys/net/ipv6 existence (instead of the old command -v ip6tables)
MWAN3_BATCH_DEPTH counter: Integer depth counter (default 0). mwan3_nft_batch_start increments it, truncating the batch file only at depth 0->1. mwan3_nft_batch_commit decrements it, committing to the kernel only at depth 1->0. mwan3_nft_exec routes to mwan3_nft_push when depth > 0, accumulating all operations into the global batch.
MWAN3_NEED_DNSMASQ_HUP flag: integer flag that gets set to 1 if a user nft set has been deleted and recreated as a result of a change to one of the flags. After mwan3_write_dnsmasq_fragments, if the flag is set then a call will be made to mwan3_dnsmasq_hup().
Batch file: Per-process temp file /tmp/mwan3_nft_batch.$$ (PID-scoped, avoids conflicts between concurrent mwan3 instances).
nft batch helpers: mwan3_nft_batch_start(), mwan3_nft_push(), mwan3_nft_batch_commit()
mwan3_nft_reload_start(): Opens the outermost batch level (depth 0->1), then writes a preamble that (1) flushes all 8 skeleton chains, (2) two-pass flush+delete all dynamic chains (mwan3_iface_in_*, mwan3_policy_*, mwan3_rule_*, mwan3_or_meta_*, mwan3_or_ct_*), (3) deletes the 6 internal mwan3 sets so mwan3_ensure_nft_framework can recreate them. User-defined sets and sticky sets are never in the delete path.
mwan3_nft_reload_commit(): Thin wrapper around mwan3_nft_batch_commit. At depth 1->0 this commits the entire accumulated batch atomically. On failure the kernel rolls back and the old ruleset continues serving.
mwan3_nft_exec(): Wrapper that runs nft commands with error logging; routes to mwan3_nft_push when MWAN3_BATCH_DEPTH > 0.
mwan3_nft_mark_expr(): Generates nftables mark-set expressions equivalent to iptables --set-xmark. Outputs meta mark set meta mark & COMPLEMENT | VALUE using &/| symbols (not and/or keywords).
mwan3_ensure_nft_framework(): Guarantees all mwan3 nftables objects exist with correct flags. When called inside a batch (MWAN3_BATCH_DEPTH > 0), skips the direct delete loop (deletes already in the preamble batch). Recreates all 6 internal sets with interval + auto-merge flags and all skeleton chains in table inet mwan3.
mwan3_or_chain_suffix(): Converts a numeric mark value to the canonical lowercase 0x%x hex string used as the suffix for OR-immediate setter chain names (e.g. 0x100 for interface 1 with default mask). Called by mwan3_build_or_chains_nft(), mwan3_or_vmap_body(), mwan3_all_marks(), and mwan3_create_policies_nft().
mwan3_or_vmap_body(): Builds the body string for a vmap statement dispatching on masked mark values into OR-immediate setter chains. Called by mwan3_set_general_nft().
mwan3_all_marks(): Enumerates every mark value that needs to appear in the restore/save vmaps: all per-interface marks plus MMX_DEFAULT, MMX_BLACKHOLE, and MMX_UNREACHABLE. Echoes a space-separated list. Used by mwan3_set_general_nft().
mwan3_build_or_chains_nft(): Materialises the per-mark setter chains used by the non-destructive vmap-dispatch save/restore. Iterates all 63 possible non-zero values within MMX_MASK and emits two chains per value: mwan3_or_meta_<imm> (non-destructive restore from ct mark to meta mark) and mwan3_or_ct_<imm> (atomic clear+set save from meta mark to ct mark, non-destructive to bits outside MMX_MASK). Called from mwan3_set_general_nft(). Always flushes and re-populates the chain bodies - an earlier idempotency check that returned early on chain existence alone could leave the chain bodies empty after a partial-failure first run, which is fatal to packet flow.
mwan3_init(): Loads config, computes mask constants (MMX_DEFAULT, MMX_BLACKHOLE, MMX_UNREACHABLE, MMX_MASK_COMPLEMENT)
mwan3_id2mask(): Bit-spreading function that maps interface IDs onto the mask
mwan3_count_one_bits(): Counts set bits in a value
Utility functions: LOG(), readfile(), mwan3_get_src_ip(), mwan3_get_true_iface(), mwan3_get_mwan3track_status(), get_uptime(), get_online_time()

Note

Shell scoping note: Functions like mwan3_id2mask and mwan3_count_one_bits receive variable names as arguments (e.g., mwan3_id2mask mmdefault MMX_MASK) and use arithmetic expansion $(($1)) to resolve them. This works in busybox ash (OpenWrt's default shell) because it uses dynamic scoping - local variables from the caller are visible in called functions.

6.3 `usr/share/ucode/mwan3/common.uc`

Shared ucode library imported by the mwan3 ucode components, the ucode analogue of common.sh. Each shared ucode routine lives here so it is defined and tested in one place. Provides:

Logging (log_open(), log_verbose(), log_msg()): all mwan3 ucode logging goes through these so the sink, tag format, and verbose gate live in one place. Output goes to syslog via the log module, the same destination logger(1) reaches, so messages appear in logread regardless of how procd handles the process stderr. log_open(ident) opens the syslog connection with LOG_PID so the tag reads ident[pid], matching the shell's ${SCRIPTNAME}[$$] convention, under facility daemon. log_verbose(enable) sets the verbose gate; the caller reads mwan3.globals.verbose_logging and pushes the parsed bool. log_msg(level, msg) emits one message, dropping debug unless verbose exactly as common.sh:LOG() gates the debug facility, mapping mwan3's warn/error vocabulary and the syslog level names onto the tokens the log module accepts and passing the message as a %s argument so it is never treated as a format string.
ucibool(val): parses a UCI bool the way the shell's config_get_bool does (1/on/true/yes/enabled are true, everything else false), so every ucode consumer interprets a UCI bool identically to the shell. It reads no config itself, so it does not couple the module to uci.
is_default_equivalent(route): the width classifier for the bypass sweeps. Returns true when a route is a default in substance: no dst at all, or a prefix of /2 or shorter for IPv4, /3 or shorter for IPv6. The family is decided by iptoarr (inet_pton), so no address text is hand-parsed and a destination it cannot parse is not default-equivalent. Each route is judged alone, so half of a split pair is excluded on its own merits. The stricter literal-only predicate used by the route mirroring and ROUTE_STATUS paths is deliberately separate and is not widened.

The module's public interface is declared as an export list (export { log_open, log_verbose, log_msg, ucibool, is_default_equivalent }) rather than inline on each function, which this ucode build does not accept.

6.4 `lib/mwan3/mwan3.sh`

The core engine. Contains all functions for managing nftables chains/sets/maps, ip rules, ip routes, policy creation, user rule classification, and status reporting. This is the largest file and the heart of the implementation. Sourced by init.d, hotplug, CLI, and rtmon scripts.

See Section 7 for detailed function reference.

6.5 `etc/init.d/mwan3`

procd service script. Handles:

start_service(): Loads mwan3-skeleton.nft via nft -f first (aborts if this fails), then runs the full init sequence: ensure framework, render ipsets, dnsmasq fragments, trackers, sets, general rules, ifup hotplug loop, nft chains, policies, user rules, conntrack flush (if flow offloading), dnsmasq HUP, rtmons.
stop_service(): Guards with service_running || exit 0. Shuts down interfaces, flushes ip routes/rules, flushes all mwan3 nft chains, deletes dynamic chains (keeps skeleton chains), flushes sets, flushes and deletes sticky maps, then $NFT delete table inet mwan3 (complete removal).
reload_service(): Atomically rebuilds the entire mwan3 ruleset in a single kernel transaction via mwan3_nft_reload_start / all build functions / mwan3_nft_reload_commit. Then updates ip rules/routing tables (outside nft), updates dnsmasq fragments if changed, and checks tracker count - falls through to stop; start if tracker count mismatches (procd cannot add/remove service instances in reload).
start_tracker(): Launches a mwan3track procd instance per enabled interface with track IPs or track_gateway.
service_running(): Returns true if $MWAN3_STATUS_DIR exists.

Startup Sequence

nft -f /lib/mwan3/mwan3-skeleton.nft         # load standalone table (abort if fails)
mwan3_init()                                 # load UCI config, compute mask constants
mwan3_ensure_nft_framework()                 # recreate 6 internal sets, ensure chains
mwan3_render_config_ipsets()                 # create user-defined sets from config ipset
mwan3_write_dnsmasq_fragments()              # write nftset confdir fragments; restart dnsmasq if changed
config_foreach start_tracker interface       # launch health probes
mwan3_update_iface_to_table()                # build iface->table mapping
mwan3_set_dynamic_sets()                     # populate dynamic sets
mwan3_set_connected_sets()                   # populate connected sets
mwan3_set_custom_sets()                      # populate custom sets
mwan3_set_general_rules()                    # ip rule add (blackhole/unreachable)
mwan3_set_member_backstops()                 # ip rule add (per-WAN unreachable backstops)
config_foreach mwan3_ifup interface "init"   # trigger ifup hotplug per interface
wait $hotplug_pids
mwan3_set_general_nft()                      # populate hook chain rules
mwan3_set_policies_nft()                     # create policy chains
mwan3_set_user_rules()                       # populate user rules chain
mwan3_flush_stale_conntrack()                # flush zero-mark conntrack entries
mwan3_dnsmasq_hup()                          # HUP dnsmasq to re-populate nftset domains
[if flow_offloading=1] flush conntrack       # force flow re-establishment under new policy
start rtmon_ipv4 + rtmon_ipv6                # route monitor daemons

6.6 `etc/hotplug.d/iface/25-mwan3`

Handles interface state change events from netifd. Triggered on ifup, ifdown, connected, and disconnected actions.

Guard Checks

Valid action and interface name
Not first-connect or shutdown
Device present for ifup/connected
procd lock (unless called from init)
Service is running ($MWAN3_STATUS_DIR exists)
nft framework is loaded (nft list chain inet mwan3 mwan3_prerouting succeeds)
Interface is enabled in UCI

There is no fw4 reload detection. table inet mwan3 is unaffected by fw4 reloads; no recovery path is needed.

Actions

Action	Operations
`ifup`	Update peer track IP (if `track_gateway` enabled), create interface nft chain, create ip rules, create routes, set hotplug state. If not init: set general rules. If online and not init: rebuild policies (nft batch), flush worse-metric sticky sets, flush unreplied conntrack. Signal tracker with USR2.
`ifdown`	Set offline state. Rebuild policies + flush sticky sets (one nft batch). Delete ip rules, delete routes, delete interface nft chain. Signal tracker with USR1. Flush conntrack.
`connected`	Set online state. Create interface nft chain, create ip rules (no-op if already present), create routes (no-op if already present). Rebuild policies (nft batch). Flush worse-metric sticky sets. Flush unreplied conntrack.
`disconnected`	Set offline state. Rebuild policies + flush sticky sets (one nft batch). Flush conntrack.

All actions call mwan3_flush_conntrack at end.

Note

ifup conditional policy rebuild: During init (MWAN3_STARTUP=init), the ifup action skips general rules and policy rebuild because the init sequence handles those after all interfaces are up. Route creation runs unconditionally on every ifup, including during init. During normal operation, policies are only rebuilt if the interface state is "online" (not for interfaces with initial_state=offline).

6.7 `usr/sbin/mwan3` (CLI)

User-facing command-line tool. Provides start/stop/restart/ifup/ifdown commands plus status reporting: interfaces, policies, connected, rules, status (all combined), and internal (detailed dump).

The use command runs an arbitrary command bound to a specific interface using LD_PRELOAD with libwrap_mwan3_sockopt.so.

The internal command shows nft list table inet mwan3 output instead of the old iptables dump.

6.8 `usr/sbin/mwan3rtmon`

Route monitor daemon, reimplemented in ucode. Runs one instance per address family (ipv4/ipv6) as a procd service. Uses ucode-mod-rtnl for direct netlink access and ucode-mod-uloop for the event loop, eliminating all ip command fork+exec overhead from the original shell implementation.

Key improvements over the shell version:

Direct netlink route monitoring via rtnl.listener() instead of ip monitor route piped to a shell read loop
Structured route data from netlink messages instead of text parsing with sed/awk
O(1) per-event cost - the refactored handler avoids per-interface ubus calls, popen subprocesses, and UCI cursor creation on each route event. Device-to-table mapping and interface state are cached and refreshed only when needed
Debounced connected set rebuild - route add and delete events trigger a 100ms debounce timer rather than an immediate full set rebuild, coalescing bursts of route changes (e.g., during interface flap) into a single rebuild. Add events for the connected set additionally check to see if any change has occurred.
Proper event loop via uloop.run() instead of a shell pipe+read loop

On startup, it performs an initial synchronization: dumps the current routing table via netlink, populates the connected set, and replicates routes into active per-interface tables. It then enters the uloop event loop to process route change notifications asynchronously.

New route: Adds CIDR networks to the connected set via nft add element, then replicates the route into active per-interface tables. Host routes (bare IPs without prefix length) are skipped as they are remote destinations. Schedules via the debouncer.
Deleted route: Schedules a debounced connected set rebuild, then removes the route from per-interface tables.

6.9 `usr/share/rpcd/ucode/mwan3`

ucode RPC service exposing ubus methods under the mwan3 object. Used by LuCI for the web interface.

Uses nft -j (JSON output mode) for reliable parsing. All set/chain queries are scoped to table inet mwan3. Rule and route queries use ucode-mod-rtnl (RTM_GETRULE, RTM_GETROUTE) directly rather than spawning ip -j subprocesses.

mwan3.status: Returns JSON data for interfaces, connected networks, and policies.
- Connected IPs: Parses nft -j list set inet mwan3 mwan3_connected_v4/v6.
- Policies: Reads membership from UCI config and cross-references mwan3track STATUS files. Every member is always reported with traffic share percentage.
- Interfaces: Reads status from /var/run/mwan3track/ files and queries procd/netifd via ubus. Tracking IPs discovered by globbing TRACK_* files. Per-IP latency and packetloss populated when check_quality=1.
mwan3.nftset_members { set: "<name>" }: Returns members of a named nft set in table inet mwan3 as a flat array of strings. Correctly unwraps counter-decorated elements ({"elem":{"val":"...","counter":{...}}}) that nft emits for sets with the counter flag. Used by the Simulator tab for nftset rule matching and connected-network bypass detection.
mwan3.nftset_info {}: Returns the name, address-family type, counters flag, and runtime element count of all non-mwan3 sets in table inet mwan3. Used by the rule editor and IP Sets status tab.
mwan3.nftset_elements { set: "<name>", max: N }: Returns paginated elements of a named user-defined set with per-element packet/byte counters (when the set has counter enabled). Default maximum 200 elements; supports up to 5000. Used by the IP Sets status tab for runtime member display.
mwan3.resolve_host { host: "<name>", family: "" }: Resolves a hostname to IP addresses by invoking /bin/busybox nslookup against the local DNS server (127.0.0.1). Returns { v4: [...], v6: [...] }. The family parameter can be set to ipv4 or ipv6 to restrict resolution to A or AAAA records respectively; omit or leave empty for both. Querying via the local DNS server has the side effect of populating any dnsmasq nftset entries configured for the domain. Used by the Simulator tab to support hostname input in the source and destination fields.
mwan3.nftset_flush { set: "<name>" }: Flushes all elements from the named set via nft flush set inet mwan3 <name>. Returns {} on success or { error: "..." } on failure. Used by the Flush button on the IP Sets status tab.
mwan3.nftset_reload { set: "<name>" }: Flushes the set then re-adds all static entries from the UCI config ipset section (list entry values and the loadfile if configured). Returns {} on success or { error: "..." } on failure. Used by the Reload button on the IP Sets status tab.
mwan3.nftset_resolve { set: "<name>" }: Sends SIGHUP to dnsmasq to clear its cache, then queries each domain configured under the set's list domain option via the local dnsmasq instance (127.0.0.1). The DNS queries trigger dnsmasq's nftset= population mechanism as a side effect. Returns { resolved: N } where N is the count of domains that resolved successfully. Only applicable for sets with list domain entries. Used by the Resolve button on the IP Sets status tab.
mwan3.routing_health {}: Compares UCI configuration against live kernel state. Reads iif_rule_base, fwmark_rule_base, and unreachable_rule_base from UCI globals (with the same defaults and ordering constraint validation as mwan3_init), derives iface_max dynamically from mmx_mask, and derives the source base as unreachable_rule_base + iface_max + 1. Per interface, checks presence of the iif rule, fwmark lookup rule and unreachable rule, adds the source rule for interfaces of family ipv4 (tested against the IPv4 rule set alone, since the source band is IPv4 only), and reports routing table default route presence. Reports stale ip rules across all four priority tiers. Returns a rule_bases object, the derived source base included, so the frontend can display configured priorities dynamically rather than assuming fixed offsets.

6.10 `Makefile`

Package build recipe. Key dependency changes:

Old Dependency	New Dependency
`+ip`	`+ip-full`
`+ipset`	`+kmod-nft-core`
`+iptables`	`+nftables-json`
`+IPV6:ip6tables`	`+ucode`
`+iptables-mod-conntrack-extra`	`+ucode-mod-rtnl`
`+iptables-mod-ipopt`	`+ucode-mod-uloop`
	`+ucode-mod-uci`
	`+ucode-mod-ubus`
	`+ucode-mod-fs`
	`+ucode-mod-socket`
	`+ucode-mod-log`
	`+libnetfilter-conntrack`

PKG_BUILD_DEPENDS adds libnetfilter_conntrack and libmnl for the headers and libraries needed to compile mwan3ct.c. The runtime +libnetfilter-conntrack dependency provides the shared library that mwan3ct links against on the target.

mwan3ipcheck.c is also compiled in the same build step. It links only against libc (inet_pton, strtol, standard string functions) and requires no additional build or runtime dependencies.

The ucode dependencies are required by the reimplemented mwan3rtmon route monitor daemon. ucode-mod-socket is required by mwan3-diag for address normalisation. ucode-mod-log provides the syslog binding used by the shared logging functions in common.uc (Section 6.3).

Also installs:

mwan3-skeleton.nft to $(1)/lib/mwan3/
common.uc to $(1)/usr/share/ucode/mwan3/ (shared ucode library, see Section 6.3)
mwan3-migrate-ipset-v4.sh to $(1)/lib/mwan3/ (one-shot migration helper, deleted from the router after postinst runs it)
mwan3-remove-firewall-include UCI defaults to $(1)/etc/uci-defaults/ (removes legacy firewall.mwan3_reload UCI section from v3.x)
mwan3-lb-test to $(1)/usr/sbin/
mwan3ct to $(1)/usr/sbin/
mwan3ipcheck to $(1)/usr/bin/

The preinst script stops mwan3 before APK replaces any files. This handles the upgrade case where procd watches /etc/init.d/ via inotify and auto-starts mwan3 when the init script is replaced.

The postinst script:

Removes any mwan3 chains and sets still in table inet fw4 (from v3.x upgrades)
Conditionally reloads fw4 only if the legacy firewall.mwan3_reload UCI section exists (v3.x only; avoids triggering queued hotplug events on a clean v3.5 install)
Runs mwan3-migrate-ipset-v4.sh to copy config ipset sections from /etc/config/firewall to /etc/config/mwan3, then removes the migration script
Restarts rpcd
Stops mwan3 a second time (by this point procd's auto-start has completed, so this stop removes auto-start's ip rules)
Starts mwan3 cleanly

6.11 `usr/sbin/mwan3track`

Interface health probe daemon. One procd service instance is launched per enabled mwan3 interface that has tracking IPs configured. Runs as a shell script; largely unchanged from the iptables version except for the addition of track_gateway and check_quality support.

Probe methods

Configured via the track_method UCI option. Supported values: ping (default), arping, httping, nping-tcp/nping-udp/etc., nslookup. All probes are wrapped via LD_PRELOAD with libwrap_mwan3_sockopt.so (the WRAP helper), which intercepts setsockopt to set SO_BINDTODEVICE on the probe socket. This binds the probe to the physical interface device regardless of mwan3 routing marks, ensuring the probe exits on the correct WAN.

Score-based hysteresis

mwan3track maintains a score counter for each interface:

Each probe round: if host_up_count >= reliability (enough IPs responded), score increments; otherwise score decrements.
When score reaches up threshold from below: fires connected hotplug event.
When score reaches up threshold from above (on decline): fires disconnecting then disconnected hotplug events.
Score is clamped between 0 and down + up.

Once the reliability threshold is met in a round, remaining unprobed IPs are marked skipped - they are not probed further that round.

Status files

Written to $MWAN3TRACK_STATUS_DIR/<iface>/ (default /var/run/mwan3track/<iface>/):

File	Content
`STATUS`	Current state: `online`, `offline`, `connecting`, `disconnecting`, `disabled`
`SCORE`	Current score counter
`TURN`	Number of probe rounds completed
`LOST`, `ONLINE`, `OFFLINE`, `TIME`	Loss count, uptime timestamps
`TRACK_<ip>`	Per-IP probe result: `up`, `down`, or `skipped`. Always a status string, regardless of `check_quality`.
`LATENCY_<ip>`	[check_quality=1 only] Latency in ms for this IP from the most recent probe round.
`LOSS_<ip>`	[check_quality=1 only] Packet loss as a percentage for this IP from the most recent probe round.
`GATEWAY`	Gateway IP written by `mwan3_update_peer_track_ip()` when `track_gateway=1`. Read by `mwan3_load_track_ips()` and prepended to the probe list.

check_quality

When check_quality=1, probes capture latency (ms) and packet loss (%) per IP. Three-state evaluation: fail (loss >= failure_loss OR latency >= failure_latency), pass (loss <= recovery_loss AND latency <= recovery_latency), grey zone (neither - probe result neither increments nor decrements score). Default thresholds (1000ms/500ms/40%/10%) are conservative, suited for monitoring without frequent false failovers.

Signal handling

procd sends SIGUSR1 (ifdown event) and SIGUSR2 (ifup event) to trigger immediate state transitions without waiting for the next probe interval.

6.12 `usr/sbin/mwan3-lb-test`

Load balancing distribution verifier.

mwan3-lb-test [-6] -c <client_ip> <policy_name> [ip1 ip2 ...]
mwan3-lb-test cleanup

Verifies that numgen-based load balancing produces the expected traffic distribution across policy members. Key design:

Iteration count (NITER): computed from member weights as base_N = total_weight / GCD(weights), NITER = base_N * ceil(30 / base_N). Ensures per-member expected counts are whole numbers and NITER >= 30 always.
Test rule: inserts a temporary ICMP-only rule into mwan3_rules matching a nft address set. Using ICMP prevents TCP/UDP traffic to the same IPs (DNS forwarders, Android clients bypassing local DNS, etc.) from contaminating the counter.
Client isolation (-c <client_ip>, required): inserts a forward chain drop rule blocking pings to the test set from all LAN clients except the nominated test client, and an mwan3_output return rule bypassing mwan3 marking for any router process pinging the same IPs. Both rules are scoped to the test set and removed by cleanup.
Destination pool: well-known public IPs. Excludes any IPs already configured as mwan3 track_ip values - mwan3track pings those via mwan3_output -> mwan3_rules, which would match the test rule and inflate counts.
IPv6 mode (-6): uses meta l4proto ipv6-icmp ip6 daddr @set, ping6, and a separate pool of well-known public IPv6 IPs.
IP overrides: mwan3-lb-test -c <client_ip> <policy> ip1 ip2 ... for sites where defaults are unreachable or fully tracked.
Windows command: outputs a cmd.exe for loop alongside the Linux shell loop. Windows ping uses a fixed ICMP identifier (id=1), causing conntrack entry reuse on repeated pings to the same destination; the Windows command uses a longer inter-ping delay (30/TRACK_COUNT + 3 seconds) so the full cycle exceeds the 30s ICMP conntrack timeout. The IP list is formatted with ^ line continuation at 4 IPs per line.
cleanup subcommand: removes stale mwan3_lb_test_* sets and rules left by a run that was killed before cleanup could execute.
Cleanup on exit: removes test set and rules on normal exit, SIGINT, SIGTERM, and SIGPIPE. Startup sweep removes stale artifacts from aborted prior runs.

See §17.1 for context on why this tool was added and the numgen contamination issues it was designed to detect.

6.13 `usr/sbin/mwan3-diag`

Network diagnostic report generator.

mwan3-diag

A ucode script that collects a comprehensive snapshot of the network state relevant to mwan3 operation and prints it to stdout. Intended to produce a complete, shareable report without manual redaction.

Collects: interface addresses, routing tables (including all per-WAN tables), policy rules, neighbour cache, mwan3 interface status, mwan3 UCI configuration, the complete mwan3 nftables ruleset, the fw4 mangle chains that interact with mwan3 packet marking, and the last 200 lines of the mwan3 log.

Before printing any output the script builds a map of every public routable IPv4 and IPv6 address present in the collected data and replaces each one with a stable placeholder -- PUB4_1, PUB4_2, PUB6_1 and so on -- throughout the entire report. The same address always receives the same placeholder, so cross-references between sections remain consistent. Private addresses (RFC1918, link-local fe80::, ULA fc00::/7, loopback) are left unchanged. The elements of user-defined nftables sets are replaced with { ... }.

See §17.2 for context.

6.14 `usr/sbin/mwan3ct`

Targeted conntrack entry flush helper.

mwan3 needs to flush conntrack entries at multiple places in the code. These flushes are designed to be as targeted as possible in order to leave other unrelated conntrack entries unmolested.

In some cases, the conntrack CLI tool is unable to perform the necessary targeted flush. For example, mwan3 needs to flush only UNREPLIED conntrack entries when an interface comes online to break the cycle where a stale mark pins traffic to the wrong WAN. conntrack -D silently ignores the -u status filter because the delete path in conntrack-tools omits the status filter from the kernel request, making UNREPLIED-only deletion impossible via the CLI.

The workaround is to parse conntrack -L output in shell, extract 5-tuple fields from each matching entry, and issue a separate conntrack -D call per entry. However, this is sensitive to output format changes and requires multiple subprocesses per entry.

Additionally, flushing all mwan3-marked entries with conntrack requires a shell loop spawning one conntrack process per mark value (up to 63 for the default 6-bit mask).

mwan3ct is a small C binary that calls libnetfilter_conntrack's NFCT_Q_FLUSH_FILTER directly, passing mark and status filters to the kernel in a single netlink message. The kernel performs the filtered delete atomically - no dump, no iteration, no per-entry delete is needed.

mwan3ct flush [--mark <val>/<mask>] [--mark-any <mask>] [--status <val>/<mask>]

--mark <val>/<mask>: delete entries with an exact mark match. Single kernel call.
--mark-any <mask>: delete entries with any non-zero mark value within the mask. Internally loops over all possible mark IDs (up to 63 for the default 6-bit mask), issuing one kernel call per ID. All within a single process and netlink socket.
--status <val>/<mask>: filter by conntrack status bits. For example, --status 0/0x2 matches entries where IPS_SEEN_REPLY (0x2) is not set, i.e. UNREPLIED entries.
--mark and --mark-any are mutually exclusive. At least one filter option is required.
--mark and --status filters can be combined.

Exit code 0 on success, including when no entries were matched. Errors are printed to stderr.

6.15 `lib/mwan3/mwan3-get-addr.uc`

Address lookup helper.

Queries the kernel via RTM_GETADDR (netlink) to find an IP address in one of two modes:

By device: given <family> <device>, returns the first suitable address on that device. For IPv6, skips link-local (fe80::/10) and non-global-scope addresses. For IPv4, accepts scope 0 (global) and scope 253 (link).
By prefix: given <family> "" <prefix>, returns the first global-scope address whose string representation begins with the prefix. Used for IPv6 prefix matching.

Prints the address stripped of any prefix-length to stdout and exits 0, or exits 1 if no matching address is found. Replaces ip -f inet[6] addr show dev <device> shell parsing.

6.16 `lib/mwan3/mwan3-manage-rules.uc`

Routing policy rule manager.

Manages ip rules (policy routing entries) via netlink (RTM_GETRULE, RTM_NEWRULE, RTM_DELRULE, RTM_GETROUTE, RTM_GETADDR). Supports six modes:

check <family> <prio1> [prio2...]: returns a decimal bitmask where bit N is set if the Nth listed priority is absent from the live rule table. Used by mwan3.sh to decide which rules need to be created.
check-route <family> <table_id> <device>: exits 0 if a default route for the given output device exists in the specified table, 1 otherwise. Used before populating per-interface tables to detect whether the route is already present.
delete-iface <id> <iif_base> <fwmark_base> <mmx_mask> <src_prio>: removes the iif lookup, fwmark lookup and source lookup rules for a single interface (identified by its sequential index). Each rule is located by priority and verified by type before deletion, a source rule by carrying a source address at src_prio, so every address rule the interface holds goes in the one pass; rules not present are silently skipped. The unreachable backstop rule is not removed here; it is managed by mwan3_set_member_backstops() and persists until service stop.
add-general <family> <bh_prio> <bh_mark> <ur_prio> <ur_mark> <mask>: adds the shared blackhole and unreachable rules common to all interfaces. Rules already present at those priorities are not re-added.
add-backstop <family> <prio> <mark> <mask>: adds one per-interface fwmark unreachable rule at the given priority, the Tier 3 backstop. A rule already present at that priority is not re-added.
add-src <device> <table_id> <prio>: adds one source lookup rule per global-scope IPv4 address carried on the device, all at the given priority, the Tier 4 return rules. IPv4 only.

Replaces ip rule add/del shell invocations that required fork-exec per operation and text parsing for presence checks.

6.17 `lib/mwan3/mwan3-list-routes.uc`

Route table lister.

Dumps routes from a given routing table via RTM_GETROUTE and prints destination prefixes to stdout, one per line. Accepts a numeric table ID or the special value main. Filters out: default-equivalent routes, host routes (prefix-length 32/128), link-local prefixes (fe80::/10, 169.254.0.0/16), IPv4 multicast (first octet >= 224), and duplicate destinations.

A route is default-equivalent when it has no destination at all, or its prefix is /2 or shorter for IPv4, /3 or shorter for IPv6. That covers the literal defaults 0.0.0.0/0 and ::/0 and the covering forms some VPN clients install in their place, such as a 0.0.0.0/1 plus 128.0.0.0/1 pair or the IPv6 global-unicast prefix 2000::/3; see the bypass set groups in Section 4. The test is is_default_equivalent() from common.uc (Section 7.2).

Used by mwan3.sh to populate the mwan3_connected_v4/v6 sets from the main table and the mwan3_custom_v4/v6 sets from each table listed in globals.rt_table_lookup.

6.18 `lib/mwan3/mwan3-create-iface-route.uc`

Per-interface routing table populator.

Copies routes from the main routing table (and any additional tables listed in the rt_table_lookup UCI global) into a single per-interface routing table via RTM_NEWROUTE with NLM_F_CREATE | NLM_F_REPLACE. Uses ubus (network.interface dump) to map interface names to their kernel devices and table IDs; the _4/_6 suffix convention for dual-stack interface pairs is recognised. Routes already present in the target table (matched by dst/oif/gateway/priority) are skipped. When source_routing is set, the route's source address is included in the copy.

Replaces ip route add table <id> shell invocations that required fork-exec and text parsing of ip route show output.

6.19 `usr/bin/mwan3ipcheck`

IP address and CIDR validation binary. Accepts a single argument and prints one of four classification strings to stdout, exiting 0 on success or 1 on failure.

Output	Exit	Meaning
`ipv4`	0	All elements are valid IPv4 addresses or IPv4 CIDR prefixes
`ipv6`	0	All elements are valid IPv6 addresses or IPv6 CIDR prefixes
`mixed`	1	Comma-separated list contains a mix of IPv4 and IPv6 elements
`invalid`	1	Any element failed validation, input is empty, or argument count is wrong

The argument may be a single address, a single CIDR prefix, or a comma-separated list of addresses and prefixes. Leading and trailing whitespace around each comma-separated token is stripped before classification.

Address validation uses inet_pton() - first trying AF_INET, then AF_INET6. Any address accepted by mwan3ipcheck is guaranteed to be accepted by nft, which uses the same parser internally.

CIDR validation: when a / is present, the prefix length is validated against the family maximum (32 for IPv4, 128 for IPv6). Out-of-range values and leading zeros in the prefix length are rejected. A trailing / with no digits is rejected.

Comma-separated lists: each token is classified individually. If all tokens are the same family the list returns that family. If tokens span both families the result is "mixed", which the rule-building path rejects since a single nft anonymous set cannot contain both IPv4 and IPv6 elements.

The binary has no dependencies beyond libc (inet_pton, strtol, and standard string functions only).

7. Function Reference

7.1 common.sh Functions

Function	Purpose
`LOG facility message...`	Logs to syslog. Suppresses `debug` level by default.
`mwan3_nft_exec args...`	Runs `nft` with arguments, logs errors. Returns 1 on failure. When `MWAN3_BATCH_DEPTH > 0`, routes to `mwan3_nft_push` instead.
`mwan3_nft_batch_start`	Increments `MWAN3_BATCH_DEPTH`; truncates `/tmp/mwan3_nft_batch.$$` only at depth 0->1.
`mwan3_nft_push line`	Appends a line to the batch file.
`mwan3_nft_batch_commit`	Decrements `MWAN3_BATCH_DEPTH`; executes `nft -f /tmp/mwan3_nft_batch.$$` and removes the temp file only at depth 1->0.
`mwan3_nft_reload_start`	Opens outermost batch (depth 0->1) and writes preamble: flush 8 skeleton chains, two-pass flush+delete all dynamic chains, delete 6 internal sets. User-defined sets and sticky sets are untouched.
`mwan3_nft_reload_commit`	Thin wrapper around `mwan3_nft_batch_commit`. Commits entire accumulated batch atomically at depth 1->0. On failure, kernel rolls back and old ruleset continues.
`mwan3_nft_mark_expr value mask`	Outputs `meta mark set meta mark & COMPLEMENT \| VALUE`. Uses `&` and `\|` symbols (not keywords). Equivalent to iptables `--set-xmark VALUE/MASK`.
`mwan3_ensure_nft_framework`	Recreates the 6 internal mwan3 sets with `interval` + `auto-merge` flags (skipping the direct delete loop when inside a batch - deletes already in the preamble). Adds all 8 skeleton chains in `table inet mwan3`. Idempotent for chains.
`mwan3_build_or_chains_nft`	Builds the 126 per-mark setter chains used by vmap-dispatch save/restore (63 `mwan3_or_meta_<imm>` restore chains + 63 `mwan3_or_ct_<imm>` atomic clear+set save chains). Always flushes and re-populates chain bodies. See §2 Connmark Operations.
`mwan3_or_chain_suffix mark`	Converts a numeric mark value to the canonical lowercase `0x%x` hex string used as the suffix for OR-immediate setter chain names (e.g. `0x100` for interface 1 with default mask). Called by `mwan3_build_or_chains_nft`, `mwan3_or_vmap_body`, `mwan3_all_marks`, and `mwan3_create_policies_nft`.
`mwan3_or_vmap_body reg mark...`	Builds the body string for a vmap statement dispatching on masked mark values into OR-immediate setter chains. Called by `mwan3_set_general_nft()`.
`mwan3_all_marks`	Enumerates every mark value that needs to appear in the restore/save vmaps: all per-interface marks (IDs 1..`MWAN3_INTERFACE_MAX` bit-spread through `MMX_MASK`) plus `MMX_DEFAULT`, `MMX_BLACKHOLE`, and `MMX_UNREACHABLE`. Echoes a space-separated list. Used by `mwan3_set_general_nft()` to build the vmap body.
`mwan3_init`	Loads UCI config, creates status dirs, computes all mask constants (`MMX_MASK`, `MMX_DEFAULT`, `MMX_BLACKHOLE`, `MMX_UNREACHABLE`, `MMX_MASK_COMPLEMENT`, `MWAN3_INTERFACE_MAX`), and reads `iif_rule_base`, `fwmark_rule_base`, `unreachable_rule_base` from UCI globals into `MWAN3_IIF_RULE_BASE`, `MWAN3_FWMARK_RULE_BASE`, `MWAN3_UNREACHABLE_RULE_BASE`, reverting all three to defaults (1000/2000/3000) if either ordering constraint is violated.
`mwan3_id2mask id mask`	Bit-spreads `id`'s bits into positions where `mask` has 1-bits. Arguments are variable names (indirect evaluation).
`mwan3_count_one_bits var`	Counts 1-bits in the value named by `var` (indirect evaluation). Uses `n&(n-1)` trick.
`mwan3_get_true_iface out_var iface`	Resolves virtual interface names (appends _4 or _6 suffix if that interface exists in netifd).
`mwan3_get_src_ip out_var iface`	Gets the source IP for an interface, with fallbacks for IPv6-PD prefixes.
`readfile var path`	Reads entire file into variable. Returns 1 if file doesn't exist.
`mwan3_get_mwan3track_status out_var iface`	Returns tracker status: `disabled`, `down`, `paused`, or `active`.
`get_uptime [out_var]`	Returns system uptime in seconds (integer).
`get_online_time out_var iface`	Returns how long the interface has been online.

7.2 common.uc Functions

Functions in the shared ucode library usr/share/ucode/mwan3/common.uc (see Section 6.3), imported by the mwan3 ucode components.

Function	Purpose
`log_open(ident)`	Opens the syslog connection for the calling component with `LOG_PID` and facility `daemon`, so the tag reads `ident[pid]`, matching the shell components' `${SCRIPTNAME}[$$]` convention.
`log_verbose(enable)`	Sets the module-wide verbose gate. The caller reads `mwan3.globals.verbose_logging` (via `ucibool`) and pushes the parsed bool; `debug` messages are suppressed unless it is true, mirroring `common.sh:LOG()`.
`log_msg(level, msg)`	Emits one syslog message. `debug` is dropped unless verbose; every other level always logs. Maps mwan3's `warn`/`error` vocabulary and the syslog level names onto the tokens the `log` module accepts (an unknown level falls back to `notice`), and passes `msg` as a `%s` argument so it is never treated as a format string.
`ucibool(val)`	Parses a UCI bool the way the shell's `config_get_bool` does: `1`/`on`/`true`/`yes`/`enabled` are true, everything else false. Reads no config itself, so it does not couple the module to uci.
`is_default_equivalent(route)`	Returns true when a route is a default in substance: no `dst` at all, or a prefix of `/2` or shorter for IPv4, `/3` or shorter for IPv6. The family comes from `iptoarr` (inet_pton), so no address text is hand-parsed and an unparseable destination is not default-equivalent. Used by every producer that fills the connected and custom bypass sets.

7.3 mwan3.sh Functions

7.3.1 Set Management Functions

Function	Purpose
`mwan3_set_connected_ipv4`	Flushes and repopulates `mwan3_connected_v4` from main routing table. Adds `224.0.0.0/3` for multicast. Self-contained: starts and commits its own nft batch.
`mwan3_set_connected_ipv6`	Same for IPv6. Skips if `$NO_IPV6`. Self-contained batch.
`mwan3_set_connected_sets`	Calls both `_ipv4` and `_ipv6` functions.
`mwan3_set_custom_set table_id`	Callback for `config_list_foreach`. Adds routes from the given table to custom sets. Pushes to an existing batch (does not start/commit).
`mwan3_set_custom_sets`	Flushes and repopulates custom sets from all `rt_table_lookup` entries in globals config.
`mwan3_set_dynamic_network`	Callback for `config_list_foreach`. Classifies a single `bypass_network` CIDR as IPv4 (contains `.`) or IPv6 (contains `:`) and pushes the appropriate `add element` command to the current batch.
`mwan3_set_dynamic_sets`	Flushes `mwan3_dynamic_v4/v6` then repopulates from `bypass_network` entries in globals UCI config. Called at startup and in both fw4 reload recovery paths.

Note

Why are connected functions self-contained? mwan3_set_connected_ipv4/ipv6 each manage their own batch because they may be called from contexts that are not already inside a larger batch (init.d, hotplug). mwan3rtmon does not call these shell functions; it has its own independent populate_connected_set() that operates via direct netlink calls and writes to the same sets using its own nft_batch.

7.3.2 General Rule Setup

Function	Purpose
`mwan3_set_general_rules`	Adds `ip rule` entries for blackhole and unreachable marks (both IPv4 and IPv6) at priorities `MWAN3_FWMARK_RULE_BASE + MM_BLACKHOLE` and `MWAN3_FWMARK_RULE_BASE + MM_UNREACHABLE`. These are ip policy rules, not nftables rules.
`mwan3_set_member_backstops`	Installs one static unreachable `ip rule` per configured interface in the `MWAN3_UNREACHABLE_RULE_BASE + id` priority band. These rules are created at service start and reload and removed only at stop, so traffic carrying a member's mark is rejected rather than leaking to the main routing table when that member's fwmark rule is absent (interface down, disabled, or briefly mid-rebuild). Reserved last-resort marks are never enumerated here so `last_resort` behaviour is unaffected.
`mwan3_set_general_nft`	Builds the per-mark OR setter chains via `mwan3_build_or_chains_nft()`, then populates all hook chain rules: IPv6 RA bypass (prerouting), vmap-dispatched connmark restore (if mark unset), jump to `mwan3_ifaces_in`, `fib daddr type local return` (prerouting, after ifaces_in), jumps to custom/connected/dynamic/rules chains, connmark save, and post-rules connected re-check. Idempotent: checks if rules already exist before adding. Uses a single batch for all operations.

Rules added by `mwan3_set_general_nft()`

For each of connected/custom/dynamic chains, adds mark-setting rules that match against the corresponding sets and apply MMX_DEFAULT.

For the prerouting and output hook chains, adds (in order):

[prerouting only] IPv6 RA bypass (accept ICMPv6 types: nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, nd-redirect)
Non-destructive connmark restore (if meta mark & MMX_MASK == 0): ct mark & MMX_MASK vmap { ... -> jump mwan3_or_meta_<imm> }. The setter chains contain meta mark set meta mark | <imm>, so non-mwan3 bits in meta mark are preserved across the restore.
Jump to mwan3_ifaces_in (if still 0)
[prerouting only] fib daddr type local return (if still 0): packets destined for the router's own IP return immediately. Placed after mwan3_ifaces_in so that WAN traffic arriving on a mwan3 interface is already marked by the iface_in catchall (mark != 0) and never reaches this check. Only traffic on non-WAN interfaces (LAN, loopback) with mark still zero reaches the fib local check. See §c4bcd951e.
Jump to custom, connected, dynamic chains (if still 0)
Jump to mwan3_rules (if still 0)
Atomic connmark save (always): meta mark & MMX_MASK vmap { ... -> jump mwan3_or_ct_<imm> } dispatches into setter chains that clear mwan3's bits and set the new value in a single nft expression (ct mark set ct mark & MMX_MASK_COMPLEMENT | <imm>). No intermediate state is visible to other CPUs. Bits in ct mark owned by other packages survive unchanged.
Post-rules: jump to custom/connected/dynamic again for non-default marks

See §2 Connmark Operations for the rationale and the kernel-limitation context.

7.3.3 Interface Management

Function	Purpose
`mwan3_update_iface_to_table`	Populates `mwan3_iface_tbl`: a space-separated string of `name=id` pairs for all configured interfaces, where `id` is the sequential table number. Called lazily (on first use) by `mwan3_get_iface_id()`. Also called explicitly during `start_service()` to prime the cache before interface chains are created.
`mwan3_get_iface_id out_var iface`	Looks up `mwan3_iface_tbl` for the given interface name and writes its table ID into the named output variable. Calls `mwan3_update_iface_to_table()` on first use if the cache is empty.
`mwan3_create_iface_nft iface device`	Creates (or flushes) `mwan3_iface_in_<iface>` chain. Adds rules matching on `iifname` and address family: source in connected/custom/dynamic → MMX_DEFAULT; otherwise → interface mark. Adds jump from `mwan3_ifaces_in` if not already present. Also installs a per-interface `mwan3_postrouting` SNAT rule for IPv6 interfaces when the `snat6` UCI option is set. Stale `mwan3_snat_<iface>`-tagged rules from a prior incarnation are removed first. See §4.
`mwan3_rebuild_iface_nft iface`	Rebuilds a single interface's nft chain if the interface is enabled, the correct family is available, and the interface is currently up (verified via ubus). Used during `reload_service` to rebuild all interface chains within the atomic batch. Calls `mwan3_create_iface_nft()` after resolving the L3 device from netifd.
`mwan3_delete_iface_nft iface`	Removes the jump rule from `mwan3_ifaces_in` (by handle lookup), removes any `mwan3_snat_<iface>`-tagged rules from `mwan3_postrouting` (comment-tag match), then flushes and deletes the interface chain.
`mwan3_delete_iface_map_entries iface`	Iterates all `mwan3_sticky_v[46]_*` sets in the `inet` family, finds sets whose name ends with `_<id>` (this interface's id), and pushes a flush for each into the caller's open nft batch, so the flushes commit in the same transaction as the policy rebuild. The set enumeration reads committed kernel state and so stays outside the batch. Sets are flushed rather than deleted because rule chains may still reference the set name. Must be called within an open nft batch.
`mwan3_create_iface_rules iface device`	Adds `ip rule` entries: pref `id+MWAN3_IIF_RULE_BASE` (iif lookup) and pref `id+MWAN3_FWMARK_RULE_BASE` (fwmark lookup). For an IPv4 interface it also delegates to `mwan3-manage-rules.uc add-src` for the source lookup rules at pref `id+MWAN3_UNREACHABLE_RULE_BASE+MWAN3_INTERFACE_MAX+1`, one per global-scope IPv4 address on the device. The per-WAN unreachable backstop at `id+MWAN3_UNREACHABLE_RULE_BASE` is managed separately by `mwan3_set_member_backstops()` and is not installed here.
`mwan3_rebuild_iface_rules iface`	Rebuilds a single interface's ip rules and routing table if the interface is enabled, its family is available, and it is currently up with an L3 device (verified via ubus). Called from `reload_service` via `config_foreach`, after the nft batch has committed, because ip rules and routes are kernel state that nftables does not own. Calls `mwan3_create_iface_rules()`, which deletes the interface's existing rules before adding them, then flushes and repopulates the routing table via `mwan3_delete_iface_route()` and `mwan3_create_iface_route()`. Unlike `mwan3_rebuild_iface_nft()` it does not consult the hotplug state file, so an interface that is up at netifd but tracked down keeps its routing in place.
`mwan3_delete_iface_rules iface`	Removes the iif lookup, fwmark lookup and source lookup ip rules for this interface (both IPv4 and IPv6) by delegating to `mwan3-manage-rules.uc delete-iface`. The ucode script locates each rule by priority offset and table ID and deletes it via RTM_DELRULE. Handles rule-base changes and `mmx_mask` changes transparently. The unreachable backstop at `id+MWAN3_UNREACHABLE_RULE_BASE` is not deleted here; it persists until service stop.
`mwan3_create_iface_route iface`	Copies routes from the main routing table (and any `rt_table_lookup` extra tables) into the per-interface routing table by invoking `mwan3-create-iface-route.uc`. The ucode script reads source routes via rtnl, skips routes already present in the target table, and writes each qualifying route with RTM_NEWROUTE. Passes the `source_routing` UCI flag to control whether source addresses are copied.
`mwan3_delete_iface_route iface`	Flushes the per-interface routing table. Unchanged.

Handle-Based Rule Deletion

nftables doesn't support deleting rules by match criteria (like iptables -D chain match...). Instead, mwan3_delete_iface_nft() uses:

handle=$($NFT -a list chain inet mwan3 mwan3_ifaces_in | \
    grep "jump mwan3_iface_in_$1" | sed -n 's/.*# handle \([0-9]*\)/\1/p')
$NFT delete rule inet mwan3 mwan3_ifaces_in handle "$handle"

The -a flag shows rule handles in comments, which can then be used for targeted deletion.

7.3.4 Policy & Load Balancing

Function	Purpose
`mwan3_set_policy member_config`	Callback per policy member. Tracks lowest metric per address family and accumulates online members as `iface:id:weight` tuples into `$policy_members_v4` and `$policy_members_v6`. Uses caller's variables (dynamic scoping).
`mwan3_create_policies_nft policy`	Creates/flushes the `mwan3_policy_<name>` chain. Iterates members via `mwan3_set_policy`, then builds the chain: single online member gets a direct mark-set rule; multiple online members get a `numgen` rule; last-resort rule (unreachable/blackhole/default) is appended.
`mwan3_set_policies_nft`	Before creating policy chains, enumerates all existing `mwan3_policy_*` chains in `inet mwan3` and deletes any whose name is not in the current UCI policy config (orphaned chain sweep). Then iterates all policy configs and calls `mwan3_create_policies_nft` for each.

7.3.5 Sticky Routing

Function Purpose

mwan3_get_policy_members_for_family policy family Iterates the members of a policy config via config_list_foreach. For each member whose interface matches the requested family, resolves the interface id and computes its mark, and accumulates <id>:<mark> tuples into $_policy_member_marks. Used by the sticky path in mwan3_set_user_nft_rule() to enumerate per-member sets.

mwan3_flush_worse_metric_sticky iface On recovery of a higher-priority interface, flushes the per-rule sticky sets of lower-metric same-family members of each policy the recovered interface participates in, scoped to the rules that use that policy. Set names embed the rule name and member id, so a member that appears in a separate load-balancing policy with equal-ranked peers keeps those pins untouched. Called after the policy rebuild on the connected and ifup (online) hotplug arms.

Function	Purpose
`mwan3_get_policy_members_for_family policy family`	Iterates the members of a policy config via `config_list_foreach`. For each member whose interface matches the requested family, resolves the interface id and computes its mark, and accumulates `<id>:<mark>` tuples into `$_policy_member_marks`. Used by the sticky path in `mwan3_set_user_nft_rule()` to enumerate per-member sets.
`mwan3_flush_worse_metric_sticky iface`	On recovery of a higher-priority interface, flushes the per-rule sticky sets of lower-metric same-family members of each policy the recovered interface participates in, scoped to the rules that use that policy. Set names embed the rule name and member id, so a member that appears in a separate load-balancing policy with equal-ranked peers keeps those pins untouched. Called after the policy rebuild on the `connected` and `ifup` (online) hotplug arms.

7.3.6 User Rules

Function	Purpose
`mwan3_set_user_nft_rule rule ipv`	Builds and adds a single nft rule to `mwan3_rules`. Translates UCI config options (proto, src_ip, dest_ip, src_port, dest_port, src_iface, ipset, ipset_src) into nft match expressions. For sticky rules, calls `mwan3_get_policy_members_for_family()` and builds a `mwan3_rule_<name>` chain with per-member address sets and OR-immediate restore/save (see §8). Handles logging rules. Pre-creates missing nft sets (e.g., dnsmasq nftsets that haven't started yet) to prevent batch failures. When `family` is explicitly `ipv4` or `ipv6` but none of `src_ip`, `dest_ip`, `ipset`, or `ipset_src` are set, prepends `meta nfproto ipv4/ipv6` to the nft match to prevent a bare rule from matching both address families. Also translates `proto icmp` to `meta l4proto ipv6-icmp` when `family` is `ipv6` (in an inet table, proto 1 is ICMPv4 only). `src_ip` and `dest_ip` are validated via `mwan3ipcheck`; a result of `"invalid"` or `"mixed"` causes the rule to be skipped with a warning log. When a comma is present in `src_ip` or `dest_ip`, the value is wrapped in nft anonymous set syntax (`{ addr1, addr2 }`) for the generated match expression.
`mwan3_set_user_rules`	Flushes `mwan3_rules` chain, then iterates all rule configs in config order, calling `mwan3_set_user_nft_rule`for both IPv4 and IPv6. Uses a single batch.
`mwan3_set_user_iface_rules iface device`	Called on ifup to check if the rules chain needs rebuilding (if this interface is a `src_iface` in any rule).

UCI-to-nft Match Translation

UCI Option	nft Expression
`proto tcp`	`meta l4proto tcp`
`src_ip 10.0.0.0/8`	`ip saddr 10.0.0.0/8` (or `ip6 saddr`)
`src_ip 10.0.0.1,10.0.0.2`	`ip saddr { 10.0.0.1, 10.0.0.2 }`
`dest_ip 1.2.3.4`	`ip daddr 1.2.3.4`
`dest_ip 1.2.3.4,5.6.7.8`	`ip daddr { 1.2.3.4, 5.6.7.8 }`
`src_iface lan`	`iifname "br-lan"`
`src_port 80,443`	`th sport { 80, 443 }`
`dest_port 8080`	`th dport { 8080 }`
`ipset my_set`	`ip daddr @my_set`
`ipset_src my_set`	`ip saddr @my_set`
`use_policy balanced`	`jump mwan3_policy_balanced`
`use_policy default`	`meta mark set ... \| MMX_DEFAULT`

Missing nft Set Pre-creation

When a user rule references an ipset (destination nft set) or ipset_src (source nft set) that doesn't exist yet in table inet mwan3, nft -f would roll back the entire batch atomically. To prevent this, mwan3_set_user_nft_rule() pre-creates any missing referenced sets with the appropriate type. The same logic applies to both ipset and ipset_src.

A set is pre-created at most once per render. Kernel queries inside an open batch see pre-batch state, so the renderer carries its own record of the sets it has already added to the batch, as name=addrtype tokens reset at the start of each render. The first rule to reference a missing set settles its address type for the whole render; a later rule referencing the same name from the other family is skipped, silently when it is a family=any rule and with a warning when its family is explicit. Without the record, two rules of different families referencing one missing set each added it with a conflicting type, nft rejected the batch, and the router was left with no user rules at all.

7.3.7 User-defined nft Set Management

Function	Purpose
`mwan3_render_config_ipsets`	Iterates all `config ipset` UCI sections and calls `_mwan3_render_one_ipset` for each. Creates user-defined sets in `table inet mwan3`. Called from `start_service` and `reload_service`.
`_mwan3_nft_time_to_sec str`	Converts an nft human-unit timeout string (e.g. `1h`, `5m`, `300s`) to an integer number of seconds. Recognised suffixes: `d` (days), `h` (hours), `m` (minutes), `s` (seconds, default if no suffix). Used by `_mwan3_ipset_needs_delete` to compare the live kernel timeout against the desired spec in a common unit.
`_mwan3_ipset_needs_delete name type counter timeout maxelem`	Returns 0 (true) if the named set exists in `table inet mwan3` AND its flags (type, counter, timeout, size) differ from the desired spec; returns 1 if the set is absent or all flags match. Called by `_mwan3_render_one_ipset` during reload to decide whether to queue a `delete set` before recreating the set with new flags. Timeout values are compared in seconds after converting nft's human-unit display (e.g. `1h`, `5m`) via `_mwan3_nft_time_to_sec`.
`_mwan3_render_one_ipset name`	Creates one user-defined nft set. Builds a `type ipv4_addr/ipv6_addr; flags interval;` declaration; appends `auto-merge;` only when no timeout is configured, since auto-merge coalesces contiguous addresses into merged ranges with a single expiry, silently discarding per-element TTLs in DNS-populated sets. Optionally appends `counter;` (standalone statement, NOT a flag keyword) and `timeout Ns;`. On the start path, unconditionally deletes any existing set before creating. On the reload path, calls `_mwan3_ipset_needs_delete` and only queues a `delete set` when flags differ; otherwise `add set` is idempotent and dnsmasq-populated elements survive. Sets `MWAN3_NEED_DNSMASQ_HUP` if a domain-populated set is deleted. Adds inline `list entry` elements and `loadfile` contents.
`mwan3_write_dnsmasq_fragments`	For each `config ipset` with `list domain` entries, writes a dnsmasq confdir fragment enabling `nftset=/domain/FAMILY#inet#mwan3#setname`. Compares against previously written fragments; restarts dnsmasq only when content changes. Silently does nothing if no domain sets are configured.
`mwan3_cleanup_orphaned_ipsets`	Queries `nft list table inet mwan3` for user-defined sets (those not prefixed `mwan3_`), compares against configured set names, and pushes `delete set` for any orphans. Called in `reload_service` after `mwan3_render_config_ipsets`. Prevents stale sets accumulating when a set is removed from UCI config.

7.3.8 Status Reporting

Function	Purpose
`mwan3_report_iface_status iface`	Shows interface online/offline status with uptime. Checks ip rules, nft chain existence in `table inet mwan3`, and default route presence.
`_mwan3_report_policies_for_family family`	Calls ubus `mwan3 status` for the policies section, then iterates the policy/member hierarchy for the given family (`ipv4` or `ipv6`), printing each policy name with its member interfaces and current traffic percentages. Called by `mwan3_report_policies_v4()` and `mwan3_report_policies_v6()`.
`mwan3_report_policies_v4/v6`	Delegates to `_mwan3_report_policies_for_family()` for IPv4 and IPv6 respectively.
`_mwan3_report_connected_set set_name`	Parses the named connected set using `nft -j list set inet mwan3 <set_name>` and `jshn.sh`. Iterates the JSON element array and prints prefix elements as `addr/len` and range elements as `start-end`. Handles both address families. Called by `mwan3_report_connected_v4()` and `mwan3_report_connected_v6()`.
`mwan3_report_connected_v4/v6`	Thin wrappers that call `_mwan3_report_connected_set` with `mwan3_connected_v4` and `mwan3_connected_v6` respectively. `mwan3_report_connected_v6` returns early if `$NO_IPV6` is set.
`mwan3_report_rules_v4/v6`	Parses `nft list chain inet mwan3 mwan3_rules`.
`mwan3_mark_to_name mark`	Resolves a numeric mark value to an interface name (or "default"/"blackhole"/"unreachable").

7.3.9 Lifecycle & Hotplug

Function	Purpose
`mwan3_ifup iface caller`	Resolves interface status via ubus, then triggers the 25-mwan3 hotplug script with `ACTION=ifup`. When called from init, runs in background.
`mwan3_interface_hotplug_shutdown iface [ifdown]`	Triggers ifdown or disconnected hotplug event for an interface.
`mwan3_interface_shutdown iface`	Calls hotplug shutdown then cleans track state files.
`mwan3_set_iface_hotplug_state iface state`	Writes state (`online`/`offline`) to status file.
`mwan3_get_iface_hotplug_state iface`	Reads state from status file (defaults to `offline`).
`mwan3_flush_conntrack iface action`	Two-path conntrack flush. First, iterates the `flush_conntrack` UCI list for this interface: for each configured action that matches the current hotplug action, writes `f` to `$CONNTRACK_FILE` to flush the entire global conntrack table. Second, on `ifdown` and `disconnected`, uses `mwan3ct flush --mark MARK/MMX_MASK` to selectively delete only the conntrack entries for this interface's fwmark. Both paths run; the global flush path runs first.
`mwan3_flush_marked_conntrack`	Flushes all conntrack entries that have any `MMX_MASK` bit set, iterating the full mwan3 id-space (IDs 1..`MWAN3_INTERFACE_MAX` plus default/blackhole/unreachable marks). Called from `reload_service` so that live flows re-enter the classification chains and are re-evaluated against new rules rather than staying pinned to a previously saved ct mark.
`mwan3_update_peer_track_ip iface`	If `track_gateway` is enabled, queries `ifstatus` for the point-to-point peer address and writes it to `$MWAN3TRACK_STATUS_DIR/<iface>/GATEWAY`.
`mwan3_track_clean iface`	Removes track status directory for the interface.
`mwan3_dnsmasq_hup`	Sends SIGHUP to running dnsmasq instances via `ubus call service signal '{"name":"dnsmasq","signal":1}'`. Called once from `start_service` after sets are created so dnsmasq resolves domain entries into nft sets. No longer uses mwan3evtd (which was removed ).
`mwan3_flush_stale_conntrack`	Flushes conntrack entries with no mwan3 mark (`0x0/MMX_MASK`) after mwan3 restart. New connections arriving during the brief startup window before iface_in chains exist get ct mark=0; this clears them so they re-establish correctly. Called from `start_service`.
`mwan3_flush_unreplied_conntrack`	Flushes all mwan3-marked conntrack entries that have never received a reply (`IPS_SEEN_REPLY` not set). Called after `mwan3_set_policies_nft` on the `connected` action and on `ifup` when `status=online`. Breaks the cycle where a stale mark from a previous policy state pins traffic to the wrong WAN: because no reply arrives the application keeps retrying, refreshing the conntrack timeout indefinitely. Flushing only UNREPLIED entries leaves established connections untouched.

7.4 mwan3rtmon Functions

usr/sbin/mwan3rtmon (see Section 6.8) is the ucode route-monitor daemon; one instance runs per address family (ipv4 and ipv6). It mirrors kernel routing changes into mwan3's connected and custom nft sets and into the per-interface routing tables, and maintains each tracked interface's ROUTE_STATUS file. Shared logging (log_open, log_verbose, log_msg), ucibool and is_default_equivalent are imported from common.uc (Section 7.2).

Function	Purpose
`main()`	Entry point. Parses the address family from `ARGV[0]` (`ipv4`/`ipv6`), opens syslog with `log_open`, loads config, connects to ubus, and binds the rtnl route listener before running the initial dumps so route changes during the dump phase are queued rather than lost to a subscribe-vs-dump race. Runs the initial custom-set, connected-set, per-interface-route, and route-state population, installs SIGTERM/SIGINT/SIGHUP handlers (SIGHUP reloads config and re-sweeps), arms a 20-second defensive route-state re-sweep (the nested `schedule_periodic_route_sweep()`), then enters the uloop event loop. One process runs per family.
`log_msg(level, msg)`	Local logging wrapper that prefixes the `[family]` tag and delegates to `common.uc`'s `log_msg` (imported as `mlog`).
`load_config()`	Loads the `mwan3` UCI config: sets the verbose gate via `log_verbose(ucibool(...))`, reads `mmx_mask` and `source_routing`, builds `extra_table_set` from `rt_table_lookup`, and builds `iface_list` (name, family, sequential table id, enabled, has_tracking) in UCI declaration order. Called at startup and on SIGHUP.
`build_dev_table_map()`	From the ubus `network.interface` dump, maps each enabled same-family interface's `l3_device` to its table id in `dev_table_map`.
`refresh_active_chains()`	Caches the set of existing `inet mwan3` chains from `nft -j list chains inet` into `active_chains`.
`is_iface_nft_active(ifname)`	Returns whether `mwan3_iface_in_<ifname>` is present in the cached chain set.
`get_active_tids()`	Returns a `{tid: name}` map of same-family interfaces whose per-interface nft chain is currently active.
`get_track_status(ifname)`	Reads the tracker's `PID` and `STARTED` status files and returns `disabled`, `down`, `paused`, or `active`, mirroring the shell tracker-status logic.
`nft_exec(cmd)`	Runs a single `nft` command, logging a warning with the command and its output on non-zero exit.
`nft_batch(cmds)`	Writes a list of commands to a per-family batch file, `/var/run/mwan3/mwan3rtmon_nft_batch.<family>`, and applies them atomically with `nft -f`, logging on failure. One instance runs per address family and each is single-threaded, so the family suffix alone keeps the two instances' files apart without locking; the file is removed on every exit path after creation.
`build_route_for_table(route, tid)`	Builds a route object targeting table `tid`, copying the relevant `ROUTE_FIELDS`, dropping `tos == 0`, and including `src` only when `source_routing` is enabled.
`apply_route_to_table(route, tid, is_new)`	Adds (`RTM_NEWROUTE`, create+replace) or deletes (`RTM_DELROUTE`) the route in table `tid` via rtnl, logging add failures.
`is_cidr_route(route)`	Returns true when the route's `dst` prefix length is shorter than the family maximum (32/128), i.e. a network route rather than a host route.
`is_default_route(route)`	Returns true for a null `dst`, `0.0.0.0/0`, or `::/0`. Route mirroring and the `ROUTE_STATUS` check rely on this narrow meaning, so it is deliberately left strict. The bypass sweeps use the wider `is_default_equivalent` from `common.uc` instead.
`is_linklocal_route(route)`	Returns true for a `dst` under `fe80::/` or `169.254.`.
`table_has_default(routes, tid, dev)`	Returns true if the route dump holds a default route in table `tid` via device `dev`. This is the datapath predicate behind `ROUTE_STATUS`: the interface's fwmark ip rule looks up exactly this table, so a table with no default black-holes the traffic marked to it even when the main table still has a default via the same device.
`tracked_interfaces()`	Returns `[{name, tid, dev}]` for the enabled, tracked, same-family interfaces currently bound to an l3 device (device from the ubus dump). Interfaces with no device yet are skipped and re-evaluated on a later route change.
`write_route_state(ifname, state)`	Atomically writes the interface's `ROUTE_STATUS` file (temp file + rename in the same tmpfs directory) only when the value changes, and logs the transition (`down` at warn, the return at notice).
`refresh_route_states()`	Recomputes every responsible interface's route state from one full route dump and persists it via `write_route_state`. Called at startup, on SIGHUP, from the periodic sweep, and debounced from the route-event handler.
`arm_route_state_refresh()`	Arms or reschedules the debounce timer that collapses a burst of route events into a single `refresh_route_states` sweep.
`get_all_routes()`	Dumps routes for this family and filters to the main table plus any `rt_table_lookup` extra tables.
`route_still_exists(route)`	Re-dumps and checks whether a route still exists in the main table; used to verify delete events before acting on them.
`populate_connected_set()`	Flushes and repopulates `mwan3_connected_v4/v6` from CIDR, non-default-equivalent, non-link-local routes (plus `224.0.0.0/3` for IPv4), gated by a sorted-element fingerprint so an unchanged set skips the nft write.
`populate_iface_routes()`	Copies main and extra-table routes into the active per-interface tables: a device-matched route goes to that interface's table, other network routes broadcast to all active tables.
`repopulate_custom_sets()`	Rebuilds `mwan3_custom_v4/v6` from the `rt_table_lookup` tables, excluding default-equivalent and link-local routes; used at startup and on SIGHUP reload.
`handle_custom_set_event(route, is_new)`	Mirrors a single `rt_table_lookup` route add or delete into `mwan3_custom_v4/v6` so those destinations receive `MMX_DEFAULT` and bypass WAN policy selection; default-equivalent and link-local routes are skipped.
`handle_route_event(event)`	The rtnl listener callback for `RTM_NEWROUTE`/`RTM_DELROUTE`: mirrors `rt_table_lookup` routes into the custom set, debounces the connected-set rebuild, arms the route-state refresh, and updates the per-interface tables, verifying delete events and gating broadcast routes on tracker status.

7.5 mwan3track Functions

usr/sbin/mwan3track (see Section 6.11) is the per-interface tracker; one shell process runs per tracked interface. It probes the interface's track_ip targets on an interval, maintains a hysteresis score, writes per-target and per-interface status files under $MWAN3TRACK_STATUS_DIR/<iface>/, and fires connected/disconnected (and the transitional connecting/disconnecting) hotplug events as the score crosses its thresholds. Signals drive its lifecycle: USR1 is ifdown, USR2 is ifup, HUP is config reload, TERM is stop.

Function	Purpose
`stop_subprocs`	Kills the tracker's background jobs (the running probe and sleep subshells). Used by the signal handlers to break out of the current round.
`WRAP cmd...`	Runs the probe command `cmd` under `LD_PRELOAD` of the mwan3 sockopt wrapper, exporting `FAMILY`, `DEVICE`, `SRCIP`, and `FWMARK` so the probe leaves via the interface's own address and is marked for its own table.
`clean_up`	`TERM` handler: logs the stop, kills subprocesses, and exits.
`if_down`	`USR1` handler: sets `IFDOWN_EVENT` and stops subprocesses so the loop registers the ifdown.
`if_up`	`USR2` handler: sets `IFUP_EVENT`, marks the tracker started, and stops subprocesses.
`config_reload`	`HUP` handler: sets `RELOAD_EVENT` and stops subprocesses so the loop reloads config.
`ping_test_host`	Echoes the family's loopback address (`127.0.0.1` or `::1`), used to test whether a ping variant works.
`get_ping_command`	Selects the usable ping binary/variant (`ping -4/-6`, `ping6`, or `/bin/ping`) and echoes it; returns non-zero if none is available.
`validate_track_method method`	Verifies the tool for the configured `track_method` (`ping`/`arping`/`httping`/`nping-*`/`nslookup`) is installed, logging a warning and returning non-zero if not.
`validate_wrap`	Ensures the `libwrap_mwan3_sockopt` shared library exists; logs an error and exits if it is missing.
`disconnected [suppress]`	Marks the interface offline: writes `STATUS`/`OFFLINE`/`ONLINE`, zeroes the score, and (unless `suppress` is `1`, as on first connect) fires the `disconnected` hotplug event when the previous status was online or disconnecting.
`connected [firstconnect]`	Marks the interface online: writes `STATUS`/`ONLINE`/`OFFLINE`, resets the per-round counters, and fires the `connected` hotplug event (passing `FIRSTCONNECT`).
`disconnecting`	Enters the transitional `disconnecting` state once and fires the `disconnecting` hotplug event, an early warning before the interface is declared offline.
`connecting`	Enters the transitional `connecting` state once and fires the `connecting` hotplug event, an early signal before the interface is declared online.
`disabled`	Marks the interface `disabled` and calls `stopped`.
`firstconnect`	Resolves the true interface, L3 device, and source IP, determines the initial status (`initial_state`, default online), marks the tracker started, then calls `connected` or `disconnected` accordingly. Called at startup and after ifup.
`stopped`	Sets `STARTED=0` and writes the `STARTED` status file; the tracker then idles until a signal.
`started`	Sets `STARTED=1` and writes the `STARTED` status file.
`update_status track_ip state [latency loss]`	Writes the per-target `TRACK_<ip>` status file (and, when supplied, `LATENCY_<ip>`/`LOSS_<ip>`) for one probe target.
`load_tracking_config`	Reads all per-interface tracking UCI options (family, track_method, reliability, count, timeout, interval, up/down thresholds, size, max_ttl, failure/recovery intervals, check_quality and its latency/loss thresholds), clamping `check_quality` off for track methods that produce no latency/loss samples. Called at startup and on `HUP` reload.
`main iface`	Entry point and probe loop. Takes the per-interface lock and writes `PID`, installs the signal traps, loads config and the probe-target list, then loops: each round it reads `ROUTE_STATUS` (a `down` written by `mwan3rtmon` counts as a failed round with no probing), probes each target with `WRAP`, adjusts the hysteresis score, and calls the state transitions; between rounds it services pending ifdown/ifup/reload events.

The nested helpers mwan3_load_track_ips and mwan3_list_track_ips, defined inside main(), assemble the probe-target list (static track_ip entries plus the discovered gateway when track_gateway is set) and prune stale per-target status files; they are not listed separately.

8. Load Balancing with numgen

The iptables version used -m statistic --mode random --probability P to distribute traffic. This required inserting rules in specific order and computing running probabilities. The nftables version uses numgen inc mod N map { ... }, which is simpler and more deterministic.

How numgen Works

numgen inc mod N generates a counter that increments on each packet and wraps at N. The map { range : value } maps counter values to marks.

# Example: wan (weight 3) + wanb (weight 2) = mod 5
# wan gets range 0-2 (3 values), wanb gets range 3-4 (2 values)

nft add rule inet mwan3 mwan3_policy_balanced \
    meta mark & 0x3f00 == 0 \
    meta mark set numgen inc mod 5 map { 0-2 : 0x0100, 3-4 : 0x0200 }

Warning

Kernel limitation on compound set expressions: An early implementation tried meta mark set meta mark & COMP | numgen inc mod ... to preserve non-mwan3 bits while applying the numgen result. This fails with "Operation not supported" because the kernel cannot mix two register sources (meta mark and numgen) in one set expression. The solution is to use meta mark set numgen ... directly; the meta mark & MMX_MASK == 0 guard condition ensures the mwan3 bits are already zero before the numgen result is applied.

Build Algorithm (in `mwan3_create_policies_nft`)

Iterate policy members via config_list_foreach
Track lowest metric per family (v4/v6 separately); only members at the lowest metric are included
Accumulate online members as iface:id:weight tuples
Calculate total weight = sum of all member weights
If single member: direct meta mark set (no numgen overhead)
If multiple members: build numgen map entries with ranges proportional to weight
Append last-resort rule (unreachable/blackhole/default)

Note

Difference from iptables version: The iptables version used probabilistic matching (--probability) which is statistically correct over many packets but can have short-term imbalance. The nftables numgen inc counter gives perfectly deterministic round-robin distribution at the configured weights.

9. Sticky Routing Detail

Sticky routing ensures that repeat connections from the same source IP use the same WAN interface (important for HTTPS sessions, banking sites, etc.). The iptables version used ipset hash:ip,mark sets. The nftables version uses per-member address sets with OR-immediate vmap dispatch for non-destructive mark restore.

Why Per-Member Sets Instead of a Single Map

With the vmap-dispatch infrastructure, the correct approach is a plain address set per policy member, with the mark encoded in the chain name rather than the map value. The corresponding mwan3_or_meta_<mark> setter chain ORs only the mwan3 bits into meta mark, preserving everything else.

Data Structure

# Created per policy member for rule "https"
# (policy "balanced" has members: wan id=1 mark=0x100, wanb id=2 mark=0x200)
nft add set inet mwan3 mwan3_sticky_v4_https_1 { type ipv4_addr; flags timeout; timeout 600s; }
nft add set inet mwan3 mwan3_sticky_v4_https_2 { type ipv4_addr; flags timeout; timeout 600s; }

One set per policy member, per rule, per address family. Sets hold source addresses only (no value side). The mark to apply is encoded in which set the saddr is found in.

Rule Chain Structure (for sticky rule "https", policy "balanced", wan=id1 wanb=id2)

chain mwan3_rule_https {
    # Restore: if saddr is in this member's set, OR its mark into meta mark
    # (non-destructive - pbr bits in meta mark are preserved)
    ip saddr @mwan3_sticky_v4_https_1 jump mwan3_or_meta_0x100
    ip saddr @mwan3_sticky_v4_https_2 jump mwan3_or_meta_0x200

    # New flows (no set match, mark still 0) fall through to policy
    meta mark & 0x3f00 == 0 jump mwan3_policy_balanced

    # Save: after policy assigns a mark, record saddr in the matching member's set
    meta mark & 0x3f00 == 0x100 update @mwan3_sticky_v4_https_1 { ip saddr timeout 600s }
    meta mark & 0x3f00 == 0x200 update @mwan3_sticky_v4_https_2 { ip saddr timeout 600s }
}

The chain is built once, complete for every family whose traffic can reach it. A rule with an explicit family, or with a family-anchoring match element (src_ip, dest_ip, ipset, ipset_src), reaches the chain in one address family only and gets that family's arms. A family=any rule with no such element renders as a single family-agnostic line in mwan3_rules, so traffic of both families reaches its chain; the IPv4 pass therefore builds both arms, ip saddr lookups and updates against the mwan3_sticky_v4_* sets and ip6 saddr against the mwan3_sticky_v6_* sets. proto icmp is the exception that renders a line on both passes, because ICMP and ICMPv6 are distinct L4 protocols; the IPv6 pass then points its own line at the chain the IPv4 pass has already built rather than appending to it.

The layout is load-bearing: every lookup precedes the single fall-through to the policy, and the update rules follow it. A pinned source must OR its member mark in before the fall-through can hand the packet to the policy. A lookup placed after the fall-through would let the policy re-mark the flow first and the late lookup would then OR the old pin into the new mark, producing a value that matches no interface.

Flow for a Sticky Rule

Packet arrives at mwan3_rules chain
Matches the user rule → jump mwan3_rule_https
Returning source: saddr is found in one of the per-member sets → jump mwan3_or_meta_<mark> ORs only the mwan3 bits into meta mark (non-destructive). The policy jump guard is false (mark already set), so policy is skipped. The matching update rule refreshes the timeout.
New source: no set match, mark stays 0. Falls through to the policy chain which assigns a mark. The matching update rule then adds saddr to the corresponding member's set with the configured timeout.
After timeout seconds of inactivity the set entry expires and the source is re-evaluated at next connection.

Set Lifecycle

A sticky entry normally persists until its per-entry timeout, but two failover events also flush sticky sets directly, so a pinned source can be re-evaluated before its timeout expires. When a member fails (a soft disconnected failure with the link still up) or goes down (ifdown), that member's own sticky sets are flushed by mwan3_delete_iface_map_entries in the same nft transaction that drops it from the policy, so a source pinned to the failed member re-picks the survivor on its next new connection rather than staying nailed to a member that is gone. When a higher-priority member recovers, mwan3_flush_worse_metric_sticky flushes the per-rule sticky sets of the lower-metric same-family members of each policy the recovered member participates in, scoped to the rules that use that policy (the snap-back), so new flows return to the recovered member; an equal-ranked load-balancing policy clears nothing, because no member outranks another.

In every case only the sticky sets are touched, and only new flows re-pick as a result. Established connections are not moved, because they are pinned by their saved conntrack mark rather than by the sticky sets (see Sections 10.3 and 10.4).

10. Service Lifecycle and Conntrack Management

10.1 Start

/etc/init.d/mwan3 start
  +--> nft -f /lib/mwan3/mwan3-skeleton.nft  (abort if fails)
  +--> mwan3_init()                    compute masks
  +--> mwan3_ensure_nft_framework()    recreate 6 internal sets, ensure chains
  +--> mwan3_render_config_ipsets()    create user-defined sets from config ipset
  +--> mwan3_write_dnsmasq_fragments() write nftset fragments; restart if changed
  +--> start_tracker per interface     launch mwan3track
  +--> mwan3_update_iface_to_table()   build iface->table mapping
  +--> mwan3_set_dynamic/connected/custom sets
  +--> mwan3_set_general_rules()       ip rule add blackhole/unreachable
  +--> mwan3_set_member_backstops()    per-WAN unreachable backstop ip rules
  +--> mwan3_ifup per interface        trigger hotplug (creates iface chains + ip rules)
  +--> wait for hotplug completion
  +--> mwan3_set_general_nft()         populate hook chain rules
  +--> mwan3_set_policies_nft()        create policy chains
  +--> mwan3_set_user_rules()          populate user rules chain
  +--> mwan3_flush_stale_conntrack()   flush zero-mark conntrack entries
  +--> mwan3_dnsmasq_hup()             SIGHUP dnsmasq to populate nftset domain sets
  +--> [if flow_offloading=1]          flush conntrack (force flow re-establishment)
  +--> start mwan3rtmon (ipv4 + ipv6)  route monitor daemons

10.2 Reload

/etc/init.d/mwan3 reload
  +--> [table inet mwan3 absent]       stop + start, then return
                                       (the skeleton load that creates the
                                       table is skipped when every interface
                                       is disabled)
  +--> mwan3_nft_reload_start()        open batch, flush/delete dynamic state
  +--> mwan3_ensure_nft_framework()    recreate 6 internal sets
  +--> mwan3_render_config_ipsets()    add user-defined sets (skips delete inside batch)
  +--> mwan3_cleanup_orphaned_ipsets() delete user sets not in current config
  +--> mwan3_set_dynamic/connected/custom sets
  +--> config_foreach mwan3_rebuild_iface_nft  (checks ubus for up status)
  +--> mwan3_set_general_nft()
  +--> mwan3_set_policies_nft()
  +--> mwan3_set_user_rules()
  +--> mwan3_nft_reload_commit()       commit all as single atomic kernel transaction
  +--> mwan3_update_iface_to_table()   (outside nft)
  +--> mwan3_set_general_rules()       (outside nft)
  +--> mwan3_set_member_backstops()    (outside nft)
  +--> mwan3_write_dnsmasq_fragments() restart dnsmasq only if fragment content changed
  +--> [if MWAN3_NEED_DNSMASQ_HUP=1]   mwan3_dnsmasq_hup() repopulates
                                       domain sets lost on a flag change
  +--> [tracker count check]           if mismatch: stop + start (procd cannot add/remove
                                       service instances in reload_service)

User-defined sets (config ipset) and sticky sets (mwan3_sticky_*) survive the reload intact - they are never in the preamble delete path.

10.3 Interface Up and Tracker Recovery (hotplug)

netifd signals ifup for $INTERFACE
  +--> 25-mwan3 hotplug script
       +--> mwan3_update_peer_track_ip()   write gateway IP (track_gateway)
       +--> mwan3_create_iface_nft()       create/flush chain, add rules
       +--> mwan3_create_iface_rules()     ip rule add (iif, fwmark, source)
       +--> mwan3_create_iface_route()     copy routes to per-iface table
       +--> mwan3_set_iface_hotplug_state "online/offline"
       +--> [if not init startup:]
       |    +--> mwan3_set_general_rules()          ensure ip rules exist
       |    +--> [if online:]
       |         +--> [nft batch:] mwan3_set_policies_nft()
       |         +--> mwan3_flush_worse_metric_sticky()
       |         +--> mwan3_flush_unreplied_conntrack()
       +--> procd_send_signal track_$INTERFACE USR2

connected fires for $INTERFACE (link-layer negotiation, or tracker recovery)
  +--> 25-mwan3 hotplug script
       +--> mwan3_set_iface_hotplug_state "online"
       +--> mwan3_create_iface_nft()       flush chain, update rules
       +--> mwan3_create_iface_rules()     recreate ip rules (idempotent)
       +--> mwan3_create_iface_route()     recreate routes (idempotent)
       +--> [nft batch:] mwan3_set_policies_nft()
       +--> mwan3_flush_worse_metric_sticky()
       +--> mwan3_flush_unreplied_conntrack()

The connected action fires on two distinct events: link-layer negotiation completion (PPPoE authentication, for example) after ifup routing state is already in place, and tracker recovery, when mwan3track judges a previously-failed interface to be working again without a physical reconnect. On a tracker recovery the interface's routing rules and route may have been removed by a prior ifdown; the connected arm recreates them unconditionally so the interface is fully routable before the policy re-selects it. On an ordinary link-negotiation event the routing is already in place and the recreate calls are harmless no-ops. The snap-back (mwan3_flush_worse_metric_sticky) returns new flows from lower-ranked same-family backups to the recovered interface; established flows on those backups are left to complete undisturbed.

Automatic Gateway Tracking (`track_gateway`)

When option track_gateway '1' is set on a config interface section, mwan3 automatically discovers the point-to-point peer/gateway IP and adds it to the tracking list at runtime. This is useful for PPPoE and other point-to-point links where the next-hop gateway IP changes on each connection and is not known in advance. Without this option, users must manually configure static track_ip addresses (typically public DNS servers) that may not test the actual link peer.

mwan3_update_peer_track_ip() queries ifstatus for the interface's ptpaddress field (the point-to-point peer IP). If found, the gateway IP is written to $MWAN3TRACK_STATUS_DIR/<iface>/GATEWAY. mwan3track reads this file and prepends the gateway IP to the front of the probe list, ensuring it is always probed first on every round regardless of the reliability threshold; static track_ip entries follow after the gateway. On interface bounce, the hotplug ifup action calls mwan3_update_peer_track_ip() again, overwriting the state file with the new peer IP.

The gateway IP is stored as ephemeral state rather than committed to UCI, preventing stale IP accumulation across reboots or gateway changes. An interface definition may specify only track_gateway and omit static tracking IPs entirely. The option is silently ignored if no next-hop peer address is found (e.g., on Ethernet WAN interfaces).

Warning

IPv4 only in practice. For IPv6 point-to-point links, the peer is typically a link-local address (e.g., fe80::1). Pinging link-local addresses requires interface scope specification (ping6 fe80::1%pppoe-wan), which mwan3track's probe mechanism does not handle. The option will be silently ignored if no peer address is found.

UCI Configuration:

config interface 'wan'
    option enabled '1'
    option track_gateway '1'
    # track_ip entries are optional when track_gateway is used
    # list track_ip '8.8.8.8'

The corresponding LuCI control is the "Track gateway" checkbox in the Interface tab (see Section 15.1.2).

10.4 Interface Down and Soft Failure (hotplug)

netifd signals ifdown for $INTERFACE
  +--> 25-mwan3 hotplug script
       +--> mwan3_set_iface_hotplug_state "offline"
       +--> [nft batch:] mwan3_set_policies_nft()
       |                 mwan3_delete_iface_map_entries()
       +--> mwan3_delete_iface_rules()         ip rule del
       +--> mwan3_delete_iface_route()         ip route flush table
       +--> mwan3_delete_iface_nft()           remove chain + jump rule
       +--> procd_send_signal track_$INTERFACE USR1
       +--> mwan3_flush_conntrack()            flush this iface's ct entries

disconnected fires for $INTERFACE (tracker-detected soft failure, link up)
  +--> 25-mwan3 hotplug script
       +--> mwan3_set_iface_hotplug_state "offline"
       +--> [nft batch:] mwan3_set_policies_nft()
       |                 mwan3_delete_iface_map_entries()
       +--> mwan3_flush_conntrack()            flush this iface's ct entries

On ifdown, the policy is rebuilt before the interface's routing rules are removed, ensuring the policy drops the interface from the choosable set before its routes are gone; a packet is never tagged for an interface whose routing has already been torn down. On disconnected, the interface remains physically up so routing is left intact; only the policy and stickiness records are updated. In both cases the policy rebuild and sticky flush commit as one nft transaction, in that fixed order: the rebuild runs first so the newly built policy cannot re-seed the sticky records that the flush is about to clear. The per-interface conntrack flush follows outside the transaction as a separate operation on kernel state that nftables does not own, and immediately moves established flows off the failed interface rather than leaving them to hang until TCP retransmit timeouts expire.

10.5 Stop

/etc/init.d/mwan3 stop
  +--> service_running || exit 0
  +--> mwan3_interface_shutdown per interface   trigger ifdown hotplug
  +--> flush ip routing tables (1..MWAN3_INTERFACE_MAX)
  +--> delete ip rules in configured ranges
  +--> $NFT delete table inet mwan3   (complete removal)
  +--> rm -rf status dirs

Result: table inet mwan3 no longer exists. Clean slate.

The single table delete is the whole nftables teardown: one kernel transaction drops every chain, set and map mwan3 created. mwan3 previously dismantled those objects one at a time, flushing all its chains, deleting the dynamic ones, flushing its sets and deleting its sticky maps before removing what remained. That sequence dates from mwan3 operating inside table inet fw4, a table it did not own and could not drop; with its own table it is redundant work, and removing it is what makes stop fast.

10.6 Conntrack Management

mwan3 uses the Linux connection tracking table (conntrack) to persist routing decisions across the packets of a single connection. On the first packet of a new flow, mwan3 stamps the packet with an interface fwmark and saves that mark into the conntrack entry's ct mark field. All subsequent packets in the same connection restore the ct mark back to the packet mark at the start of the prerouting/output chain, bypassing policy re-evaluation entirely. This means that routing policy changes do not take effect for established connections until their conntrack entries are removed.

What mwan3 does not do is often as important as what it does:

Stop performs no conntrack flush. All conntrack entries survive mwan3 stop intact. The marks are inert while the service is down since the nft table no longer exists to restore them. On restart with an unchanged config, interface IDs are assigned in the same order, so the saved marks remain valid and established connections survive the stop+start cycle without disruption. Flushing on stop would destroy that benefit for no practical gain.
Reload also performs no conntrack flush. Existing connections keep their saved ct marks and continue routing via whatever interface they were assigned at classification time. New connections are classified immediately under the new rules. This is intentional: a conntrack flush on every UCI commit would disrupt all active connections. The tradeoff is that existing flows are not rerouted when policy rules change. If that behaviour is needed, a stop followed by start (which does flush; see below) is required.

Conntrack actions mwan3 does take:

1. Start - zero-mark flush (mwan3_flush_stale_conntrack)

At the end of start_service, after all nft rules are in place, mwan3 runs:

mwan3ct flush --mark 0x0/$MMX_MASK

This removes every conntrack entry whose mwan3 mark bits are all zero. The target is entries created during the mwan3 startup window: the skeleton loads the hook chains immediately, making them live, but those chains are empty until mwan3_set_general_nft populates them at the end of the start sequence. Connections established during that window pass through prerouting with no classification rules present and are recorded in conntrack with ct mark=0. WireGuard's PersistentKeepalive refreshes the entry every 25 seconds, so without this flush the entry can survive indefinitely, causing the traffic to miss the mwan3 mark restore and fall through to the main routing table.

Only zero-mark entries are removed. Correctly-marked active connections are untouched.

2. Start - flow offloading flush

If software flow offloading is enabled (uci get firewall.@defaults[0].flow_offloading is 1), mwan3 flushes the entire conntrack table immediately after the zero-mark flush:

echo f > /proc/net/nf_conntrack

When flow offloading is active, the kernel fast-paths established flows directly through the flowtable, bypassing nftables entirely. Any flow that was offloaded before mwan3 started will never see the mwan3 prerouting chain and will never get a mark. Flushing everything forces all connections to re-enter the nftables pipeline and be classified correctly.

This does not apply to hardware flow offloading, which uses different kernel mechanisms and is not affected by writing to /proc/net/nf_conntrack.

3. Interface down / soft failure - selective flush (mwan3_flush_conntrack)

After the policy is rebuilt for a failing interface, the hotplug script calls mwan3_flush_conntrack. This function has two independent paths that both run on ifdown and disconnected:

First, the UCI flush_conntrack path: if the per-interface flush_conntrack UCI option contains ifdown, mwan3 writes f to /proc/net/nf_conntrack, flushing the entire global conntrack table. This is the legacy behaviour, retained for users who need it. Because it disrupts all connections on all interfaces simultaneously, it is not set by default.

Second, the selective flush path, which always runs on both ifdown and disconnected regardless of the UCI option:

mwan3ct flush --mark <iface_mark>/$MMX_MASK

This removes only the conntrack entries that belong to the failing interface (those whose masked mark bits equal that interface's assigned fwmark). The effect is that TCP and UDP flows that were using the failed WAN immediately re-establish under the new failover policy rather than waiting out a TCP retransmit timeout (typically 15-30 seconds). Connections on healthy interfaces are completely undisturbed.

4. Interface up - UCI flush only

On ifup the mwan3_flush_conntrack function runs the same UCI flush_conntrack check: if the option contains ifup, the entire conntrack table is flushed. New connections will be classified correctly by the rebuilt chains; existing connections that survived will re-use their stored ct marks as before.

5. Interface online - unreplied flush (mwan3_flush_unreplied_conntrack)

When an interface transitions to online - either via the connected action (mwan3track declares the interface online) or via ifup when status=online (interface comes up with initial_state set to online) - mwan3 runs:

mwan3ct flush --mark-any $MMX_MASK --status 0/0x2

This removes all mwan3-marked conntrack entries that have never received a reply (IPS_SEEN_REPLY not set). These UNREPLIED entries can result from a race during startup or policy rebuild where a flow gets classified before the nft chains reflect the current state. The stale mark pins traffic to the wrong WAN; because no reply arrives the application keeps retrying, refreshing the conntrack timeout indefinitely. Flushing only UNREPLIED entries is non-disruptive to established connections. The next packet from the affected application creates a fresh conntrack entry that is classified under the current policy. This flush is not performed during init - the zero-mark flush in start_service handles the startup window.

Summary table

Event	Selective (per-iface) flush	Unreplied flush	Global flush	Zero-mark flush
`start`	-	-	if flow_offloading	yes (always)
`reload`	-	-	-	-
`ifdown`	yes (always)	-	if `flush_conntrack` includes `ifdown`	-
`disconnected`	yes (always)	-	if `flush_conntrack` includes `disconnected`	-
`ifup`	-	if `status=online`	if `flush_conntrack` includes `ifup`	-
`connected`	-	yes (always)	-	-
`stop`	-	-	-	-

Implications for users

Changing mwan3 policy (via reload) does not reroute existing TCP connections. If a load-balance rule changes from 50/50 to 80/20, existing flows continue on whatever interface they were originally assigned. This is usually the right behaviour - rerouting mid-session would break most protocols - but means the new distribution only takes effect as connections naturally expire and re-establish.

The selective ifdown flush is the most operationally important flush. Its effect depends on the protocol.

For UDP-based protocols that identify connections by something other than the IP 4-tuple - WireGuard being the primary example - the flush allows transparent failover. WireGuard authenticates peers by cryptographic identity rather than source address. When the conntrack entry is removed and the next keepalive is reclassified via the failover WAN, the packet arrives at the remote peer from a new source IP, the peer authenticates it by key, updates its roaming endpoint, and the tunnel continues without interruption.

For TCP connections - file downloads, SSH sessions, browser connections - the picture is different. TCP identifies connections by the 4-tuple including source IP. When packets exit via the failover WAN, masquerade applies the new WAN's IP, and the remote end has no connection matching that source. It sends RST and the connection terminates. The connection was already dead the moment the WAN failed; what the flush controls is how quickly the application finds out. Without the flush, packets are silently blackholed (the old interface's ip rule no longer exists) and the application hangs in retransmit limbo until the TCP timeout expires - potentially minutes. With the flush, the RST arrives immediately and the application can reconnect at once.

The zero-mark flush on start is defensive infrastructure against a corner case that most users never notice explicitly. Its absence manifests as a WireGuard tunnel that routes through the wrong interface after a reboot, or a DNAT connection that falls to the main routing table immediately after mwan3 restarts following an fw4 reload.

Custom behaviour via mwan3.user

The default conntrack management fits most deployments. Three cases where additional action on interface recovery may be useful or desirable for some are:

Load-balanced configurations: when a failed WAN recovers, the automatic ifdown flush has already cleared entries for the failed interface, but all surviving flows remain pinned to the WANs that carried the load during the outage. They will not redistribute until they expire naturally.
Failover with a preferred primary: when the primary WAN recovers, existing connections stay on the failover WAN until natural expiry rather than snapping back. New connections return to the recovered primary on their own, since the snap-back clears the worse-metric sticky pins so the next new flow re-picks the higher-priority WAN, and where stickiness is not in use new flows re-pick by policy regardless; only established connections linger, which is the case this bullet addresses. In cost-sensitive setups where the primary is cheaper or unmetered, lingering on the failover WAN has a direct cost.
Application of new policies / rules to all flows immediately: a config save will cause all new connections be be immediately routed according to the new policy / ruleset, but will leave existing connections intact rather than interrupting them. Users can choose to override that behaviour and make the new policy take effect immediately even for existing connections, but at the cost of interrupting connections such as SSH, file downloads, etc.

In both cases, flushing all mwan3-marked conntrack entries on ifup causes connections to be reclassified immediately and routed according to the current policy:

#!/bin/sh
[ "$ACTION" = "ifup" ] || exit 0

. /lib/functions.sh
. /lib/mwan3/common.sh
. /lib/mwan3/mwan3.sh
config_load mwan3
mwan3_init
mwan3_flush_marked_conntrack

Note that the flush is broader than strictly necessary, although it's still a lot less of a hammer than conntrack -F. Since the recovering interface has no conntrack entries (they were removed on ifdown), the flush affects connections on all other interfaces, including any that were healthy throughout the outage. There is no way to target only those connections that moved as a result of the failure. The trade-off is that every connection is briefly interrupted on any interface recovery event. For most TCP connections the interruption is imperceptible; long-lived sessions such as SSH or active file transfers will be reset.

For this reason, default behaviour is not to flush all marked conntrack entries but to allow this action to fall-back on a per-installation level to mwan3.user through use of the mwan3_flush_marked_conntrack supplied expressly for this purpose.

11. Atomic Non-destructive Reload

reload_service is a single atomic nft -f batch that rebuilds the entire ruleset while the old one is still serving traffic, committing in one kernel transaction with zero traffic disruption window.

MWAN3_BATCH_DEPTH Counter

MWAN3_BATCH_DEPTH is an integer counter (default 0) in common.sh. It enables nested batch accumulation:

mwan3_nft_batch_start: increments depth; truncates batch file only at depth 0->1.
mwan3_nft_batch_commit: decrements depth; commits to kernel only at depth 1->0.
mwan3_nft_exec: routes to mwan3_nft_push when depth > 0, accumulating all operations.

This means individual build functions (which each call mwan3_nft_batch_start / mwan3_nft_batch_commit internally) work correctly both standalone (immediate kernel commit) and when called from inside a reload batch (operations accumulate, committed as one transaction at the end).

Preamble (mwan3_nft_reload_start)

Opens the outermost batch (depth 0->1), then writes:

Flush all 8 skeleton chains (mwan3_prerouting, mwan3_output, mwan3_postrouting, mwan3_ifaces_in, mwan3_rules, mwan3_connected, mwan3_custom, mwan3_dynamic).
Two-pass flush then delete all dynamic chains (mwan3_iface_in_*, mwan3_policy_*, mwan3_rule_*, mwan3_or_meta_*, mwan3_or_ct_*). Two passes are required because mwan3_or_meta_* chains are referenced by mwan3_rule_* sticky chains; a single alphabetical pass would attempt to delete or_meta before flushing rule, producing "Device or resource busy".
Delete the 6 internal mwan3_* sets so mwan3_ensure_nft_framework recreates them with correct flags.

What is NOT in the preamble: user-defined sets (from config ipset), sticky sets (mwan3_sticky_*). These survive the reload intact, preserving dnsmasq-populated addresses and sticky routing state.

Batch Guards in Build Functions

Six locations in build functions query kernel state (chain/set existence) that are bypassed when inside a batch (MWAN3_BATCH_DEPTH > 0), because the kernel still shows the pre-preamble state until the batch commits:

mwan3_set_general_nft idempotency guard
mwan3_create_iface_nft SNAT loop / jump rule check
mwan3_create_policies_nft (unconditional add+flush inside batch)
mwan3_set_policies_nft orphan cleanup
mwan3_ensure_nft_framework delete set loop
_mwan3_render_one_ipset delete set (skipped inside batch when flags are unchanged, to preserve dnsmasq-populated elements; queued inside batch when flags differ)

Commit

mwan3_nft_reload_commit calls mwan3_nft_batch_commit. At depth 1->0 the entire accumulated batch is submitted to the kernel as a single transaction. If nft -f returns non-zero, the kernel rolls back completely and the old ruleset continues serving traffic. No partial states are possible.

12. User-defined nft Sets

config ipset sections in /etc/config/mwan3 create named nft sets in table inet mwan3. These sets can be referenced in rules via the ipset (destination) and ipset_src (source) UCI options.

UCI Configuration

config ipset 'youtube_ipv4'
    option name     'youtube_v4'
    option family   'ipv4'
    option enabled  '1'
    option maxelem  '0'          # 0 = unlimited (default)
    option timeout  '0'          # 0 = no timeout (default), seconds
    option counters '0'          # 0 = no per-element counters (default)
    list entry  '8.8.8.8'
    list entry  '203.0.113.0/24'
    list domain 'youtube.com'
    list domain 'googlevideo.com'
    option loadfile '/etc/mwan3/custom-ips.txt'

Options

Option	Values	Purpose
`name`	string	nft set name in `table inet mwan3`. Must be unique.
`family`	`ipv4` or `ipv6`	Address family. Determines set type (`ipv4_addr` or `ipv6_addr`).
`enabled`	`0`/`1`	Skip this set if `0`.
`maxelem`	integer	Maximum elements. `0` = unlimited (default).
`timeout`	integer	Per-element timeout in seconds. `0` = no timeout.
`counters`	`0`/`1`	Enable per-element packet/byte counter tracking.
`list entry`	CIDR or address	Inline address/CIDR added at startup.
`list domain`	domain name	Causes mwan3 to write a dnsmasq `nftset=` confdir fragment for this domain. dnsmasq populates the set via DNS resolution.
`loadfile`	path	File containing addresses/CIDRs, one per line. Loaded at startup.

counter Syntax

counter is a standalone nft set statement, NOT a flag keyword. It appears after the type/flags declarations on its own line:

set youtube_v4 {
    type ipv4_addr
    flags interval
    auto-merge
    counter
}

Placing counter inside the flags list causes an nft parse error. The auto-merge line appears here only because this set has no timeout; a set with a timeout omits it (see auto-merge and timeouts below).

auto-merge and timeouts

User-defined sets are declared with flags interval, and auto-merge is appended only when no timeout is configured. auto-merge coalesces any run of contiguous addresses into a single merged range, a useful memory optimization for static CIDR sets, but the merged range carries only one timeout for the whole run. On a timeout-bearing set populated host by host by an external resolver, for example a list domain set filled by dnsmasq nftset= (or smartdns), that coalescing discards the per-element TTLs: under the incremental insertion a resolver uses, the merged range keeps the timeout of whichever element most recently extended it, so a short-lived member extending a run of long-lived ones evicts the whole range early, and a long-lived member extending a run keeps a short-lived member alive well past its TTL. For resolver-populated sets the per-element TTL is the entire point, so auto-merge is dropped whenever a timeout is set. The interval flag is retained either way, so timeout sets can still hold CIDR entries; only the coalescing is dropped.

dnsmasq Integration

For each list domain entry, mwan3_write_dnsmasq_fragments writes a dnsmasq confdir fragment:

nftset=/youtube.com,googlevideo.com/4#inet#mwan3#youtube_v4

The fragment restarts dnsmasq only when its content has changed (comparing against the previously written fragment). On the next DNS query for the domain, dnsmasq adds the resolved address to the nft set automatically. A mwan3_dnsmasq_hup call from start_service forces initial resolution. reload_service also calls mwan3_dnsmasq_hup when MWAN3_NEED_DNSMASQ_HUP is set -- see Flag Changes below.

Flag Changes

nft add set is idempotent and does NOT update flags (type, counter, timeout, size) on an existing set. When a set's flags are changed via LuCI or UCI command and the config is applied, _mwan3_ipset_needs_delete detects the mismatch by querying the live kernel set and comparing against the desired spec. If any flag differs, the set is queued for deletion inside the reload batch before being recreated with the new flags. The reload batch preamble has already flushed mwan3_rules (removing all references to user sets) before the delete fires at commit time, so the delete is safe.

If the deleted set had list domain entries, its dnsmasq-populated elements are lost when it is recreated empty. _mwan3_render_one_ipset sets MWAN3_NEED_DNSMASQ_HUP=1 in this case. After mwan3_write_dnsmasq_fragments completes in reload_service, a mwan3_dnsmasq_hup call is issued to repopulate the set via DNS re-resolution.

Migration from fw4 ipsets

mwan3-migrate-ipset-v4.sh (a one-shot script run from postinst and immediately deleted) copies config ipset sections from /etc/config/firewall to /etc/config/mwan3 if any of the sets in /etc/config/firewall are referenced in an mwan3 user rule. The match UCI option required by fw4 is silently ignored by mwan3 (mwan3 does not call config_get match).

Orphan Cleanup

mwan3_cleanup_orphaned_ipsets is called in reload_service after mwan3_render_config_ipsets. It queries nft list table inet mwan3 for user-defined sets (those not prefixed mwan3_), compares against currently configured set names, and emits delete set for any orphans. This prevents stale sets accumulating when a set is removed from UCI config via LuCI.

Rule Integration (`ipset` and `ipset_src`)

The ipset UCI option on a rule section matches the destination address against a named nft set. The ipset_src option provides the complementary source address match. Both can be set on the same rule and are ANDed together.

UCI Example

config rule 'corp_to_wan2'
    option ipset_src corp_clients
    option ipset     blocked_dests
    option use_policy wan2_policy

This generates:

ip saddr @corp_clients ip daddr @blocked_dests meta mark & 0x3f00 == 0 jump mwan3_policy_wan2_policy

If the named set does not yet exist in table inet mwan3 at rule-install time, mwan3_set_user_nft_rule() pre-creates it with the appropriate type to prevent the nft batch from failing atomically. A present ipset_src is treated as an implicit family qualifier by the meta nfproto guard condition in mwan3_set_user_nft_rule(), consistent with src_ip, dest_ip, and ipset.

13. Unchanged Files

File	Reason
`etc/hotplug.d/iface/26-mwan3-user`	Just calls `/etc/mwan3.user`, no firewall code
`etc/mwan3.user`	User script template, no firewall code
`etc/uci-defaults/mwan3-migrate-flush_conntrack`	UCI migration, no firewall code

14. Diagnostic Commands

# Full mwan3 table dump
nft list table inet mwan3

# List specific chain
nft list chain inet mwan3 mwan3_prerouting
nft list chain inet mwan3 mwan3_output
nft list chain inet mwan3 mwan3_postrouting
nft list chain inet mwan3 mwan3_ifaces_in
nft list chain inet mwan3 mwan3_policy_balanced

# Check internal set contents
nft list set inet mwan3 mwan3_connected_v4
nft list set inet mwan3 mwan3_connected_v6
nft list set inet mwan3 mwan3_custom_v4
nft list set inet mwan3 mwan3_dynamic_v4

# Check user-defined sets (config ipset)
nft list set inet mwan3 youtube_v4
nft list table inet mwan3 | grep '^\tset '   # all sets in mwan3 table only

# Check sticky set entries (per-member sets, id suffix = interface id)
nft list set inet mwan3 mwan3_sticky_v4_https_1
nft list set inet mwan3 mwan3_sticky_v6_https_1

# List all mwan3 chains (names only)
# Note: nft list chains only accepts an optional family, not a table name
nft list chains inet | grep mwan3_

# List OR-immediate setter chains (vmap dispatch)
nft list chains inet | grep mwan3_or_

# Verify marks are being set (add temporary counter)
nft add rule inet mwan3 mwan3_prerouting meta mark and 0x3f00 != 0 counter

# Check connmarks
conntrack -L -o mark

# Check ip rules
ip rule list | grep -E '^[1-3][0-9]{3}:'

# Check per-interface routing table (table N for interface N)
ip route list table 1

# JSON output (for scripting/debugging)
nft -j list set inet mwan3 mwan3_connected_v4
nft -j list chain inet mwan3 mwan3_policy_balanced

# Check mwan3 status
mwan3 status
mwan3 internal

# RPC query
ubus call mwan3 status '{"section":"interfaces"}'
ubus call mwan3 status '{"section":"connected"}'
ubus call mwan3 status '{"section":"policies"}'

# Query user-defined set info and members
ubus call mwan3 nftset_info '{}'
ubus call mwan3 nftset_elements '{"set":"youtube_v4","max":200}'

15. luci-app-mwan3

luci-app-mwan3 is the LuCI web interface for mwan3, installed as a separate package from the core mwan3 daemon. It is organised into two menu groups: Network > MultiWAN Manager for configuration and Status > MultiWAN Manager for runtime monitoring and diagnostics.

15.1 Network App

The Network app provides tabs for configuring every aspect of mwan3: global parameters, WAN interfaces, policy members, policies, traffic rules, nftables IP sets, a traffic simulator, a static configuration analyser, and user notification scripts.

15.1.1 Globals

The Globals tab (globals.js) edits the config globals UCI section.

Mark mask (mmx_mask): The hexadecimal bitmask used for policy mark operations. Determines the number of bits available for interface encoding, which sets MWAN3_INTERFACE_MAX (the maximum number of simultaneously active interfaces). A live computed display shows the derived interface limit as the value is changed. Default 0x3F00 (6 bits, 60 interfaces).

Rule base priorities: Three numeric fields set iif_rule_base, fwmark_rule_base, and unreachable_rule_base. Live cross-field validation enforces both ordering constraints (see Section 5.5) on blur; an error is shown if either constraint would be violated. When any rule base priority changes, saving and applying the form triggers mwan3 restart to rebuild all ip rules at the new priorities.

Additional routing tables (rt_table_lookup): A dynamic list of extra routing table IDs whose routes are merged into each per-interface routing table alongside routes from the main table.

Bypass networks (bypass_network): A dynamic list of CIDRs added to the connected-network bypass sets (mwan3_connected_v4/mwan3_connected_v6), causing traffic to those networks to skip mwan3 policy routing.

Logging: A checkbox to enable mwan3 logging and a numeric loglevel field (0-7 following syslog severity levels).

Verbose logging (verbose_logging): A checkbox that enables debug-level log output from mwan3 daemons. When enabled, the LOG() function emits debug messages to syslog from mwan3.sh, common.sh, and mwan3track. Takes effect on HUP-triggered config reloads without requiring a full restart.

15.1.2 Interface

The Interface tab (interface.js) presents a GridSection where each row represents a config interface section in the mwan3 UCI config. Clicking a row opens the per-interface modal editor.

Tracking configuration: The tracking method field supports ping (always available), httping, nping, and arping. The httping, nping, and arping options are only shown if the corresponding binary is detected on the router via fs.stat. The track_ip field accepts a dynamic list of addresses or hostnames to probe. track_method, reliability, count, size, max_ttl, timeout, interval, failure_interval, recovery_interval, keep_failure_interval, down, and up thresholds are all configurable.

Quality tracking: The check_quality checkbox enables latency and loss threshold monitoring. When enabled, four additional fields appear: failure_latency, failure_loss, recovery_latency, and recovery_loss.

Track gateway (track_gateway): A checkbox shown only for IPv4 interfaces. When enabled, mwan3 automatically uses the interface's network gateway as the tracking target (see Section 10.3).

IPv6 SNAT (snat6): A text field shown only for IPv6 interfaces. Accepts empty/0 (disabled), 1 (SNAT to the interface's primary global address), or a literal IPv6 address for fixed-source pinning (see Section 4).

Conntrack flushing (flush_conntrack): Controls whether the global conntrack table is flushed on interface state changes. Per-interface conntrack entries are flushed automatically on ifdown regardless of this setting.

Validation: Interface section names must be 15 characters or fewer and unique across all section types in the mwan3 config. The interface's current network metric is shown as a read-only value derived from the netifd network configuration.

15.1.3 Member

The Member tab (member.js) presents a GridSection for config member sections. Each member has three fields: interface (a dropdown of currently configured mwan3 interfaces), metric (1-256), and weight (1-1000). Members are the atomic routing units referenced by policies.

An informational banner at the top of the section directs users to the Policy Builder (see Section 15.1.4) for normal member management. The Member tab is preserved for direct inspection and manual adjustment of metric and weight values.

15.1.4 Policy

The Policy tab (policy.js) presents a GridSection where each row represents a config policy section. The grid shows two read-only derived columns - "IPv4 Priority order" and "IPv6 Priority order" - alongside an editable last_resort column. The derived columns render the policy behaviour in symbolic form: a single interface name for single-interface policies; [iface N%, iface N%, ...] for load-balanced tiers; [primary] --> [failover] --> ... for failover chains.

Policy Builder modal: Clicking "Add..." or the edit button on an existing policy opens the policy builder. The builder presents the policy as an ordered list of tiers. Tier 1 is labelled "Primary"; subsequent tiers are labelled "Failover (N)". Each tier contains one or more interface entries, each with a family selector (IPv4/IPv6), an interface selector filtered to interfaces of the chosen family, and a percentage share field.

Shares are per-family within each tier: all IPv4 entries in a tier must sum to 100%, and similarly for IPv6. When an entry is added to a family group its share is computed by redistributing 100% equally across all entries in that family group (first entry absorbs any rounding remainder). Changing an entry's family redistributes both the old and new family groups. A single-entry family group has its share field locked at 100 and shown greyed out.

Member management: On save the builder computes GCD-reduced integer weights from the percentage inputs, then for each (interface, metric, weight) tuple finds an existing member section with exactly those values and reuses it (regardless of name), or creates a new section named <iface>_m<metric>_w<weight>. After saving, any member sections that were in the old use_member list but are no longer referenced by any policy become orphans. A "Delete unused member definitions" checkbox in the page header (persisted in localStorage) controls automatic versus prompted orphan cleanup.

Validation: Real-time validation reports missing or invalid policy name, name already in use, unselected interface on any entry, and family share totals that do not equal 100%. The Save button is disabled while any error is present.

15.1.5 Rule

The Rule tab (rule.js) presents a GridSection where each row represents a config rule section. Rules are evaluated in UCI declaration order; order matters.

Fields: family (ipv4/ipv6/any), proto (all/tcp/udp/icmp/icmpv6/...), src_ip, src_port, dest_ip, dest_port, fwmark, sticky, timeout, ipset_src (source NFT set), ipset (destination NFT set), logging, use_policy, and enabled.

NFT set dropdowns: The ipset_src and ipset fields are populated from the mwan3.nftset_info rpcd method. Disabled sets are excluded from the dropdowns. A destination set must match the rule's selected address family; combining a set with a same-dimension IP field (ipset with dest_ip, or ipset_src with src_ip) is flagged as an error since both constrain the same traffic dimension. A cross-set family mismatch between ipset_src and ipset on the same rule is also flagged.

Fwmark field: The fwmark field renders as a combined value/mask input. On read, the stored fwmark and fwmask UCI options are combined; on write they are split back. Validation checks hex format and flags any overlap between the user mask and MMX_MASK.

Column merging: The grid merges source port into the Source column and destination port into the Destination column. The merged cell renders as address:port or set_name:port when both are set, address or set_name alone when only an address or set is present, *:port when only a port is set, and - when neither is set. src_port and dest_port are modalonly, remaining fully editable in the modal.

Enable column: An inline Enable checkbox in the grid uses o.editable = true for direct toggle without opening the modal. The checkbox is rendered non-interactive when the rule is disabled and references at least one currently disabled set, preventing re-enable into a broken state.

Address family validation: src_ip and dest_ip accept a single address or CIDR, or a comma-separated list of addresses and CIDRs. Each element in a comma-separated list is individually validated for format and family consistency. A list containing a mix of IPv4 and IPv6 elements is rejected. Family consistency between the rule and any referenced NFT set is checked on save.

15.1.6 IP Sets

The IP Sets configuration tab (ipset.js) presents a GridSection for config ipset sections.

Field	UCI option	Description
Name	`name`	Identifier referenced in rule `ipset`/`ipset_src` fields. Must start with a letter or `_`; may contain letters, digits, `_`, `.`, `-`. Names beginning with `mwan3_` are reserved.
Family	`family`	`ipv4` or `ipv6`. Sets the nftables element type (`ipv4_addr` or `ipv6_addr`).
IPs / Networks	`entry`	Static addresses or CIDR subnets. Validated against the selected family on blur.
Domains	`domain`	Domain names resolved by dnsmasq and added to the set at runtime. mwan3 generates corresponding `nftset=` fragments in the dnsmasq config.
Include File	`loadfile`	Path to a file of addresses or CIDRs (one per line, `#` comments ignored) loaded at service start.
Max Entries	`maxelem`	Maximum element count. Empty means no limit.
Timeout	`timeout`	Per-element lifetime in seconds. `0` means entries do not expire.
Counters	`counters`	Enables per-element packet and byte counters in the nftables set.
Enabled	`enabled`	Whether the set is created and populated at service start.

Deletion of a set is blocked in the UI if any rule references the set name in its ipset or ipset_src field; the user is shown which rules reference the set. The enabled checkbox is rendered non-interactive when the set is referenced by at least one enabled rule, preventing the set from being disabled while a rule depends on it.

The entry and loadfile fields define the static content restored by the Reload button on the IP Sets status tab. The domain fields define the domains queried by the Resolve button; the Resolve button is only shown for sets that have at least one domain entry.

15.1.7 Simulator

The Traffic Path Simulator (simulator.js) accepts source IP or hostname, destination IP or hostname, fwmark (hex), protocol, source port, destination port, and address family, then reports which mwan3 rule would match and which policy would handle that traffic.

Hostname resolution: If a hostname is entered in either IP field, the simulator calls mwan3.resolve_host before running the simulation. This invokes busybox nslookup against 127.0.0.1, which has the side effect of populating any dnsmasq nftset entries configured for that domain. Resolved addresses are shown inline below the field. If resolution fails or returns no addresses for the selected family the simulation is aborted with an error message. When family is "IPv4 and IPv6", IPv4 addresses are preferred. Strings that are not valid IP addresses and do not resemble a valid hostname are rejected before simulation. IP address inputs are validated before the simulation runs; malformed addresses are rejected.

Port fields: Source and destination port fields are shown only when the protocol is set to TCP or UDP. All fields except protocol are optional; a blank field means "any" and rules with a constraint on that field will not match. The fwmark field accepts a hex value and is matched against rules that have an fwmark/fwmask constraint; a blank fwmark is treated as 0x0 (unmarked packet).

Simulation logic: On each simulate press, UCI configuration, live policy state, and the connected-network sets (mwan3_connected_v4/mwan3_connected_v6) are reloaded fresh. The simulator first checks whether the destination falls in a directly-connected network and, if so, reports that mwan3 rules are bypassed. Otherwise it evaluates each enabled UCI rule in declaration order, fetching live nftset membership via mwan3.nftset_members for any sets referenced by those rules. Elements in counter-enabled sets are correctly unwrapped from the {"elem":{"val":"...","counter":{...}}} JSON structure used by nft. The first matching rule is the result; any additional rules that also match are listed as shadowed rules.

15.1.8 Configuration

The Configuration tab (configuration.js) performs a static analysis of the mwan3 UCI configuration with no live system state consulted. It loads the current UCI config, builds lookup tables for interfaces, members, policies, and rules, then reports issues at two severity levels: error (a broken reference or empty policy that will cause traffic to be silently misrouted or blackholed) and warning (an orphaned object that is defined but has no effect).

Checks performed:

Member references an undefined interface
Member is defined but not used by any policy
Policy has no members
Policy references an undefined member
Policy has multiple members that all reference the same interface (no real redundancy; failover will not occur if that interface goes down)
Policy is defined but not used by any rule
Rule references an undefined policy
Rule is shadowed by an earlier rule (the earlier rule matches a superset of the later rule's traffic, so the later rule is unreachable)
Interface is defined but not referenced by any member

Rule shadowing check: ruleAContainsB() determines whether rule A (earlier) is a superset of rule B (later). The check is deliberately conservative: it only flags clear containment. CIDR containment is computed precisely for both IPv4 (32-bit unsigned arithmetic) and IPv6 (BigInt). Port specs use conservative matching: a rule with no port restriction contains any other; two rules with non-empty port specs are only flagged as contained when the specs are identical strings. When both rules reference an NFT set, the set names are compared: if both rules reference the same nftset the shadowing check proceeds normally; if the set names differ the check is skipped conservatively since set membership cannot be evaluated statically. If only one rule uses a set the check is also skipped. Address family is respected: an IPv4-only rule does not shadow an IPv6-only rule.

15.1.9 Notify

The Notify tab (notify.js) provides a raw textarea editor for /etc/mwan3.user. This file is sourced by the mwan3 hotplug handler at the end of interface up/down events, allowing site-specific shell commands to run in response to WAN state changes. The tab reads the file via fs.read on load and writes it back via fs.write on save; no UCI is involved.

The menu node admin/network/mwan3/notify declares an ACL dependency on luci-app-mwan3-notify, so the tab is absent entirely for a session that does not hold that group. Where the group is held read-only, notify.js sets isReadonlyView from L.hasViewPermission(), which suppresses handleSave and disables the textarea, leaving the hook visible but not editable.

15.2 Status App

The Status app provides live monitoring and diagnostic tools. All tabs that show runtime data poll automatically via the LuCI poll mechanism.

15.2.1 Overview

The Overview tab (overview.js) polls mwan3.status and displays a three-section summary of all mwan3 interface states, active policies, and configured rules.

Interfaces section: A CSS grid (repeat(auto-fill, 13em)) of per-interface status cards. Each card has a coloured border (green/red/orange/grey by status) and shows the interface name, status label, and uptime or online/offline duration. Duration is rendered without seconds by formatDuration() (days, hours, minutes) to keep the display stable under polling.

Policies section: A CSS grid of per-policy cards in the same bordered card style. Each card lists its member interfaces with live traffic share percentages. Member lines use white-space: nowrap. A member's line is coloured green when its share is nonzero, yellow when it is online but carrying zero percent (standby or failover member not currently active), and grey when offline.

Rules section: A table of enabled rules. The Match column summarises the rule in compact addr:port format by fmtAddr(): address:port when both are set, address alone, *:port for port-only, or (all traffic) for unconstrained rules. Both source (ipset_src) and destination (ipset) NFT sets are shown where configured.

15.2.2 Interface Status

The Interface Status tab (detail.js) provides per-interface tracking detail, live-polled from mwan3.status. An interface selector at the top of the page drives the display.

For the selected interface, the tab shows a header card with status, tracking method, and composite reliability score, followed by a table of individual tracking targets. Each row shows the target address, probe status (up/down/skipped), current latency, and current loss percentage. When check_quality is disabled, the latency and loss columns show "Not enabled". Down targets show an infinity symbol for latency. Rows are sorted: up targets first, then down, then skipped.

15.2.3 Routing Health

The Routing Health tab (routing.js) polls mwan3.routing_health and performs a live sanity check of the ip rules and routing tables mwan3 maintains for each configured interface.

Per-interface cards: Each card shows the interface name, sequential index, and mwan3track-reported status, followed by four status rows, five on IPv4 interfaces:

IP rule (iif) at priority iif_base + index: expected present when the interface is online.
IP rule (fwmark) at priority fwmark_base + index: expected present when the interface is online.
IP rule (unreachable) at priority unreachable_base + index: expected present when the interface is online.
IP rule (source) at priority src_base + index: expected present when the interface is online. Rendered only for IPv4-family interfaces, the only ones for which routing_health reports a src_rule object. A device holding several global IPv4 addresses carries one rule per address at this priority.
Routing table (table ID = index): whether the per-interface routing table has a default route.

Each row renders a status badge: "Present" (green) or "Missing" (red) when the item is expected to be present (interface online); neutral grey "Present"/"Absent" for every other state, where presence is reported without judgement. The card border is green when all rows are in the expected state, yellow for partial state or for an interface reported offline or disconnecting, red when the interface is online and no rules are present, and grey for disabled or unknown interfaces.

Summary card: A single line at the top reports total healthy, degraded, and failing interface counts, plus stale rule count if any.

Stale rules: ip rules whose priority falls within the mwan3 priority ranges but do not correspond to any currently configured UCI interface are listed in a separate table. Stale rules indicate that mwan3 was not cleanly shut down after an interface was removed from the configuration.

Field guide: A reference panel at the bottom explains each field (Index, iif rule, fwmark rule, unreachable rule, source rule) in plain language, using the actual configured base priority values read from the routing_health response rather than hardcoded defaults. The source base is derived by the backend as unreachable_base + iface_max + 1 rather than read from UCI; the fallback in mwan3/constants.js is 3061, the value the default bases and the default 0x3F00 mask imply.

15.2.4 IP Sets (Status)

The IP Sets status tab (ipsets.js) displays runtime information about all user-defined sets currently present in table inet mwan3. At page load it calls mwan3.nftset_info to enumerate sets and their metadata, then renders each as a collapsible panel.

Panel header: Shows the set name, family, element count (with N+ suffix when the display is truncated), a Counters badge when per-element counters are enabled, and action buttons: Resolve (only for sets with at least one domain entry), Reload, Flush, and Expand/Collapse.

Panel body: Shows UCI metadata (static entry count, domain count, loadfile name, maxelem, timeout), the list of configured domain names, and a table of current set members. Members are loaded lazily on first expand via mwan3.nftset_elements with a default limit of 200 elements. "Load more (1000)" and "Load all (5000)" buttons are shown when the set has more elements than the current display. Elements in counter-enabled sets show packet count and byte count (formatted as B/KiB/MiB/GiB) alongside the address.

Actions:

Flush (mwan3.nftset_flush): empties the set.
Reload (mwan3.nftset_reload): flushes then re-adds static entries from UCI (entry values and loadfile contents).
Resolve (mwan3.nftset_resolve): HUPs dnsmasq and re-queries each configured domain, triggering dnsmasq's nftset population as a side effect.

After each action the header count and, if the panel is expanded, the member table refresh automatically.

15.2.5 Diagnostics

The Diagnostics tab (diagnostics.js) provides an interactive runner for per-interface diagnostic commands. An interface selector and a task selector drive the operation; a Run button executes the selected task and displays output in a <pre> element. All buttons are disabled during execution.

Task	Implementation	Description
Ping gateway	`luci-mwan3 diag gateway <iface>`	Pings the interface's current gateway 5 times via `mwan3 use`
Ping tracking IPs	`luci-mwan3 diag tracking <iface>`	Pings each configured track_ip 5 times via `mwan3 use`
Check IP rules	`luci-mwan3 diag rules <iface>`	Checks presence of all three ip rules for the interface
Check routing table	`luci-mwan3 diag routes <iface>`	Shows `ip route list table <id>` for the interface
Force interface up	`mwan3 ifup <iface>`	Triggers the ifup hotplug sequence
Force interface down	`mwan3 ifdown <iface>`	Triggers the ifdown hotplug sequence

15.2.6 Troubleshooting

The Troubleshooting tab (troubleshooting.js) presents the full output of mwan3 internal ipv4 and mwan3 internal ipv6 as a set of collapsible <details> panels, one panel per output section. The output is parsed by splitting on underline-delimited section headings (lines of = characters).

The nftables dump section is filtered to remove the mwan3_or_(meta|ct)_* vmap setter chains from the display, replacing them with a comment indicating how many chains were removed. These chains are an implementation detail of the connmark operation (see Section 2) and would otherwise dominate the output. The IPv6 internal output deduplicates sections that are identical to the IPv4 output (Software Version and nft tables), showing only the IPv6-specific sections. Each section's content is shown in a <pre> element with max-height: 250px and vertical scroll.

15.3 Backend: Helper Script and ACL

Helper Script (`luci-mwan3`)

The /usr/libexec/luci-mwan3 shell script provides backend commands invoked by the diagnostics tab.

diag subcommand: diag gateway <iface>, diag tracking <iface>, diag rules <iface>, and diag routes <iface>. The diag rules function reads iif_rule_base, fwmark_rule_base, and unreachable_rule_base from UCI config globals, computes the expected priority for each of the three ip rule tiers, queries ip rule, and reports found/missing counts. The diag routes function shows ip route list table <id> for the interface's sequential table ID.

nftset subcommand: nftset dump lists all user-defined set names in table inet mwan3 by running nft list table inet mwan3 and extracting set names via awk, then filtering out mwan3's own internal sets (prefixed mwan3_). This subcommand is no longer called by the rule editor (which uses the mwan3.nftset_info rpcd method directly) but is retained for command-line use.

Note

nft list sets inet mwan3 is not used because it lists sets from all inet tables, not just mwan3's table. nft list table inet mwan3 is the correct form.

ACL Permissions (`luci-app-mwan3.json`)

The rpcd ACL file grants the LuCI frontend permission to call rpcd methods and execute files. It defines three ACL groups:

luci-app-mwan3 (configuration access): Read permission for mwan3.status, mwan3.nftset_info, mwan3.nftset_members, mwan3.nftset_elements, mwan3.resolve_host, and the mwan3 and network UCI packages. Write permission for mwan3.nftset_flush, mwan3.nftset_reload, mwan3.nftset_resolve, and the mwan3 UCI package. Optional-binary detection needs two grants working together: the list permission on /usr/bin/httping, /usr/bin/nping and /usr/bin/arping, which is what rpcd's stat handler checks on the path, and the stat method on the ubus file object, which the HTTP ubus endpoint checks before dispatching the call. The read block also grants cgi-io exec and exec permission for luci-mwan3 nftset dump.

luci-app-mwan3-status (status and diagnostic access): Read permission for mwan3.status, mwan3.nftset_info, mwan3.nftset_members, mwan3.nftset_elements, mwan3.resolve_host, and mwan3.routing_health, together with cgi-io exec and exec permission for /usr/sbin/mwan3 status. Write permission for mwan3.nftset_flush, mwan3.nftset_reload, mwan3.nftset_resolve, the exec method on the ubus file object, and exec permission for the luci-mwan3 diag subcommands, mwan3 internal ipv4 and mwan3 internal ipv6, mwan3 ifup, and mwan3 ifdown.

luci-app-mwan3-notify (hotplug hook editing): Read permission for /etc/mwan3.user together with the read method on the ubus file object. Write permission for the same path together with that object's write and remove methods. The group is separate so that delegating mwan3 configuration does not imply the ability to edit the hook: /etc/mwan3.user is a shell script the hotplug handler runs as root on every interface event, so write access to it is equivalent to root. Before the split these grants sat in the configuration group. A session without the group does not see the Notify tab, read gives a read-only view, and write gives editing.

15.4 rpcd Methods

The following methods are implemented in usr/share/rpcd/ucode/mwan3 and declared in root/usr/share/rpcd/acl.d/luci-app-mwan3.json.

mwan3.status {}

Returns JSON data for interfaces, connected networks, and policies. Used by the Status tab.

Connected IPs: Parses nft -j list set inet mwan3 mwan3_connected_v4/v6.
Policies: Reads membership from UCI config and cross-references mwan3track STATUS files. Every member is always reported with traffic share percentage.
Interfaces: Reads status from /var/run/mwan3track/ files and queries procd/netifd via ubus. Tracking IPs discovered by globbing TRACK_* files. Per-IP latency and packetloss populated when check_quality=1.

mwan3.nftset_info {}

Returns the name, address-family type, counters flag, and runtime element count of all non-mwan3 sets in table inet mwan3. Used by the rule editor nftset dropdown and the IP Sets status tab.

mwan3.nftset_elements { set: "<name>", max: N }

Returns paginated elements of a named user-defined set with per-element packet/byte counters (when the set has counter enabled). The set name is validated against ^[a-zA-Z0-9_-]+$. Default maximum 200 elements; supports up to 5000. Returns { elements: [...], truncated: true/false }. Each element has value, and optionally packets and bytes for counter-enabled sets. Used by the IP Sets status tab for runtime member display.

mwan3.nftset_members { set: "<name>" }

Returns the current members of a named nft set in table inet mwan3. The set name is validated against ^[a-zA-Z0-9_-]+$. Returns { members: [ ... ] }. Used by the Simulator for ipset rule matching and connected-network bypass detection.

When a set has the counters flag enabled, nft wraps each element in the JSON output as {"elem": {"val": "<addr>", "counter": {...}}} rather than returning the address directly. nftset_members unwraps this structure so that counter-enabled sets return the same flat address list as non-counter sets.

mwan3.resolve_host { host: "<name>", family: "<ipv4|ipv6|>" }

Resolves a hostname by querying the local dnsmasq instance via /bin/busybox nslookup. Querying 127.0.0.1 populates any dnsmasq nftset entries configured for the domain as a side effect. Returns { v4: [ ... ], v6: [ ... ] }. When family is "ipv4" only A records are queried; when "ipv6" only AAAA records; when empty (or omitted), both are resolved. The hostname is validated against ^[a-zA-Z0-9._-]+$ before any shell invocation; invalid input returns { error: "invalid hostname", v4: [], v6: [] }. The DNS server echo line in nslookup output (which also contains an Address: prefix) is filtered by checking for the loopback address.

mwan3.nftset_flush { set: "<name>" }

Flushes all elements from the named set via nft flush set inet mwan3 <name>. The set name is validated against ^[a-zA-Z0-9_-]+$. Returns {} on success or { error: "..." } on failure. Used by the Flush button on the IP Sets status tab.

mwan3.nftset_reload { set: "<name>" }

Flushes the set then re-adds all static entries from the corresponding UCI config ipset section. Collects list entry values and, if loadfile is set, reads addresses line by line from the file (stripping comments and whitespace). Passes all entries to nft add element inet mwan3 <name> { ... } in a single invocation. Returns {} on success or { error: "..." } on failure. If no UCI section is found for the set name, or the section is disabled (enabled=0), returns {} without error after flushing. Used by the Reload button on the IP Sets status tab.

mwan3.nftset_resolve { set: "<name>" }

Looks up the named set in UCI, collects list domain entries, sends SIGHUP to dnsmasq via ubus.call('service', 'signal', {name:'dnsmasq', signal:1}) to clear its DNS cache, then queries each domain through the local dnsmasq instance (127.0.0.1) using nslookup_resolve(). The queries trigger dnsmasq's nftset= population mechanism as a side effect; no explicit nft add element call is needed. The address family (A or AAAA) is derived from the set's family UCI option. Returns { resolved: N } where N is the count of domains that returned at least one address. Returns { error: "..." } if the set is not found in UCI or has no list domain entries. Used by the Resolve button on the IP Sets status tab (only shown for sets with domain entries).

mwan3.routing_health {}

Compares the UCI configuration against live kernel state. For each mwan3 interface (by 1-based UCI order index N):

Checks for ip rule at priority IIF_BASE+N (iif), FWMARK_BASE+N (fwmark), UNREACHABLE_BASE+N (unreachable), and SRC_BASE+N (source, IPv4 interfaces only) via rtnl RTM_GETRULE. Base values read from UCI globals, defaulting to 1000/2000/3000, with SRC_BASE derived as UNREACHABLE_BASE + iface_max + 1.
Checks routing table N for a default route via rtnl RTM_GETROUTE
Reads /var/run/mwan3track/<ifname>/STATUS for current online/offline state
Reports stale ip rules (priorities in mwan3's range with no matching UCI interface)
Reports whether mwan3 is actively running (presence of any STATUS file under /var/run/mwan3track/)

16. Iptables-to-nftables Porting Notes

Key translation patterns used in this port, useful for anyone maintaining or extending the code:

iptables Concept	nftables Equivalent	Notes
`iptables -t mangle`	Chains in `table inet mwan3`	mwan3's own standalone table at mangle priority
`-A PREROUTING -j chain`	Own hook chain at `priority mangle + 1`	No need to jump from fw4's chain; own table is independent
`-A OUTPUT -j chain`	Own `type route` hook chain	Must be `type route` for mark-based rerouting
`iptables-restore -T mangle -n`	`nft -f batchfile`	Batch file for atomic multi-command operations
`-N chain`	`nft add chain inet mwan3 name`
`-F chain`	`nft flush chain inet mwan3 name`
`-X chain`	`nft delete chain inet mwan3 name`	Must be empty first
`-D chain match...`	`nft delete rule ... handle N`	Must look up handle with `nft -a`
`-j MARK --set-xmark V/M`	`meta mark set meta mark & ~M \| V`	See `mwan3_nft_mark_expr()`; use `&`/`\|` symbols not keywords
`-j CONNMARK --restore-mark --nfmask M`	vmap-dispatch into `mwan3_or_meta_<imm>` setter chains	Non-destructive masked restore. Kernel rejects the compound `(meta mark & ~M) \| (ct mark & M)`; vmap-dispatch synthesises the same effect via per-mark OR-immediate setter chains. See §2 Connmark Operations.
`-j CONNMARK --save-mark --nfmask M`	`ct mark set ct mark & ~M`, then vmap-dispatch into `mwan3_or_ct_<imm>` setter chains	Non-destructive masked save. Same kernel limitation, same vmap-dispatch workaround.
`-m mark --mark V/M`	`meta mark & M == V`
`-m set --match-set S dst`	`ip daddr @S`	Set lives in `table inet mwan3`
`-m statistic --probability P`	`numgen inc mod N map { ... }`	Deterministic round-robin instead of probabilistic
`-m multiport --dports P`	`th dport { P1, P2 }`	`th` = transport header (works for tcp/udp)
`-m icmp6 --icmpv6-type T`	`icmpv6 type { T1, T2, ... }`
`-p ipv6-icmp`	`icmpv6 type { ... }`	Protocol match is implicit
`ipset create S hash:net`	`set S { type ipv4_addr; flags interval; auto-merge; }`	Defined via `config ipset` in mwan3 UCI; `auto-merge` handles overlapping elements
`ipset add S element`	`nft add element inet mwan3 S { element }`
`ipset flush S`	`nft flush set inet mwan3 S`
`ipset create S hash:ip,mark`	`map S { type addr : mark; flags dynamic,timeout; }`	Maps store key->value pairs
`-j SET --add-set S src,src`	`update @S { ip saddr : meta mark & M }`
`-m set --match-set S src,src`	`meta mark set ip saddr map @S`	Regular map lookup (not `vmap` which requires verdicts)
Separate ipv4/ipv6 chains	Single `inet` chain + `meta nfproto`	Or just `ip`/`ip6` selectors in rules

Warning

Key kernel limitations to be aware of:

No compound two-source bitwise: Expressions like meta mark set meta mark | ct mark & X or ct mark set ct mark & ~M | meta mark & M fail with "Operation not supported". Each set expression can only draw from one register source. Workaround: synthesise the masked operation via vmap-dispatch into per-mark setter chains whose body is a single-source meta/ct mark | <constant immediate>. mwan3 uses this for masked connmark save and restore - see §2 Connmark Operations.

No numgen in compound expressions: meta mark set meta mark & COMP | numgen inc mod N map { ... } fails for the same reason. Use meta mark set numgen ... with a guard condition ensuring the target bits are already zero.

vmap vs map: vmap expects verdict values (accept/drop/jump), not data values like marks. For IP→mark lookups, use regular map.

nft add set flag immutability: Creating a set is idempotent, but flags (like auto-merge) cannot be updated on existing sets. Must delete and recreate to change flags.

17. Command Line Tools

17.1 mwan3-lb-test: Load Balancing Distribution Verifier

A diagnostic tool /usr/sbin/mwan3-lb-test verifies that load balancing is distributing traffic across policy members in the expected proportions.

Usage

mwan3-lb-test [-6] -c <client_ip> <policy_name> [ip1 ip2 ...]
mwan3-lb-test cleanup

-6 selects IPv6 mode. -c <client_ip> is mandatory and specifies the LAN client that will run the test pings. Optional IP arguments override the default destination pool. The cleanup subcommand removes stale sets and rules from an aborted run.

Design

NITER computation: The number of test iterations is computed from member weights using GCD: base_N = total_weight / GCD(weights), NITER = base_N * ceil(30 / base_N). This ensures per-member expected hit counts are whole numbers and that NITER is always at least 30.
ICMP-only test rule: A temporary meta l4proto icmp ip daddr @mwan3_lb_test_<PID> counter rule is inserted into mwan3_rules ahead of user rules. The ICMP restriction prevents DNS queries, TCP connections, and other traffic from contaminating the count. -6 mode uses meta l4proto ipv6-icmp ip6 daddr @set.
Client isolation: A forward chain drop rule blocks pings to the test destination set from all LAN clients except the nominated test client (-c). An mwan3_output return rule bypasses mwan3 marking for any router process pinging the same IPs. Both rules are scoped to the test set and removed on exit.
Tracking IP exclusion: The default destination pool excludes IPs already configured as mwan3 track_ip values. mwan3track pings those IPs via mwan3_output -> mwan3_rules, which would match the test rule and inflate the count.
Windows command: A cmd.exe for loop is output alongside the Linux shell loop. Windows ping uses a fixed ICMP identifier (id=1), causing conntrack entry reuse on repeated pings to the same destination. The Windows command uses an inter-ping delay of 30/TRACK_COUNT + 3 seconds so the full cycle through all test IPs exceeds the 30s ICMP conntrack timeout, ensuring each revisit generates a fresh conntrack entry. The IP list is formatted with ^ line continuation at 4 IPs per line.
Cleanup: Removes the temporary set and rules on normal exit, SIGINT, SIGTERM, and SIGPIPE. A startup sweep removes stale mwan3_lb_test_* sets and rules from any aborted previous run.

Files changed: usr/sbin/mwan3-lb-test (new), Makefile

17.2 mwan3-diag: Network Diagnostic Report

mwan3-diag is a ucode diagnostic script installed to /usr/sbin/mwan3-diag that collects a comprehensive snapshot of the network state relevant to mwan3 operation. It is intended to produce a report that can be posted in a forum thread or bug report without manual redaction.

Usage

mwan3-diag

The script collects interface addresses, routing tables (including all per-WAN tables), policy rules, neighbour cache, mwan3 interface status, the mwan3 UCI configuration, the complete mwan3 nftables ruleset, the fw4 mangle chains that interact with mwan3 packet marking, and the last 200 lines of the mwan3 log.

Before printing any output the script builds a map of every public routable IPv4 and IPv6 address present in the collected data and replaces each one with a stable placeholder -- PUB4_1, PUB4_2, PUB6_1 and so on -- throughout the entire report, including free-form text such as nftables rules and log lines. The same address always receives the same placeholder, so cross-references between sections remain consistent. Private addresses (RFC1918, link-local fe80::, ULA fc00::/7, loopback) are left unchanged as they are diagnostically important. The elements of user-defined nftables sets are replaced with { ... } rather than disclosed.

Files changed: usr/sbin/mwan3-diag (new), Makefile

18. Changelog

18.1 Version 3.6.12

Summary: Adds IPv4 source-address return routing. Router-originated traffic sourced from a WAN's own address must leave by that WAN, but a sender that routes itself outside netfilter's view followed the main table out the default WAN and was dropped upstream or masqueraded to the wrong address. WireGuard is the case that motivates it, where replies to a handshake arriving on a non-default WAN left by the default WAN and the handshake never completed. One ip rule per global-scope address on each WAN now sends those lookups back to the arrival WAN. The rpcd status method reports the new rules and the web interface shows them alongside the existing three on its routing health view.

A route wide enough to carry the whole internet without being a literal default is now treated as part of a default route when the bypass sets are built. VPN clients routinely install exactly that, OpenVPN with redirect-gateway def1 and split-tunnel WireGuard installing a pair of half-width routes rather than a default, and some IPv6 providers pushing the global-unicast prefix. One such route anywhere mwan3 scanned handed the entire address family to the bypass and silently stopped rule steering, sticky pinning, blackholing and inbound reply-path classification from working. Routes in the extra lookup tables are also no longer recorded in the connected-networks set, where a stale copy could outlive the route itself and keep a destination exempt from policy routing for hours.

Three faults in rule building are corrected. An nft set referenced by rules of both address families but defined nowhere in the config was created twice with conflicting types, which nft rejected, leaving the router with no user rules at all. A sticky rule matching both families built its per-rule chain with IPv4 lookups only, so IPv6 flows were never pinned and stickiness silently applied to one family, and for ICMP the late IPv6 lookup produced a mark matching no interface at all. IPv6 policy members are no longer added on kernels without IPv6, where a policy with no usable IPv4 members steered IPv4 traffic to an IPv6 member instead of falling through to its last resort action.

Three further faults affect the service itself. A service that started with every interface disabled never created its nft table, so a later config change reloaded against nothing and left the router with no ruleset until a manual restart, and now falls back to a full restart instead. The two route monitor instances shared a single nftables batch file and could race on it around WAN transitions and prefix renewals, leaving the bypass sets stale, and each now has its own. The route seeding helper counted an interface that omits the enabled option as enabled, where every other consumer of an interface section treats it as disabled, which withheld that device's routes from every other interface's table.

The diagnostic report no longer prints public addresses that appear only in the nft ruleset or the log, service stop time is reduced by dropping tear-down code left over from the fw4 era, and the shipped base files now link to the nftables documentation rather than the legacy iptables page. In the web interface, address lists in table columns are reduced to their first entry and a count of the remainder, so a long comma-separated list no longer stretches or overflows the cell, and the address checks now call the framework's own parsers.

mwan3: update documentation links in shipped base files

The default UCI config and the hotplug script both linked to the legacy iptables wiki page. Point them at the nftables page, whose sections describe the configuration options and hotplug environment that this package actually implements.

mwan3: keep extra lookup table routes out of the connected set

Destinations with a route in one of the rt_table_lookup tables are exempt from policy routing. The route monitor recorded these routes in two places: the set that tracks them and the set that tracks directly connected networks. The first is updated whenever a route in those tables is added or deleted, but the second is only rebuilt when the main routing table changes. Deleting a route from an extra table therefore ended its exemption in one set while the copy in the other kept it alive, and traffic to that destination continued to bypass policy routing, for hours or days on a quiet router.

Record only main table routes in the connected networks set, as the startup code already does. Nothing is lost by this: the set dedicated to the extra table routes grants the same exemption and is kept current.

mwan3: keep default-equivalent routes out of the bypass sets

Destinations that mwan3 should not steer are collected into bypass sets built by scanning the routing tables, and anything in those sets is marked for the main table before and after the user rules run. The scan recognises a default route only in its literal form, so a route wide enough to carry the whole internet without actually being a default looks like an ordinary network to it. VPN clients routinely install exactly that: OpenVPN with redirect-gateway def1 and split-tunnel WireGuard install a pair of half-width routes and not a default, and some IPv6 providers push the global-unicast prefix. One such route anywhere mwan3 scans hands the entire address family to the bypass, and rule steering, sticky pinning, a blackhole and inbound reply-path classification stop working silently.

Consider a route at or below /2 for IPv4 and /3 for IPv6, or with no destination at all, to be a component of a default route. Nothing a real network hands out is that wide. Anyone who genuinely wants a broad bypass still has the bypass_network option. Routes are judged one at a time, so a client that has installed only half its pair is covered too.

Only the bypass sets use the wider definition. Filling an interface's own table with routes, and taking that interface offline when its table holds no default both still require a literal default route.

mwan3: fix interface enabled in route seeding helper

The interface route seeding helper read an interface section's enabled option with a default of 1, where every other consumer of an interface section defaults it to 0. The init script, the hotplug handler, the interface state paths and the route monitor all treat a section that omits the option as disabled.

The defect is longstanding: the shell function this helper replaced called config_get_bool without its default argument, so a section with no enabled option left the variable empty and the numeric test meant to skip the interface failed on an empty string rather than skipping it.

Counting the interface as enabled put its device into the device-to-table map, which withheld that device's routes from every other interface's table while delivering them to a table that is never built, because mwan3 never brings up a disabled interface. Nothing relied on that.

The visible consequence was that traffic in another interface's table addressed to a subnet directly connected to that device found no specific route and fell through to that table's default route.

mwan3: fix public addresses leaking into the mwan3-diag report

Some public addresses were printed unmasked. The anonymiser replaces only addresses learned from neighbours, routes, rules, interface addresses, status and config, so an address appearing only in the nft ruleset or the log, such as a NAT map element, was printed as-is.

Now the nft and log text the report prints is also swept for addresses before placeholders are assigned, so every public address in the report is masked.

mwan3: give each mwan3rtmon instance its own nft batch file

mwan3rtmon runs as two long-lived instances, one per address family, and both wrote their nftables batches to the same fixed-name file, opened with truncation and never removed. When both families' routes changed at the same time, as is common around WAN transitions and prefix renewals, the two instances could race on the shared file: one could truncate or rewrite it while the other was writing or loading it with nft -f, so a load could apply a truncated, mixed or wrong-family batch, or fail outright, leaving the connected or custom destination sets stale until the next route change rebuilt them. The last batch file was also left behind after every load.

Name the batch file with the instance's address family, which is sufficient because one instance runs per family and each instance is single-threaded, and remove the file on every exit path after it is created, matching the per-process, self-cleaning convention of the shell batch machinery in common.sh.

mwan3: create a missing set only once per rebuild

A rule can match on an nft set that mwan3 does not define in its own config, leaving it to be created in table inet mwan3 by something else, or never created at all through a config error. mwan3 creates an empty set when it finds one missing, as an unresolved set reference would break the atomic nft -f batch load.

The address type came from whichever rule needed the set, so two rules matching the same missing set from different families each created it, with conflicting types. nft rejected the batch, leaving the router with no user rules at all.

mwan3 now remembers the sets it creates as it builds. The first rule to need a missing set settles its address type, while a later rule of the other family is skipped, quietly if it is a family=any rule, or with a warning if the rule is family specific.

mwan3: skip IPv6 policy members on kernels without IPv6

Ensure that IPv6 policy members are only added when the kernel supports IPv6. Such members previously bypassed member selection but were still added to the chain, so a policy with no usable IPv4 members steered IPv4 traffic to an IPv6 member instead of falling through to its last_resort action.

mwan3: complete sticky chains for family=any rules

A sticky rule with family=any and no family-specific match element is rendered once, as a single line matching both address families, but the per-rule sticky chain it jumps to was built with IPv4 sets, lookups and updates only. IPv6 flows matching the rule entered the chain, missed every lookup and were never pinned: the policy re-evaluated each new flow, so connections from one source could egress different WANs. Stickiness silently applied to IPv4 only.

For proto=icmp such a rule renders on both family passes and each pass appended its own family's sticky arm to the shared chain. The IPv6 lookups landed after the IPv4 arm's fall-through, so pinned IPv6 sources were re-marked by the policy first and the late lookup then ORed the old pin into the new mark, producing a value matching no interface.

Fix both by building the per-rule chain exactly once, complete for every family whose traffic can reach it: all lookups, then the single fall-through to the policy, then all update rules. Rules with an explicit family or a family-specific match element render as before.

mwan3: report IPv4 source rules in routing_health

The routing_health method reported the iif, fwmark and unreachable ip rules per interface but not the IPv4 source-address return routing rules, so a consumer could neither display them nor factor them into an interface's health.

For each IPv4-family interface, add a src_rule object carrying the rule's presence and priority (unreachable base + interface maximum + 1 + index), and extend rule_bases with the source base. Presence is tested against a family-filtered index of the IPv4 rule dump so an unrelated IPv6 rule sharing the priority is not counted. A device with several global IPv4 addresses holds one rule per address at this single priority, so presence means the slot is populated rather than that every address is covered. IPv6-family interfaces carry no src_rule key.

mwan3: fix whitespace in rpcd module

Adjust blank-line spacing around several comment blocks in the rpcd module for consistency with comment formatting elsewhere, and remove one blank line between two variable declarations.

mwan3: restart on reload when the nft table is missing

When the service starts with every interface disabled, start_service returns before creating the nft table, yet the status directory makes it report running, so a config change drives a reload that rebuilds against the missing table. The batch fails and mwan3 is left with no ruleset until a manual restart.

Guard reload_service with a table-existence check and fall back to a full restart when the table is absent.

mwan3: optimise service stop time

Optimise mwan3 service stop time by removing table inet fw4-era manual tear-down code.

mwan3: add IPv4 source-address return routing

Router-originated IPv4 traffic sourced from a WAN's own address must leave by that WAN, but a sender that routes itself outside netfilter's view follows the main table out the default WAN and is dropped upstream or masqueraded to the wrong address.

WireGuard is a good example: replies to a handshake arriving on a non-default WAN leave by the default WAN and the handshake never completes. Locally originated flows that no user rule classifies share the gap.

Install one ip rule per global-scope IPv4 address on each WAN, from <addr> lookup <table>, on the same lifecycle as the interface's existing rules. Route lookups then resolve back to the arrival WAN and the reply keeps the correct source. Marked traffic still outranks these rules, so policy verdicts and the fail-closed backstops behave as before, and the rules are inert for forwarded traffic and single-WAN setups.

Update the rpcd module to recognise the new source rules, which its routing health check would otherwise report as stale.

luci-app-mwan3: fix ACL grant for optional binary detection

Detection of the optional tracking binaries runs through fs.stat, whose ubus method was not granted by the configuration group. Sessions holding only this app's ACL groups therefore offered ping as the only tracking method even with the binaries installed. Grant the stat method.

luci-app-mwan3: replace custom validator calls with framework calls

The rule and IP set address checks ran the ipaddr datatype through a custom validator object.

Add ipAddrOrPrefix() and ipv6Cidr() to the validators module, both built on the framework's parseIPv4() and parseIPv6(). The three address checks now call ipAddrOrPrefix(). The custom validator stays for the types the framework publishes no parser for.

luci-app-mwan3: summarise address lists in table columns

The source and destination address options accept a comma-separated list, and every table that previews a rule rendered that list in full. A comma-joined list is a single long token with nothing to break on, so it stretches an auto-layout table column and overflows the cell of a fixed-layout one.

Add a shared formatter that reduces such a list to its first entry followed by a count of the remainder. The stored value is untouched and the editing widgets still show every entry, so the complete list remains available in the rule modal.

The source preview is reshaped to build its text from a list of parts, matching the destination preview directly below it, which already does so because it appends a mark element.

The translation template is regenerated. The only changes are moved source line references, not strings.

luci-app-mwan3: regenerate translation template

The translation template had not been regenerated since the last change that added strings to it. Regenerate it from current sources. No functional change.

luci-app-mwan3: show IPv4 source rules on Routing Health

The routing_health backend now reports the IPv4 source-address return routing rules, but the Routing Health tab rendered only the iif, fwmark and unreachable rows and ignored the source band.

Render a fourth "IP rule (source)" row on IPv4-family interface cards, keyed off the src_rule object so it never appears where the concept does not apply, and fold the source rule into the online health judgement so a missing rule degrades the card. Document the rule in the field reference, covering its purpose, lifecycle, the one-rule-per-address behaviour and its IPv4-only scope, and update the closing summary for the fourth row. Add a src fallback to the shared rule-base defaults for when the backend does not report bases.

18.2 Version 3.6.11

Summary: Fix the nft rule rendering order such that it preserves strict config order.

mwan3: render user rules in config order

The user rule renderer looped over address family first and the config rules second, so the installed mwan3_rules chain held the entire IPv4 render pass followed by the entire IPv6 render pass. Config order was preserved within each pass but not across the pass boundary.

A family=any rule with no IP-version-specific match element (for example a bare protocol and port rule, or an fwmark rule) is emitted once, on the IPv4 pass, as a single family-agnostic line that also matches IPv6 traffic at runtime. Because the whole IPv4 pass rendered first, that line always landed ahead of every IPv6-only rule in the chain regardless of its position in the config, and shadowed any IPv6 rule the configuration placed before it. Rule evaluation is first match, so no config ordering could place an IPv6-only rule ahead of a family-agnostic one, and reordering rules in the config silently failed to change evaluation order.

Invert the loop nesting: iterate the config rules in order and render each rule's IPv4 and IPv6 variants at that rule's position. The per-rule, per-family emission function is unchanged, so every rendered line is identical; only the order of the lines in mwan3_rules changes, and it now always matches config order. The chain is a flat first-match list with no structural dependency on the old pass grouping, and per-rule sticky chains and sets are keyed by rule and family, so their content is unaffected.

For configurations that interleave family-agnostic rules with family-specific ones, evaluation order now follows config order strictly.

18.3 Version 3.6.10

Summary: This release corrects six longstanding bugs, all present in the original codebase, fixes some privilege escalation issues, corrects a tracking-health edge-case, modularises and refactors the LuCI app for maintainability and to make exclusive use of LuCI's published API, and makes ucode scripts log to syslog. Normal operation while every internet connection is healthy is unchanged throughout.

Four of the changes address what happens at the moment a connection fails, recovers, or is brought up or down. When a connection was detected as failed while still physically connected, the records of which sessions and devices were using it were not cleared, so traffic stayed pointed at the dead connection for minutes instead of moving to a working one.

When a preferred connection later recovered, traffic that had diverted to a backup did not return to it. Both are now corrected: on failure the records are cleared immediately, and on recovery new traffic returns to the better connection while sessions already in progress finish where they are.

The order in which routing and connection selection are updated during a transition was also wrong in both directions: on recovery, a connection was made available for traffic before its route was rebuilt, so traffic was assigned to it before it could carry any. On shutdown, a connection's route was removed before traffic stopped being assigned to it. Both orders are now correct.

A per-connection safety block that rejects traffic with no working route was being removed when a connection went down, at the exact moment it was needed. It is now kept in place for as long as the service runs, so traffic with no route is rejected rather than escaping over the default internet link.

All forwarding-rule changes during a transition are now applied as a single all-or-nothing update, closing a brief window in which the rules were only partly in place and traffic could be steered by an incomplete set of them.

A separate fix removes a fallback rule that silently ran in place of a policy's configured behaviour for when all its connections are down, sending traffic out the default link even when the policy was explicitly set to block it.

The sixth fix corrects corruption of per-entry expiry times in address sets populated by a DNS resolver: a kernel range-merging optimisation was discarding the individual expiry of each address and replacing it with the expiry of whichever address had last extended the merged range, causing entries to expire too early or be kept alive too long.

Three further fixes harden the service against a user who has been granted only limited, delegated control of the mwan3 configuration, closing routes by which that restricted role could take over the whole router or another user's web session. Two of them address ways such a user could gain administrator-level command execution: one web-interface permission let them edit a script that mwan3 runs as the administrator on every connection event, and two internal helpers built system commands out of configuration values, such as a domain name or an address-set entry, without first checking them. That editing permission is now separated into its own group that ordinary configuration access no longer carries, and every value is checked before it reaches a command. The third fix escapes configuration text before it appears on the status pages, so text saved by one user can no longer run as code in the browser of anyone who later views those pages.

Several further, unrelated changes round out the 3.6.10 release. A missing default route on an interface's own routing table is now treated as a tracking failure, so an interface that its tracker would otherwise still report as online is taken offline instead of blackholing the traffic steered to it. The routing-health view in the web interface has been updated to match current behaviour, in which the per-interface reject rule is a permanent fixture rather than one that appears and disappears with the interface, so its presence on an offline interface is now shown as expected rather than as a fault. The web interface itself has also been reworked to build on LuCI's standard, supported components and split into shared modules for easier maintenance, with no change to how it behaves aside from a few small display corrections. Finally, the ucode components now log to the system log with a per-component tag, matching the shell components.

mwan3: validate ipset elements and resolver hosts before use

The rpcd backend runs as root. Two of its methods built command lines from UCI-derived values with no validation and ran them through popen, executing via a shell fork. nftset_resolve passed each domain straight to the nslookup command line and nftset_reload joined the section's entry values and the lines of a loadfile into an nft add element command line. A user holding only a delegated config role has UCI mwan3 write plus these methods, so a crafted domain, entry, or loadfile line could potentially run arbitrary commands as root.

The same entry list is consumed a second time by the ordinary ipset apply path, which builds an add element batch that is later committed with nft -f. There a crafted value could close the element braces early and inject rogue nftables statements, executed as root on the next apply.

Reject anything that is not a plain hostname before it reaches nslookup. Validate every ipset element, from both the entry list and the loadfile, ensuring it is a valid IPv4 or IPv6 address, a CIDR, or a hyphen range, and drop anything else before it reaches a command line. Apply the element check both in the ucode and the shell path.

mwan3: log ucode components to syslog

Route the ucode logging through a new shared logging function housed in common.uc that writes to syslog with a per-component tag, just like the shell components do via the logger tool.

Add a ucode-mod-log dependency.

mwan3: take an interface offline when its table has no default route

An interface's tracker reports it online on the strength of its probes, which reach their target over a connected or point-to-point route and do not need a default route. So an interface whose own routing table holds no default route still reports online: mwan3 marks traffic to it, its fwmark ip rule sends that traffic to the interface's table, the table has no default, and the per-interface unreachable backstop drops it. The interface black-holes every flow steered to it while still showing online.

Make a missing default route count as a tracking failure. The route monitor, which already maintains each interface's own table, checks whether that table holds a default route via the interface's device and writes up or down to a small file in the tracker's status directory. The tracker reads it once per round and, when it reads down, skips the probe so the round fails; the existing score and hysteresis then carry the interface offline, and back online when the route returns, with no new state and no new hotplug arm. The check is on the interface's own table, not the main table, because the fwmark rule looks up exactly that table, and it is recomputed from a full route dump rather than a single route event, so an interface that never held a default also reads down. An absent file is treated as up, so a tracker with no route monitor is unaffected, and the unreachable backstop is unchanged, so traffic still marked to a downed interface fails closed rather than leaking.

mwan3: batch each failover transition into one nft transaction

When a WAN's usability changes, whether it stops being usable (a tracker-detected soft failure or an administrative or carrier down) or becomes usable again (a tracker-detected recovery, or coming back up while already online), mwan3 rebuilds its routing policy, and on a teardown also clears the per-source stickiness records that kept traffic on the failed WAN. Until now mwan3 issued these as several separate firewall operations.

Because they were separate, the firewall could be observed in an intermediate state during the transition: a packet could be classified against a policy that had been flushed but not yet fully rebuilt, and on a teardown the stickiness records were cleared in a step that could briefly disagree with the rebuilt policy. These windows are sub-second, but they are real and serve no purpose.

Apply the policy rebuild as a single firewall transaction on every transition, both the teardown paths and the recovery paths, committed in one step. The kernel applies the whole transaction at once, so traffic is never steered by a half-built policy. The resulting ruleset is identical to what the separate operations produced; only the number of kernel transactions changes.

On a teardown the stickiness clearing joins the same transaction, in the fixed order of the policy first and the stickiness second, because the stale stickiness points at a WAN that is going away and must be cleared in the same step that drops it from the policy. On a recovery the stickiness reset that returns new traffic to the recovered WAN stays a separate step after the rebuild: it only resets records pointing at lower-priority WANs that are still working, so a connection that consults one in the instant before the reset still reaches a working WAN, and its timing is not safety-critical.

The connection-tracking cleanup stays a separate step at the end of the transition, because connection-tracking marks are kernel state outside the firewall ruleset and cannot share its transaction.

This corrects longstanding behaviour rather than a regression; the separate-operation form predates mwan3's use of atomic firewall transactions.

mwan3: install static per-WAN unreachable backstops

mwan3 sends a connection out a chosen WAN by tagging it with a firewall mark and adding an ip rule that routes that mark to the WAN's table. Next to each of those it also installs a backstop rule that rejects the mark as unreachable when the WAN has no usable route. That backstop is what keeps traffic tagged for a WAN from quietly slipping out the main routing table, and the default route, when the WAN cannot carry it.

The bug was that the backstop was added and removed together with the WAN's routing rules, so it only existed while the WAN was up. It was therefore missing at the worst possible moment: right after a WAN goes down, or during the split second its rules are torn down and rebuilt, a connection still tagged for that WAN had nothing to reject it and leaked out the default route. On a policy meant to be a hard boundary, such as a VPN, that is a genuine leak.

The fix makes the backstops permanent. mwan3 now installs one for every configured WAN when the service starts and on every reload, and removes them only when the service stops. The rules that actually route into a WAN still come and go with it, because they need the live interface, but the reject rule stays put. Traffic tagged for a WAN with no matching route, whether the WAN is down, disabled, or briefly mid-rebuild, is now rejected instead of leaking. This covers every configured WAN, including disabled and offline ones, since a WAN disabled while its connections are still live leaves tagged traffic that has to be caught.

The backstops use only real WAN marks, never mwan3's reserved marks, so a policy's last_resort setting is untouched: last_resort "default" still deliberately uses the main routing table, "unreachable" still rejects, and "blackhole" still drops. The new rules are created through the same helper that already builds mwan3's global rules, and the existing stop-time cleanup already removes them. This corrects long-standing behaviour carried over from the old iptables-based mwan3; it is not a regression.

mwan3: clear stale WAN pins on soft failover and snap back on recovery

When a WAN goes up or down, mwan3 has to bring two caches back into step with the routing policy it rebuilds. Both caches decide a packet's WAN before the policy is consulted, and both can override it: the sticky sets, which tie a source address to a WAN without checking that the WAN is still up, and the connection marks kept by conntrack, which tie an existing connection to the WAN it started on. Both refresh themselves as traffic flows, so once either points at a WAN the policy has dropped, the mistake lasts for minutes rather than for the moment of the change.

On a soft failure, where the tracker judges a WAN dead but the interface itself stays up, mwan3 dropped the WAN from the policy but cleared neither cache. A source pinned in the dead WAN's sticky set was still tagged for it, and a connection already marked for it was still restored to it, so traffic either black-holed on the dead WAN or, when that WAN's routing entry was already gone, escaped out the main routing table's default route. Every leaked packet re-armed the sticky timer, so the wrong state held itself open. A hard interface-down cleared the affected connection marks; the soft failure cleared nothing.

Now both caches are cleared on the soft-failure event, after the policy is rebuilt, in the order a teardown needs: policy first, then the sticky sets, then the connection marks. A sticky entry can stamp its WAN onto a connection's mark, but a connection's mark never feeds back into the sticky sets, so clearing in this order leaves nothing already-cleared to be re-dirtied. The targeted connection-mark flush that a hard interface-down already performed is now applied to the soft failure too. A pinned source picks the survivor on its next new connection, and an existing connection takes one reset and reconnects on the survivor instead of hanging until the dead WAN returns. Protocols that can move a live connection by themselves, such as QUIC, carry on without a reset.

On recovery, new traffic is moved back off the backups the returning WAN outranks. mwan3 ranks a policy's WANs by metric, lower winning, and keeps IPv4 and IPv6 as two separate contests, so a recovering WAN competes only with others of its own address family. When a higher-ranked WAN comes back, the sources parked on a lower-ranked backup of the same family should return to it. For each policy the recovered WAN belongs to, mwan3 clears the sticky sets of that policy's lower-ranked, same-family WANs, and only for the rules that use the policy, so a parked source re-picks on its next new connection while connections already running on a backup are left alone. The narrowing matters in two ways. Keeping to the WAN's own family means an IPv4 recovery never disturbs IPv6 stickiness, or the reverse. Keeping to the policy and its rules means a WAN that is a backup in a failover policy and an equal-ranked WAN in a separate load-balancing policy keeps its load-balancing stickiness, which clearing by WAN alone would wrongly wipe. In a load-balancing policy every WAN ranks equally, so nothing is cleared and steady-state stickiness is kept.

These are longstanding behaviours carried over from the old iptables-based mwan3: the sticky scheme never checked whether a WAN was still up, and the soft-failure path never cleared either cache.

mwan3: restore a WAN's routing on recovery and rebuild policy before teardown

When a WAN changes state, mwan3 has to update both the routing policy and that WAN's own ip rules and route. The handler that does this updated them in an order that could leave a packet tagged for a WAN whose routing was not in the matching state.

When the tracker brought a WAN back up, the handler rebuilt the policy but, unlike a normal interface bring-up, never recreated that WAN's own routing rules and route. A WAN restored this way was added back to the policy and had traffic tagged for it, yet nothing routed that traffic until the next reload, so it was misrouted or dropped. The recovery path now recreates the WAN's rules and route before rebuilding the policy, the same as a normal bring-up does, so a recovered WAN can carry traffic the instant it is tagged for. Re-creating the rules is harmless when they already exist, so a recovery that races a WAN that is already up just re-asserts the same rules.

When a WAN was taken down, by an administrator or by losing carrier, the handler removed that WAN's routing and cleared its sticky sets before rebuilding the policy. In the gap between removing the routing and rebuilding the policy, the still-stale policy could tag traffic for the downed WAN, whose routing rule was already gone, and a packet crossing that gap re-filled the sticky set that had just been cleared. The down path is reordered so the policy is rebuilt first, dropping the WAN before its routing is removed, and the sticky sets are cleared after the rebuild, so the rebuild can no longer re-fill a set it just cleared.

Both behaviours are longstanding, carried over from the old iptables-based mwan3. Steady-state routing and the built ruleset are unchanged; the difference shows only across a change of state.

mwan3: remove offline-WAN fallback that overrode last_resort

When a policy had no WANs online, the policy builder added an extra rule just ahead of the last_resort rule, as part of a broader link-selection change. It matched traffic that would have left through an offline WAN's device and tagged it for the main routing table. Because it sat before the last_resort rule, and both act only while the firewall mark is still unset, it ran instead of last_resort for that traffic.

This silently overrode a deliberate setting. An operator who sets last_resort to unreachable or blackhole is telling the policy to fail closed when none of its WANs is available. The fallback instead sent some of that traffic out the main table. It picked that traffic by which device it would leave through, not by where it was going, so during a soft failure, where the failed WAN's device is still present and its default route is still in the main table, it also caught ordinary default-routed traffic, not just traffic to directly reachable hosts, and sent it back out the dead WAN via the main table. For a one-WAN policy used as a kill switch, such as a router-originated VPN tunnel with last_resort unreachable, this defeated the fail-closed behaviour for exactly the traffic the operator meant to block.

Removing the fallback does not change reachability of directly connected or local destinations. Those are sent to the main routing table by the connected-destination and custom-destination checks that run before the policy, so that traffic never reaches the policy and never relied on this rule. The connected set is built from the main table's non-default network routes, so it already covers directly connected subnets and other non-default routes. The fallback's only remaining effect was on default-routed traffic leaving through a fully-down policy's offline WAN, which is exactly the case that should obey last_resort.

The change is narrow. With last_resort default the policy already tags all its traffic for the main routing table, so nothing changes. Only last_resort unreachable and blackhole are affected, and only for default-routed traffic leaving a fully-down policy's offline WAN, which now fails closed as set. A destination reachable only by a host route out the offline WAN is not in the connected set and now obeys last_resort too; deliberate exceptions are still available through the custom-destination set.

The now-unused offline-device bookkeeping in the per-WAN policy walk is removed with it.

Fixes: 46ed09408e55 ("mwan3: refine link selection")

mwan3: disable auto-merge on timeout-bearing config ipsets

Config ipset sections are declared as interval sets and auto-merge has been applied unconditionally. When such a set also carries per-element timeouts and is populated host by host by an external resolver, for example dnsmasq nftset= or smartdns, auto-merge coalesces any run of contiguous host addresses into a single range. The merged range carries one timeout and the per-element timeouts of the other members are discarded. Under the incremental insertion a resolver uses, the surviving timeout is that of whichever element most recently extended the range: a short-lived member that extends a run of long-lived members evicts the whole range early when its TTL fires, and a long-lived member that extends a run keeps a short-lived member alive well past its TTL. For resolver-populated sets the per-element TTL is the entire point, so this silently corrupts membership.

auto-merge remains a useful memory optimization for static CIDR sets, so restrict it to sets without a timeout. The interval flag is retained so timeout sets can still hold CIDR entries; only the auto-merge coalescing is dropped.

luci-app-mwan3: move mwan3.user editing to its own ACL group

The configuration ACL group granted read and write on /etc/mwan3.user, along with the ubus file read, write and remove methods. That file is a shell script run as root on every mwan3 hotplug event, so anyone delegated the configuration role could edit it and run arbitrary code as root. That role therefore implied full root.

Move the /etc/mwan3.user grants and the ubus file read, write and remove methods into a new luci-app-mwan3-notify group, and make the Notify menu node depend on it. The configuration group keeps only the ubus file list method and the httping, nping and arping list entries it needs for the tracking-binary presence check. A configuration delegate now gets no access to the hook by default. Read on the new group lets them view it read-only, and write lets them edit it. The Notify page also hides its Save button when the session lacks write.

luci-app-mwan3: escape config-derived values in the status views

LuCI's E() and dom.append assign a bare string child to a node's innerHTML, where it is parsed as HTML, but append an array child or a node as escaped text. The mwan3 status and analysis views passed UCI-derived free-form values straight in as bare strings. A user who can write mwan3 UCI could store markup in any of these that then runs in the browser of anyone who later opens the status pages.

Add a small helper that returns a text node, change the shared inline primitives to append their text as an array child so it is never written to innerHTML, and pass each of these values through the helper. The troubleshooting view's diagnostic output is likewise appended as an array child, which both escapes it and keeps it verbatim.

luci-app-mwan3: refactor and modularise

App-wide refactor to use the published, supported client-side API throughout and modularisation to support maintainability.

Logic that had accumulated as copies across the individual views is factored into a small shared library that the views compose from: render helpers, value formatters, address and set arithmetic, input validators, the interactive widgets, and the default constants. Checks and defaults that were restated in several places now have a single definition. The result is much less code per view and a smaller footprint for future changes.

Visible behaviour and the configuration model are unchanged, aside from a few incidental display corrections.

luci-app-mwan3: update routing health display

mwan3 now installs the per-interface unreachable rule as a static backstop: it is created for every configured interface when the service starts and removed only when the service stops. It is therefore present regardless of whether the interface is online, offline or disabled, and it survives a hard ifdown. The routing health tab historically treated that rule as one that comes and goes with the interface, so it presented the now-normal presence of the backstop on a downed interface as exceptional.

Also correct how the tab describes a soft tracker failure: only a hard ifdown removes the iif and fwmark rules and flushes the table.

Update the display to reflect the actual rule lifecycle:

Presence of rules on an offline interface is expected.
Show a disconnecting event as yellow-bordered.
Update the field guide: a soft tracker failure retains all of the interface's rules; a hard ifdown removes the iif and fwmark rules and flushes the routing table, leaving intact the unreachable rule.

18.4 Version 3.6.9

Summary: Stop router-originated IPv6 link-local traffic, most visibly odhcpd's DHCPv6 replies to LAN clients, from being policy-routed into a WAN routing table and failing, by accepting single-link scopes before classification. Clear an inherited skb mark on ingress so a tunnelled-WAN reply is no longer pinned to the wrong routing table. Emit dnsmasq nftset= directives correctly when a domain is shared across multiple ipset sections, and stage the fragment atomically under the runtime directory.

mwan3: clear inherited mark on ingress before conntrack restore

A packet decapsulated from a tunnel interface can inherit the outer packet's skb mark. On a tunnelled WAN (a tunnel broker or an L2TP link, for example) an inbound packet can therefore arrive already carrying mwan3's mark bits; the reply of a LAN-initiated flow is the common case. The inherited mark is independent of the inner address family, so the same failure occurs for IPv4 as for IPv6.

That stale mark defeats the conntrack mark restore in the prerouting chain, which is guarded on the mwan3 mark bits being clear, so the restore is skipped. The unguarded save that follows writes the inherited mark to the connection's conntrack mark, overwriting the classification the forward direction stored. Every subsequent packet of the flow then restores the wrong mark and is dispatched to the wrong routing table, pinning established flows to the wrong WAN.

Add a rule at the start of the prerouting chain, before the restore, that clears mwan3's mask bits. The restore and the interface and policy classification then re-derive the mark from a clean slate. The rule is a no-op for traffic that arrives with the bits already clear, the normal case on non-tunnel WANs, and it preserves any bits outside mwan3's mask. Mwan3 owns its mask exclusively and the coexisting policy-routing and firewall layers use disjoint bits, so no upstream hook legitimately sets these bits on ingress; the clear is therefore unconditional and covers both address families.

mwan3: combine dnsmasq nftset directives for multi-family domains

dnsmasq silently ignores duplicate nftset= directives for the same domain: when a second directive names a domain already seen, it is discarded. This means that if an IPv4 and IPv6 ipset section both list the same domain, emitting separate per-section nftset= lines causes only the first set to populate.

Replace the per-section emission with a two-pass approach: pass 1 collects all (domain, set-specifier) mappings into a temp file, pass 2 sorts by domain and emits one nftset= line per unique domain with all its set targets comma-separated. The comma-separated multi-set format is documented dnsmasq grammar for the nftset directive.

This also fixes the same-family variant of the bug: a domain appearing in two sets of the same address family previously populated only the first set, and now correctly populates both.

mwan3: stage dnsmasq nftset fragment atomically under the runtime dir

The per-instance dnsmasq nftset fragment was staged at a fixed <fragment>.new path inside the dnsmasq confdir. This has two defects:

Two concurrent mwan3 invocations both write the same staging path and then rename it, so one can clobber or read the other's half-written file.
The transient .new file sits inside the directory dnsmasq scans, meaning dnsmasq can attempt to parse a partially-written or stale staging file depending on conf-dir filter configuration.

Stage under the mwan3 runtime directory using a per-process, per-instance unique name. The existing mv -f swap remains and is now a guaranteed atomic same-filesystem rename (both paths are on tmpfs), so dnsmasq only ever observes the complete final fragment. Concurrent runs never share a staging path by construction.

mwan3: never policy-route IPv6 link-local traffic

A catch-all dest_ip ::/0 rule also matches link-local destinations. Router-originated link-local packets, most visibly odhcpd's DHCPv6 replies to LAN clients, then get a WAN fwmark and are re-routed into that WAN's routing table. That table has no link-local route for LAN devices, so sending fails with ENETUNREACH: odhcpd[1234]: Failed to send to fe80::...%lan@br-lan.10 (Network unreachable). This breaks DHCPv6 on all LANs whenever an IPv6 WAN member is online.

Link-local traffic can never leave its link, so policy-routing it is never correct. Accept it before classification in mwan3_output and mwan3_prerouting. Covered are exactly the scopes that are single-link by definition (RFC 4291): link-local unicast (fe80::/10) and interface-/link-scope multicast (ff01::/16, ff02::/16). Wider multicast scopes remain subject to policy routing.

The existing ICMPv6 ND bypass stays: NS/NA can also target global unicast addresses, which a link-local destination match does not cover. Its stale comment is rewritten.

18.5 Version 3.6.8

Summary: Fix an operator precedence regression in 3.6.7 in the address family guard.

mwan3: fix operator precedence in the address-family guard

The address-family guard in mwan3_set_user_nft_rule()combined its tests as [ ] && [ ] || [ ] && [ ], which POSIX shell evaluates as ((A && B) || C) && D rather than the intended (A && B) || (C && D), so the guard no longer skipped a mismatched-family pass. Group each && pair to restore the intended behaviour.

18.6 Version 3.6.7

Summary:

Comma-separated address lists are now accepted in src_ip and dest_ip rule options, with each element individually validated and the list converted to nft anonymous set syntax at rule-build time.

IP address validation has been overhauled. A new mwan3ipcheck binary replaces the fragile regex-based validation, using the inet_pton() function for address parsing that correctly rejects invalid input the old regex accepted silently. Of a list of 168 test cases, including both valid and invalid addresses, the regex expressions incorrectly passed 53 that were syntactically incorrect, meaning that mwan3ipcheck is a substantial reliability enhancement.

The mwan3 status command now parses nft list set output via nft -j and jshn.sh rather than a hand-built regex, fixing a bug where IPv6 prefix lengths were silently dropped and range elements were split into unrelated bare addresses.

Interface tracking log output has been substantially cleaned up. A new verbose_logging globals option activates the existing debug-level LOG() calls, which were previously permanently suppressed; the option takes effect on HUP-triggered reloads without a full restart. Several messages that fired at notice level during normal operation have been demoted to debug level, including per-host probe results during recovery, the "lost host(s)" recovery message, the "skip disconnected event" notice, and the check failure message produced when an interface is brought down manually before the hotplug USR1 signal arrives.

The lost host count reported in tracking log messages has been corrected; it previously multiplied the host failure count by pings-per-host and reported the result as "lost ping(s)" rather than "lost host(s)".

Two tracking robustness fixes have been applied. When a signal (HUP, USR1, or USR2) arrives mid-round, the partially-completed ping loop now aborts immediately rather than continuing to remaining targets and scoring the killed pings as genuine failures. During service stop, mwan3track instances are now terminated via procd_kill before the routing infrastructure is torn down, eliminating spurious 100% loss messages that appeared on every restart. A syntax error in the nping tracking method has also been fixed: when nping produces no output the result variable was empty, causing a bare [ -eq 0 ] test error; it now defaults to 1.

In luci-app-mwan3, the verbose_logging option is exposed as a checkbox on the Globals page. The rule editor's source and destination IP fields now accept comma-separated address lists with per-element validation and family consistency checking, matching the support added in mwan3. IPv6 validation across all files has been switched to validation.parseIPv6(), replacing heuristic detection. The configuration checker's rule shadowing detection now recognises when two rules reference the same nftset. A LuCI framework bug where the datatype validator corrupts CIDR values passed to custom validate functions has been worked around in the ipset entry validator.

mwan3: suppress spurious check failure log on manual ifdown

When an interface is manually brought down, the tracking pings fail immediately but the USR1 signal from hotplug has not yet arrived. This produces a misleading check failure log message moments before the interface is marked offline anyway.

The hotplug script writes the interface state to the hotplug state file before sending USR1, so mwan3track can read it to detect that an ifdown is already in progress. Skip the failure log when the hotplug state is already offline, since the message is redundant and misleading in that context.

mwan3: demote "lost host(s)" recovery message to debug

This message fires every tracking round where score has not fully recovered and at least one host failed, even though the reliability threshold was met. During recovery this adds per-round noise alongside the connecting and connected notices that already communicate the state transition.

Demote to debug since it is only useful for detailed recovery diagnostics.

mwan3: fix misleading "lost ping(s)" count in tracking log message

The lost variable counts how many track hosts failed to respond, not how many individual pings were lost. Multiplying by count (pings per host) and reporting the result as "lost ping(s)" overstated the actual loss. For example, with count=5 and 1 of 3 hosts unreachable, it reported "Lost 5 ping(s)" when only 1 host failed.

Report the host count directly as "Lost N host(s)" instead.

mwan3: demote "skip disconnected event" message to debug

This notice-level message logged when disconnected() was called on an interface that was already offline or connecting. It sat at the same syslog priority as genuine state transitions like "is offline" and "is online", adding noise that obscured real status changes.

Demote to debug since it is only useful for diagnosing redundant state transitions.

mwan3: demote per-host recovery success messages to debug

During recovery (score <= up), mwan3track logged an info message for every track IP that responded successfully, every tracking round. With multiple track IPs and a default up threshold of 5, this produced up to 15 info messages per recovery episode.

The connecting and connected notices already communicate recovery progress at the operational level. Per-host probe results during recovery are diagnostic detail, now available via the verbose_logging option.

mwan3: add verbose_logging option to enable debug log output

The LOG() function has historically suppressed all debug-level messages unconditionally, making the existing debug log calls across common.sh, mwan3.sh, and mwan3track permanently dead code.

Add a globals UCI option verbose_logging (default 0) that controls whether debug-level messages are emitted to syslog. When enabled, all existing LOG debug calls become active, providing detailed diagnostic output for troubleshooting interface tracking and nft rule setup.

The option is loaded in mwan3_init() for all scripts, and also in mwan3track's load_tracking_config() so that changes take effect on HUP-triggered config reloads without requiring a full restart.

mwan3: fix test syntax error in nping tracking method

The nping tracking method extracts the lost-packet count from nping output via grep and awk. If nping produces no output or the output does not contain a "Lost" line, the result variable is empty. The subsequent arithmetic comparison expands to [ -eq 0 ], which is a test syntax error in POSIX sh.

Default result to 1 (failure) when the extraction produces an empty string, so a missing "Lost" line is correctly treated as packet loss.

mwan3: terminate trackers before infrastructure teardown on stop

During service stop, stop_service() tears down ip rules, routing tables, and nft chains before procd sends TERM to the mwan3track instances. The trackers continue pinging during this window but their packets can no longer be routed (the fwmark-based ip rules are gone), causing every tracking target to report 100% loss. This produces spurious failure log messages on every service restart.

Stop all mwan3track instances at the start of stop_service via procd_kill, which tells procd to terminate them and mark them as intentionally stopped.

mwan3: skip tracking round when signal interrupts ping loop

When a signal (HUP, USR1, or USR2) arrives while mwan3track is mid-way through a ping round, the signal handler kills the current ping subprocess and sets an event flag. The killed ping returns non-zero, and the for loop continues to remaining tracking targets, all of which also fail. The event flags are not checked until after the for loop completes, so the entire round of killed-ping failures is scored and logged as genuine tracking failures.

Add an event flag check inside the per-IP for loop that breaks immediately when any signal flag is set. After the for loop, if any flag is set, reset the round counters and continue to the top of the while loop where event handling already runs, skipping the scoring, logging, and sleep for the aborted round.

mwan3: support comma-separated addresses in src_ip and dest_ip rule options

The mwan3ipcheck binary returns "mixed" for comma-separated address lists containing both IPv4 and IPv6 elements. Add "mixed" to the validation rejection check alongside "invalid", since a single nft anonymous set cannot contain both address families.

When a comma is present in src_ip or dest_ip, convert the value to nft anonymous set syntax by wrapping in { } with normalized spacing. Single addresses (no comma) pass through unchanged, preserving identical nft output to prior behaviour.

CIDR elements within comma lists are supported natively by nft anonymous sets and require no special handling.

mwan3: replace IPv4/IPv6 regex parsing with nft JSON in mwan3 status

The IPv6 regex had a bug where the CIDR suffix (/[0-9]+)? only bound to the final alternation branch because the 12-branch regex was joined with bare | and no outer grouping. This caused all IPv6 prefix lengths to be silently dropped from mwan3 status output. Range elements (produced by nft auto-merge on interval sets) were also mishandled, split into two unrelated bare addresses with no indication they form a range.

Replace the regex-based text parsing of nft list set output with structured JSON parsing using nft -j and jshn.sh. A new helper _mwan3_report_connected_set() handles both v4 and v6 sets, extracting prefix elements as addr/len and range elements as start-end.

Delete the IPv4_REGEX and IPv6_REGEX variables which are now unused.

mwan3: replace regex IP validation with mwan3ipcheck

Replace the grep-based IPv4/IPv6 regex validation in the rule-building path with calls to the mwan3ipcheck binary, which uses the inet_pton() function.

The previous validation used a 12-line IPv6 extended regular expression and a separate IPv4 regex via grep, which matched substrings rather than whole strings. This caused it to accept invalid input such as addresses with trailing garbage, leading zeros in octets, extra octets, and out-of-range values.

Additionally, the validation only detected family mismatches (an IPv6 address in an IPv4 rule pass, or vice versa). An invalid IP address that matched neither regex would silently pass through and produce a nft syntax error at batch commit time.

inet_pton() is the authoritative address parser used by nftables. It handles all valid representations by definition and rejects everything else, eliminating both the substring-matching problem and the missing validation for invalid addresses.

mwan3: add mwan3ipcheck address validation binary

Add a small C binary that validates and classifies IP addresses and CIDR notation using the POSIX inet_pton() function. Given a string argument, mwan3ipcheck prints "ipv4", "ipv6", or "invalid" to stdout and exits 0 on valid input or 1 on invalid input.

When a CIDR prefix is present (e.g. 192.168.1.0/24 or 2001:db8::/32), the prefix length is validated against the appropriate maximum for the detected family (32 for IPv4, 128 for IPv6). Leading zeros in the prefix length are rejected.

This binary replaces the fragile regex-based IP address validation currently used in the rule-building path. inet_pton() is the authoritative POSIX address parser and handles all valid IPv4 and IPv6 representations by definition, eliminating the need for a hand-built 12-branch IPv6 extended regular expression.

The binary has no dependencies beyond libc (uses only inet_pton, strtol, and standard string functions).

luci-app-mwan3: fix column alignment across IP set member tables

Set fixed column widths (50%/25%/25%) on both header and data cells so all expanded set tables align consistently.

luci-app-mwan3: add verbose logging option to globals page

Add a checkbox for the verbose_logging UCI option to the Network > MultiWAN Manager > Globals page, positioned below the existing Logging checkbox. This exposes the mwan3 debug-level logging toggle.

luci-app-mwan3: add comma-separated address support and enhanced validation

Add comma-separated address support to the rule editor's source and destination IP fields, matching the support already present in mwan3. Each address in a comma-separated list is individually validated for format and family consistency.

Replace IP family detection heuristics with validation.parseIPv6() across all files for robust IPv6 validation.

Improve validation in the simulator by tightening looksLikeFqdn() to reject malformed IPv4s instead of sending them to DNS resolution and validate addresses so that malformed ones are rejected before simulation.

Work around a LuCI framework bug where the datatype validator corrupts the value passed to custom validate functions for CIDR inputs. The ipset entry validator now performs its own format checking via a stub validator instead of relying on o.datatype.

Improve the configuration checker's rule shadowing detection to recognise when two rules reference the same nftset, rather than conservatively skipping all nftset comparisons.

18.7 Version 3.6.6

Summary: A race condition in the ct mark save logic has been fixed. The previous two-step clear and set sequence first cleared the MMX bits to zero and then set the new value in a consecutive rule, but between those two rules, under the right conditions, a packet on another CPU core could read the intermediate zero state, causing connections to be re-evaluated by the rules chain instead of being pinned to their assigned WAN. The clear is now folded into each setter chain as a single atomic expression, eliminating the race window and ensuring clear-set atomicity.

Adds a startup guard: mwan3 exits cleanly during startup if all interfaces are disabled, avoiding unnecessary nft chain loading and daemon startup. The default configuration now disables all interfaces and rules. Previously, a fresh install with no explicit user configuration would break the IPv6 internet because the balanced policy had no enabled IPv6 members, causing all IPv6 traffic to hit fall through to the unreachable last resort. A fresh install is now inert until explicitly configured by the user.

mwan3: skip startup when no interfaces are enabled

Add a guard in start_service after mwan3_init loads the configuration: if no interface section has option enabled '1', return without loading the nft framework or starting any daemons.

mwan3: disable all interfaces and rules in default config

The default configuration ships with the wan interface enabled and three rules (https, default_rule_v4, default_rule_v6) active. On a fresh install with no user customisation, this breaks IPv6 internet: the balanced policy has no enabled IPv6 members, so all IPv6 traffic that reaches the policy chain hits the unreachable last resort.

IPv4 works by coincidence (wan is the only enabled member and routes traffic normally), but the tracking pings and nft chain overhead serve no purpose on a single-WAN setup.

Set all interfaces and all rules to enabled=0 so a fresh install is completely inert until the user explicitly configures mwan3.

mwan3: fix ct mark save race by folding clear into setter chains

The nftables ct mark save operation used a two-step sequence: clear the MMX bits in ct mark to zero, then vmap-dispatch into a setter chain that ORed the new value in. Between those two rules, a packet on another CPU could observe ct mark with zeroed MMX bits, causing the restore step to miss the vmap lookup and allowing the connection to be reclassified to a random WAN.

Fold the clear into each save setter chain so that ct mark goes directly from old-value to new-value in a single nft expression (ct mark set ct mark & COMPLEMENT | VALUE). This eliminates the intermediate zero state entirely, providing the same single-write atomicity that iptables CONNMARK --save-mark had.

The standalone clear rules in the prerouting and output chains are removed since the setter chains now handle the masking internally.

18.8 Version 3.6.5

Summary: Replaces all ip command output parsing with direct ucode-mod-rtnl netlink calls via four new helper scripts in /lib/mwan3: mwan3-create-iface-route.uc, mwan3-get-addr.uc, mwan3-list-routes.uc, mwan3-manage-rules.uc. These scripts replace the shell invocations of the ip binary and consequent (fragile) parsing of the ip output using sed, awk and grep, helping to make mwan3 more robust and future proofing it against potential changes in the output format of ip commands, since the helper scripts return precisely the information needed by mwan3 and do not require any parsing of the output. The scripts also contribute to a substantially enhanced efficiency profile, since there is now only one call per operation instead of multiple ip subprocesses each invoking ip, sed, grep and awk.

Adds mwan3ct, a small C-language replacement for the conntrack user-space CLI tool that takes over all conntrack flush operations previously handled by the conntrack CLI tool. mwan3ct makes possible several highly targeted flush operations currently not possible with the conntrack CLI tool without extensive (fragile) parsing of the output and multiple invocations of conntrack -D.

mwan3ct flushes UNREPLIED conntrack entries when an interface comes online to prevent stale marks from pinning traffic to the wrong WAN. This was previously not possible and is now enabled via mwan3ct. It's particularly helpful in the scenario of a load-balanced connection that has keepalives that could pin a connection to a failed WAN because the keepalives maintain the conntrack entry current.

Applies minor hardening by moving nft batch files from /tmp to the root-owned /var/run/mwan3 directory and applies more restrictive umasks to the directory and file creation.

Fixes a scoping issue / buggy behaviour in nft that returns ipsets from other tables even though the query is scoped to a specific table, leading to a slow load time on luci-app-mwan3 Ipset and Rule tabs if the other tables contain large set-based blocklists (eg., adblock packages, banIP).

mwan3: fix nftset_info fetching all inet family sets

nft 1.1.6 does not scope nft list sets <family> <table> to the named table - it returns every set in the family, including sets from unrelated tables. On systems with firewall packages that maintain large element sets (e.g. IP blocklists), nft -j list sets inet mwan3 would return several megabytes of JSON containing elements from those foreign sets, making the rpcd nftset_info call take much longer than it should.

Replace nft -j list sets inet mwan3 with nft -j list table inet mwan3, which is correctly scoped to the mwan3 table and returns only its content.

Since nft list table includes set element data inline, the separate count_nftset_elements() subprocess call per set is also eliminated: replace it with a direct length() of the elem array already present in the parsed JSON.

mwan3: remove conntrack runtime dependency

mwan3ct replaced all conntrack CLI invocations in mwan3.sh. The +conntrack package dependency is no longer needed.

mwan3: improve readability of shell source files

Added blank lines around comment blocks in mwan3.sh and common.sh to improve readability. Whitespace-only change, no functional modifications.

mwan3: replace ip output parsing with ucode-mod-rtnl helpers

Replace all ip command invocations that parse output with calls to the ucode helper scripts added in the previous commit. This eliminates fragile sed/awk text parsing and reduces subprocess spawning.

Shell changes:

mwan3_get_src_ip: replace $IP address ls | sed with mwan3-get-addr.uc
mwan3_set_general_rules: replace $IP rule list | awk with mwan3-manage-rules.uc add-general
mwan3_delete_iface_rules: replace $IP rule list | awk with mwan3-manage-rules.uc delete-iface
mwan3_report_iface_status: replace $IP rule/route checks with mwan3-manage-rules.uc check and check-route
mwan3_set_custom_set, mwan3_set_connected_ipv4/v6: replace $IP route | awk | grep pipelines with mwan3-list-routes.uc
mwan3_create_iface_route: replace $IP route list | sed | while read with mwan3-create-iface-route.uc

Define MWAN3_GET_ADDR, MWAN3_LIST_ROUTES, MWAN3_MANAGE_RULES, and MWAN3_CREATE_IFACE_ROUTE variables in common.sh for consistent path references.

Remove dead code no longer reachable after these changes: mwan3_update_dev_to_table, mwan3_route_line_dev, mwan3_get_routes, mwan3_extra_tables_routes, and the MWAN3_ROUTE_LINE_EXP construction.

rpcd plugin: replace popen ip -j rule/route calls with direct rtnl.request() netlink queries, and fix field name mappings for rtnl (oif not dev, null not "default", family as integer).

Makefile: add INSTALL_BIN lines for the four .uc scripts.

mwan3: add ucode helper scripts for rtnl netlink access

Add four ucode scripts that use ucode-mod-rtnl for direct netlink communication instead of spawning ip subprocesses and parsing their text output through sed, awk, and grep pipelines. The existing approach is both fragile (sensitive to output format changes) and inefficient (each ip invocation forks a subprocess, and the subsequent text processing requires further subprocesses for each filter stage).

mwan3-get-addr.uc: query interface addresses via RTM_GETADDR
mwan3-manage-rules.uc: check, add, and delete policy rules via RTM_GETRULE/RTM_NEWRULE/RTM_DELRULE
mwan3-list-routes.uc: list CIDR routes from a routing table via RTM_GETROUTE, used for nft set population
mwan3-create-iface-route.uc: replicate routes from the main table into per-interface tables via RTM_GETROUTE/RTM_NEWROUTE

Each script produces exactly the output needed with no text parsing required. In several cases a single ucode invocation replaces what previously required multiple ip subprocess calls: delete-iface handles both address families in one call, create-iface-route performs route dump, filtering, deduplication, and table writes in a single process, and the rule check mode tests all three rule priorities from a single netlink dump.

mwan3: move nft batch files from /tmp to /var/run/mwan3

Move nft batch file paths out of the world-writable /tmp directory into the existing /var/run/mwan3 runtime directory, which is root-owned and created by mwan3_init at service start.

Restrict the runtime directory to mode 0700 and create batch files with mode 0600 via umask in common.sh and an explicit mode argument in mwan3rtmon. Nothing outside of mwan3's own root-running code accesses this directory.

This is a minor hardening change that eliminates a theoretical symlink/race window in /tmp for the nft batch files used by common.sh and mwan3rtmon.

mwan3: flush unreplied conntrack entries on interface online

When an interface transitions to online, flush conntrack entries that carry an mwan3 mark but have never received a reply. These entries can result from a race during startup or policy rebuild where a flow gets classified before the nft chains reflect the new state. The stale mark pins traffic to the wrong WAN, and because no reply arrives (the traffic is misrouted or blackholed), the application keeps retrying, refreshing the conntrack timeout indefinitely.

Flushing only UNREPLIED entries is non-disruptive to established connections. The next packet from the application creates a fresh conntrack entry that gets classified under the current policy.

The flush is called after mwan3_set_policies_nft in both the connected action (mwan3track declares interface online) and the ifup action with status=online (interface comes up with initial_state online). It is not called during init - the existing mwan3_flush_stale_conntrack handles the startup window.

mwan3: use mwan3ct for all conntrack flush operations

Replace all conntrack CLI invocations in mwan3.sh with mwan3ct calls:

mwan3_flush_stale_conntrack(): conntrack -D --mark 0x0/MASK becomes mwan3ct flush --mark 0x0/MASK
mwan3_flush_marked_conntrack(): shell loop spawning up to 63 conntrack -D processes becomes a single mwan3ct flush --mark-any
mwan3_flush_conntrack() ifdown branch: conntrack -D --mark VAL/MASK becomes mwan3ct flush --mark VAL/MASK

Since mwan3ct is part of the mwan3 package, the command -v conntrack guards and missing-tool warnings are no longer needed.

mwan3: add mwan3ct conntrack flush helper

Add a small C helper that uses libnetfilter_conntrack's NFCT_Q_FLUSH_FILTER to perform kernel-side filtered conntrack entry deletion by mark and/or status bits. This replaces all conntrack CLI usage in mwan3 with a purpose-built tool that is part of the package.

Motivations:

Fragility: the previous approach for targeted unreplied-entry deletion required parsing conntrack -L output in shell, extracting fields with parameter expansion, and issuing per-entry conntrack -D calls. mwan3ct uses the library API directly with no text parsing.
Efficiency: the "flush all marked entries" operation previously spawned up to 63 separate conntrack processes in a shell loop. mwan3ct handles this in a single process with one netlink socket.
Correctness: the conntrack CLI ignores the -u status filter on delete operations, making it impossible to selectively delete only UNREPLIED entries. mwan3ct passes status filters to the kernel correctly via NFCT_Q_FLUSH_FILTER.

The tool supports --mark <val>/<mask> for exact match, --mark-any <mask> for any non-zero mark within the mask, --status <val>/<mask> for status bit filtering, and combinations of these.

The Makefile gains build dependencies on libnetfilter_conntrack and libmnl, a runtime dependency on libnetfilter-conntrack, and removes the incorrect PKGARCH:=all (the package already compiled architecture-specific code via sockopt_wrap.c).

mwan3: fix diagnostic sanitizer treating 0.0.0.0 as a public address

The is_public_v4() function in mwan3-diag only excluded loopback, RFC1918, and link-local addresses. The "this network" block (0.0.0.0/8, RFC 1122) and multicast/reserved ranges (224.0.0.0/4 and above) were not excluded, causing them to be replaced with placeholder addresses in the diagnostic output rather than shown verbatim.

Add exclusions for 0.0.0.0/8 and 224.0.0.0/3 so that these non-routable addresses are preserved in diagnostic output alongside the other non-routable ranges already excluded.

18.9 Version 3.6.4

Summary: Fixes a regression in 3.6.3 that prevents mwan3's hotplug handler from running when the service is started but not explicitly enabled

mwan3: remove enabled check from hotplug handler

The enabled guard added in b363e9b50 causes the hotplug handler to exit silently when the rc.d symlink is absent. This breaks the startup path: start_service spawns per-interface hotplug handlers via mwan3_ifup using env -i, and those handlers exit at the enabled check without creating per-interface nft chains, ip rules, or routing tables. All WAN-bound traffic then hits the unreachable last-resort rule and connectivity is lost.

The guard was intended to prevent hotplug side effects when mwan3 is disabled, but it is redundant. The nft table existence check that follows already covers every practical scenario: the table is created by start_service and deleted by stop_service, so it is present only while the service is running. A service that was started manually without being enabled should still process hotplug events.

18.10 Version 3.6.3

Summary: Correctness fixes for edge cases and incomplete feature implementations: fixes ipset and ICMP protocol translation issues when rules use family=any; rebuilds ip rules and routing tables on reload to handle interface reordering and family changes; completes track_gateway support that was missing from two tracking status checks; adds runtime configuration reload to mwan3track via a SIGHUP handler and fixes hotplug handler initialization issues and boot race conditions.

mwan3: fix ipset+family=any batch failure and sticky chain orphan

When a rule has family=any and references a UCI ipset whose declared family is ipv6, both the ipv4 and ipv6 passes run. The ipv4 pass finds the set absent from the kernel and pre-creates it as ipv4_addr. The ipv6 pass then generates an "ip6 daddr @setname" expression against an ipv4_addr set, causing the atomic nft batch to fail and killing all user rules.

Fix by adding a UCI lookup before the absent-set guard. The new helper _mwan3_uci_ipset_addrtype() maps a set name to its UCI-declared address type. When a referenced set is absent from the kernel but present in UCI, the incompatible pass is silently skipped and the compatible pass proceeds without pre-creation, since mwan3_render_config_ipsets will add the set to the same batch. The existing external-set guard (skip ipv6 pass, let ipv4 pre-create as ipv4_addr) is retained for sets absent from both kernel and UCI.

Separately, the preamble in mwan3_set_user_rules() that pre-creates and flushes per-rule sticky chains sourced chain names by listing mwan3_rule_* chains already present in the kernel table. After a UCI rule section is deleted, mwan3_nft_reload_start removes the chain in the same batch; however, because the batch has not yet been committed when the kernel is enumerated, the chain is still visible. The old code therefore flushes and reinserts the deleted chain into the batch, causing it to survive the reload as an orphan. Change the preamble to iterate over enabled UCI rule sections instead, so only chains for active rules are created and chains for deleted rules are not recreated.

mwan3: fix ICMP protocol translation for family=any rules

UCI proto=icmp is family-agnostic, meaning ICMP appropriate to the address family of the rule. For family=any rules, mwan3 generates nft rules on two passes (ipv4 and ipv6). The ipv4 pass should emit "meta l4proto icmp" (protocol 1) and the ipv6 pass should emit "meta l4proto ipv6-icmp" (protocol 58).

Two interacting defects prevented the ipv6 translation from occurring:

The dedup optimisation that skips the ipv6 pass for family=any rules with no IP-version-specific elements incorrectly treated proto=icmp as family-agnostic. While TCP, UDP, and other L4 protocols share the same protocol number across families, ICMP (1) and ICMPv6 (58) are distinct protocols with distinct nft keywords. The dedup now excludes proto=icmp so both passes run.

The ICMP translation guard checked the UCI family config value instead of the current address-family pass variable. For family=any, the config value is "any", never "ipv6", so the translation never fired. The guard now checks the pass variable.

Without this fix, family=any rules with proto=icmp silently fail to match IPv6 ICMPv6 traffic.

mwan3: rebuild ip rules and routing tables on reload

The trigger conditions for this bug are narrow - all three must hold: mwan3 must be running with two or more interfaces, the user must reorder config interface sections in /etc/config/mwan3 without adding or removing interfaces, and the user must trigger a reload rather than a full restart.

When reload_service is called, it atomically rebuilds the entire nft ruleset using the current UCI configuration, but never rebuilds the kernel ip rules or per-interface routing tables. These kernel-side structures encode the interface table ID (tid), which is derived from the ordinal position of config interface sections in /etc/config/mwan3.

If the UCI section ordering changes between start and reload, the nft rules use the new tid-to-mark mapping while the ip rules and routing tables still reflect the old mapping. Traffic is silently routed to the wrong WAN interface.

A secondary manifestation of the same root cause: if an interface's family option changes (e.g. ipv4 to ipv6), the old ip rules remain in the wrong address family and new rules are never created.

Add mwan3_rebuild_iface_rules, called from reload_service via config_foreach, which deletes and recreates ip rules and routing tables for each enabled, up interface. Modify mwan3_delete_iface_rules and mwan3_delete_iface_route to search both address families unconditionally, so that a family change cleans up rules left behind in the old family.

mwan3: fix mwan3rtmon has_tracking ignoring track_gateway

The load_config function in mwan3rtmon determines whether an interface has tracking enabled by checking only track_ip. Interfaces configured with track_gateway but no track_ip are incorrectly treated as having tracking disabled. This causes get_track_status to return "disabled" without consulting the tracker's PID and STARTED state files, bypassing the secondary health check guard in handle_route_event.

The shell equivalent in common.sh (mwan3_get_mwan3track_status) and the init script (start_tracker) both correctly account for track_gateway. Align mwan3rtmon's has_tracking logic to match by including track_gateway in the check.

mwan3: reload tracker configuration on service reload

mwan3track reads UCI tracking parameters (interval, reliability, count, timeout, quality thresholds, etc.) once at startup and has no mechanism to update them at runtime. When reload_service rebuilds the nft ruleset, it does not restart tracker instances unless the tracker count changes, so any parameter changes made via UCI are silently ignored until a full service restart.

Add a load_tracking_config() function that extracts the parameter reads from main() into a reusable function called both at startup and on reload. Add a HUP signal handler to mwan3track that sets a reload flag and interrupts the current sleep/probe cycle. The main loop checks the flag at the same two points where IFDOWN and IFUP events are processed, re-reads UCI config, refreshes tracking parameters and track IPs, and clamps the score to the new down+up ceiling if it exceeds it.

In reload_service, send HUP to each running tracker instance after the existing rtmon HUP signals.

mwan3: fix mwan3_get_mwan3track_status ignoring track_gateway

The mwan3_get_mwan3track_status function in common.sh returns "disabled" when no track_ip list entries are configured, without checking the track_gateway option. This causes the "mwan3 interfaces" CLI command to report "tracking is disabled" for interfaces that use gateway-only tracking, even though a tracker instance is actively running and monitoring the interface.

The rpcd ucode equivalent (get_mwan3track_status) already checks both track_ip and track_gateway before returning "disabled". Align the shell function with the same logic by also reading track_gateway before the early return.

mwan3: fix hotplug handler to exit silently when mwan3 is not running

Add /etc/init.d/mwan3 enabled guard before procd_lock and mwan3_init so that hotplug events are silently ignored when mwan3 has no rc.d symlink.

Move the nft table existence check to before mwan3_init. Previously mwan3_init was called unconditionally, which created /var/run/mwan3 as a side effect and corrupted the stopped-service indicator in the case where mwan3 is enabled but has been manually stopped. With the check before mwan3_init, the handler exits without side effects whenever the nft table is absent, covering both the boot-race window and the stopped-but-enabled case.

Remove the running check that followed mwan3_init. It was dead code: mwan3_init created /var/run/mwan3 before the check ran, so service_running() always returned true and the exit path was unreachable.

18.11 Version 3.6.2

Summary: A set of defensive edge-case fixes and correctness improvements. Binds the mwan3rtmon route listener before the initial netlink dumps to close a narrow startup race window, and replaces main_route_cache with an on-demand kernel query to eliminate a class of cache-drift failures that could only manifest if route events arrived during the dump phase. Aligns shell and mwan3rtmon custom-set filtering so both paths apply identical exclusions. Adds a dormant is_default_route guard to populate_connected_set as a forward-compatibility precaution. Clamps check_quality to 0 when the configured track method cannot produce quality samples, preventing an arithmetic error in the unusual case where check_quality is paired with a non-ping method. Fixes mwan3_track_clean which targeted incorrect paths and was a no-op, and tightens ip rule deletion at stop_service to use content-based matching rather than a priority-range regex, which matters only when rule bases are configured outside the default 1000-3999 band.

mwan3: exclude default routes from mwan3rtmon connected set population

populate_connected_set filters routes with is_cidr_route and is_linklocal_route but does not check is_default_route. Currently dormant because rtnl returns dst=null for default routes, which is_cidr_route already rejects. Add an explicit is_default_route guard so the filter remains correct if the rtnl module ever emits "0.0.0.0/0" or "::/0" as a string instead of null.

mwan3: replace main_route_cache with on-demand kernel query in mwan3rtmon

main_route_cache replaced an on-demand kernel query (route_still_exists, originally from e119a57e9) with a route_key map built from the initial dump and maintained incrementally from listener events.

The cache made the per-event ECMP-suppress check O(1) but introduced a class of dump-vs-listener race failure modes: any event arriving between the snapshot used to build the cache and the listener becoming active can double-count or undercount the cache, leading to stuck suppressions or premature cleanups of per-interface routing table entries with no self-healing path. The listener-first swap in an earlier commit shifts but does not eliminate this race.

route_still_exists queries the kernel directly on every delete and is unaffected by cache drift. Cost is one RTM_GETROUTE dump per DELROUTE event whose per-iface table entry exists, single-digit milliseconds on typical routing-table sizes and infrequent in practice.

mwan3: align shell custom-set population with mwan3rtmon filtering

The shell mwan3_set_custom_set function seeds mwan3_custom_v4/v6 at start and reload so the sets are populated before mwan3rtmon starts. mwan3rtmon takes over incremental maintenance once procd schedules it.

Previously the shell and mwan3rtmon disagreed on filtering: the shell matched via IPv4_REGEX / IPv6_REGEX without excluding defaults or link-locals, while mwan3rtmon's repopulate_custom_sets excludes both via is_default_route and is_linklocal_route. Rewrite the shell function to apply the same exclusions (default, 0.0.0.0/0, ::/0, 169.254., fe80::) and require a CIDR slash in IPv4 matches, so both paths produce identical set contents.

Also extend mwan3rtmon's repopulate_custom_sets to flush the set when no rt_table_lookup entries are configured, so removal of all rt_table_lookup entries followed by a reload correctly empties the set.

mwan3: bind mwan3rtmon route listener before initial dumps

mwan3rtmon's main() previously ran the initial netlink dumps before creating the route listener. Any RTM_NEWROUTE or RTM_DELROUTE emitted between mwan3rtmon process start and the rtnl.listener() call was discarded by the kernel (no multicast subscriber existed), leaving per-iface tables and the mwan3_custom_v4/v6 sets diverged from kernel state with no self-healing path.

Bind the listener first so its multicast socket starts buffering route events immediately, then issue the dumps. ucode-mod-rtnl creates the netlink socket and joins the multicast group synchronously inside the rtnl.listener() call, so the socket buffer captures concurrent events during the dump phase even before uloop.run() begins draining them.

Also add an explicit repopulate_custom_sets() call at startup so mwan3rtmon reconciles the mwan3_custom_v4/v6 sets with kernel state once it begins running. The shell seeds these sets during start_service, but custom-table route changes between the shell pass and mwan3rtmon startup are unobserved by either side; the startup dump closes that window.

The connected set is unchanged; it already self-heals through a debounced rebuild triggered by every CIDR-route event.

mwan3: clamp check_quality for track methods that lack loss/latency

mwan3track's per-iteration case statement only populates $loss and $latency for the ping and httping methods. arping, nslookup, and nping-* set only $result. With check_quality=1 configured against one of those methods the post-case decision block reaches

`[ "$loss" -ge "$failure_loss" ] || [ "$latency" -ge ... ]`

with empty (or stale from a previous iteration) operands, which raises a busybox ash arithmetic error and leaves the per-host up/down classification undefined.

Clamp check_quality to 0 at startup when the configured track_method cannot produce quality samples, and emit a notice so the user can see that the option was overridden. Apply the same clamp in mwan3_load_track_ips so that LATENCY_/LOSS_ state files are still cleaned up after a config edit that transitions the interface from a quality-capable method to one that is not.

mwan3: clean correct per-interface state paths in `mwan3_track_clean`

mwan3_track_clean is called from stop_service via mwan3_interface_shutdown to tear down per-interface runtime state. The original implementation removed $MWAN3_STATUS_DIR/<iface> and attempted to rmdir $MWAN3_STATUS_DIR. Neither matched where the per-interface state actually lives:

tracker runtime state at $MWAN3TRACK_STATUS_DIR/<iface>/(PID, STATUS, LATENCY_*, LOSS_*, TRACK_*, GATEWAY, ...), written by mwan3track;
hotplug state file at $MWAN3_STATUS_DIR/iface_state/<iface>, written by mwan3_set_iface_hotplug_state.

The $MWAN3_STATUS_DIR/<iface> path the old code targeted does not exist, and the bare $MWAN3_STATUS_DIR rmdir always failed because that directory holds session-wide pinned state (mmx_mask and the iif_rule_base/fwmark_rule_base/unreachable_rule_base records used by mwan3_init across stop/start cycles), so the function was a no-op despite its name.

Point the rm at the correct per-interface paths and replace the bogus rmdir with a best-effort rmdir of $MWAN3TRACK_STATUS_DIR itself, which is the dir that should be empty once every iface has been cleaned. $MWAN3_STATUS_DIR is left intact because it holds state that must survive a stop/start cycle.

Errors are redirected to /dev/null so transient mismatches do not hit the console or logs. A comment in the function documents a small residual race with mwan3track that can briefly recreate the tracker dir between rm and procd_kill; the leak is bounded to one iteration and harmless on tmpfs.

mwan3: identify ip rules to delete precisely at stop_service

stop_service identified mwan3-owned ip rules with a fixed regex match on ^[1-3][0-9]{3}:, limiting cleanup to priorities 1000-3999 and deleting every rule in that band regardless of origin. Two problems:

When iif_rule_base, fwmark_rule_base or unreachable_rule_base are configured outside 1000-3999 (all three are exposed as globals and validated in mwan3_init), the affected tier of rules is leaked across service mwan3 stop and accumulates across stop/start cycles.
Even within the band, a rule placed by another package or by a local admin override sitting at one of those priorities is indistinguishable from a mwan3 rule and gets deleted.

Replace the regex with a two-gate filter parsed in pure shell from ip rule list output. A rule is deleted only when it passes both:

Priority gate: priority falls in one of the three configured rule-base ranges. The fwmark range extends to fwmark_base + MM_UNREACHABLE so the global blackhole and unreachable rules at fwmark_base + MM_BLACKHOLE / MM_UNREACHABLE are included.
Content gate: the line references a mwan3-owned routing table (lookup <N> with N in 1..MWAN3_INTERFACE_MAX), or carries an fwmark masked by mwan3's MMX_MASK.

To make the gates trustworthy at stop time, mwan3_init now persists iif_rule_base, fwmark_rule_base and unreachable_rule_base to state via uci_toggle_state at start_service, mirroring the existing iface_max persistence, and reads them back via uci_get_state on subsequent invocations. The state values reflect the rule bases the running instance actually created its rules with, even if /etc/config/mwan3 has been edited since start. The mmx_mask state file doubles as the "instance started" indicator. The validation step that reverts to default bases on ordering-constraint violation runs only in the fresh-start branch, so persisted values are always post-validation.

18.12 Version 3.6.1

Summary: Extends the legacy mwan3 custom sets that were previously only loaded statically during start_service() and reload_service() from the tables defined in the UCI global config list option rt_table_lookup to be fully dynamic, using mwan3rtmon to listen for and to add and remove routes from the custom sets in response to RTM_NEWROUTE and RTM_DELROUTE events on the tables defined with list rt_table_lookup <tableid>. Adds a SIGHUP handler to mwan3rtmon to cause it to flush and repopulate these custom sets, ensuring that their contents remain in sync with any newly added or removed list rt_table_lookup <tableid> options in the mwan3 config.

mwan3: reload mwan3rtmon config on mwan3 reload via SIGHUP

mwan3rtmon loads its UCI configuration once at startup and holds it in memory for the lifetime of the process. This includes extra_table_set, the in-memory set of routing table IDs derived from the rt_table_lookup UCI option. When mwan3 reloads without a full restart (the common path when interface count does not change), mwan3rtmon stays running with its original extra_table_set intact. If the operator changes rt_table_lookup during that reload, mwan3rtmon will continue routing events against the old table list until it is manually restarted.

Fix this by adding a SIGHUP handler to mwan3rtmon. On receipt of SIGHUP it calls load_config() to refresh all in-memory UCI state and then repopulate_custom_sets() which performs a live netlink route dump of every table now in extra_table_set and rebuilds mwan3_custom_v4/v6 from scratch. The flush-then-add pattern ensures that tables removed from rt_table_lookup have their routes evicted from the custom sets as well as tables newly added having their current routes immediately mirrored in.

reload_service() in the init script sends SIGHUP to the rtmon_ipv4 (and rtmon_ipv6 if IPv6 is enabled) procd instances via procd_send_signal after mwan3_nft_reload_commit and the ip rule updates, so the nft sets and ip rules are fully consistent before mwan3rtmon re-dumps.

The shell-level mwan3_set_custom_sets() call already present in reload_service() is retained. It runs inside the atomic nft batch and provides a synchronous static snapshot that ensures the sets are never empty during the reload window. The SIGHUP-triggered re-dump that follows is the authoritative update because it runs after the batch commits and picks up any route changes that occurred in the interval between the shell dump and the signal delivery.

mwan3: handle `rt_table_lookup` route events dynamically

The rt_table_lookup feature populated the mwan3_custom_v4 and mwan3_custom_v6 nftables sets at startup by reading existing routes from the configured tables, but did not update those sets when routes changed at runtime. Route additions and deletions in rt_table_lookup tables were therefore not reflected until mwan3 was restarted.

Add handle_custom_set_event(), called from handle_route_event() whenever a route event arrives for a table listed in extra_table_set. RTM_NEWROUTE events add the destination to the appropriate custom set; RTM_DELROUTE events remove it. Default routes and link-local routes are excluded, matching the exclusions already applied at startup.

18.13 Version 3.6

Summary: Version 3.6 adds three user-visible features to mwan3 rules. Rules now support an fwmark/fwmask option to match packets by meta mark using a masked comparison, working alongside or instead of address and ipset matching; mwan3 logs a warning if the fwmask overlaps its internal MMX_MASK since such a mask would match packets already carrying an mwan3 classification mark. The ip rule priority tiers for per-interface rules are now configurable via three new globals UCI options (iif_rule_base, fwmark_rule_base, unreachable_rule_base), shifting from the fixed 1000/2000/3000 defaults; two ordering constraints are enforced at startup and rule deletion is rewritten to use content-based matching so it remains correct across base or mmx_mask changes. Rules gain option enabled 0/1, consistent with interfaces, ipsets, and members.

The LuCI interface is updated throughout to reflect all three additions. The rule modal gains an Fwmark field with MMX_MASK overlap validation; the Globals tab gains the three base priority fields with live cross-field ordering validation and an automatic mwan3 restart when any base changes. The Routing Health tab gains the unreachable rule row per interface and marks an interface as degraded if the unreachable rule is absent; base priorities are now displayed dynamically from the rpcd endpoint rather than assumed from fixed offsets. The Traffic Simulator, rule shadowing analysis, Status Overview, and diagnostics helper are all updated to handle fwmark matching, disabled rules, and configurable bases. Mutual enable protection is added to prevent the silent misconfiguration of an enabled rule referencing a disabled IP set. mwan3track now validates that libwrap_mwan3_sockopt.so is present at startup, exiting with a clear error rather than silently producing incorrect tracking results if the library is missing.

mwan3: add per-interface unreachable rule to rpcd routing_health

The routing_health() rpcd endpoint reported iif and fwmark rule presence per interface but omitted the unreachable rule. This meant the Routing Health tab could not detect a missing unreachable rule, and would show a fully healthy interface even if its unreachable safety net was absent.

Add unreach_rule (present + priority) to each interface in the response, and include the unreachable base priority in the rule_bases object returned to the frontend.

mwan3: update rpcd `routing_health` for configurable bases and dynamic mmx_mask

The routing_health() function in the rpcd ucode module used hardcoded constants for the ip rule base priorities (1000/2000) and the maximum interface count (63). This meant the Routing Health tab reported incorrect data when configurable rule base priorities were in use, and would also be wrong for any installation with a non-default mmx_mask.

Read iif_rule_base, fwmark_rule_base, and unreachable_rule_base from UCI globals with the same defaults and ordering constraint validation as mwan3_init in common.sh. Derive mmdefault and iface_max dynamically from the configured mmx_mask by counting set bits, replacing the hardcoded MAX_IFACES constant. This ensures the blackhole and unreachable global policy rule priorities, stale rule range detection, and ordering constraint validation all reflect the actual runtime configuration.

Add the unreachable rule tier to valid priority tracking and stale rule detection. Previously only the iif and fwmark tiers were checked, so per-interface unreachable rules were invisible to the health report and could be falsely flagged as stale.

Return the active rule base priorities in the rpcd response so the frontend can display them dynamically rather than assuming fixed offsets.

mwan3: add configurable ip rule base priorities and fwmark rule matching

fwmark/fwmask rule matching is added to mwan3_set_user_nft_rule(). A policy rule may now specify an fwmark and fwmask to match against the packet's meta mark using a masked comparison. The match is address-family agnostic, operating on meta mark rather than IP-layer fields. fwmark and fwmask must be specified together; a rule with only one set is skipped with a warning. A warning is also logged if the fwmask overlaps mwan3's internal MMX_MASK, since such a mask would match packets already carrying an mwan3 classification mark.

Three new globals UCI options: iif_rule_base, fwmark_rule_base, and unreachable_rule_base allow the ip rule priority tiers at which mwan3 installs its per-interface rules to be shifted from the original fixed offsets of 1000/2000/3000. Defaults are unchanged, so existing installations are unaffected and other packages that rely on mwan3's rules being inserted at those priorities remain unaffected insofar as the defaults are not overridden.

Two ordering constraints are enforced at startup:

iif_rule_base + MWAN3_INTERFACE_MAX < fwmark_rule_base fwmark_rule_base + MWAN3_INTERFACE_MAX + 1 < unreachable_rule_base

If either constraint is violated, all three values are reverted to defaults with a logged warning.

The ip rule deletion logic in mwan3_delete_iface_rules() is rewritten to use content-based matching rather than a numeric range filter. Previously rules were deleted by checking whether their priority modulo 1000 equalled the interface id and fell in the range 1001-3999. This breaks when configurable bases place rules outside that range or at priorities that collide modulo 1000. The new logic identifies the iif rule by the iif keyword and table id, then discovers the fwmark/mask value from the lookup rule and deletes both the fwmark lookup and unreachable rules by content. This also handles upgrades from pre-configurable-base versions and MMX_MASK changes transparently.

For rules with family=any that have no IP-layer match criteria (no src_ip, dest_ip, or pset), the nft expression is address-family agnostic and would be emitted identically on both the ipv4 and ipv6 generation passes. The ipv6 pass is now skipped in that case to avoid installing a duplicate chain rule.

mwan3: add missing `ucode-mod-socket` dependency

mwan3-diag imports the ucode socket module to normalise IPv4 and IPv6 address strings via sock.sockaddr(). The ucode-mod-socket package was not listed in DEPENDS, so on a fresh install the module would be absent and mwan3-diag would abort at startup with an import error.

Add +ucode-mod-socket to the package DEPENDS.

mwan3: wire `validate_wrap()` startup check for `libwrap_mwan3_sockopt`

If libwrap_mwan3_sockopt.so is absent, mwan3track now exits with a clear error at startup rather than silently producing incorrect tracking results. Also define the library path as a single WRAP_LIB constant, and fix the file test from -x to -f.

mwan3: add option enabled support for rules

UCI rules lacked the option enabled 0/1 guard that interfaces, ipsets, and members already support. Add config_get_bool checks in two places:

mwan3_set_user_nft_rule(: skip disabled rules at install time, consistent with the default of 1 (enabled) so existing configs without the option are unaffected.
iface_rule() inside mwan3_set_user_iface_rules(): skip disabled rules when scanning for src_iface matches so a disabled rule with a matching src_iface does not trigger a needless mwan3_set_user_rules rebuild on ifup hotplug events.

luci-app-mwan3: fix hardcoded rule base priorities in diagnostics helper

The luci-mwan3 helper script used hardcoded iif (1000) and fwmark (2000) base priorities when checking ip rules for an interface. With configurable rule base priorities, the diagnostic "Check IP rules" would grep for the wrong priorities and report rules as missing when they are present at different priorities.

Read iif_rule_base, fwmark_rule_base, and unreachable_rule_base from UCI globals with the same defaults as the backend. Add the unreachable rule check that was missing entirely. Improve the output format to show each rule type individually with its expected priority and a summary count.

luci-app-mwan3: filter disabled rules from Status Overview

The Overview tab displayed all configured rules regardless of enabled state. Disabled rules have no runtime effect and their presence in the overview misrepresents the active ruleset.

Skip rules with enabled=0 so the overview reflects what mwan3 actually evaluates at runtime.

luci-app-mwan3: display per-interface unreachable rule in Routing Health

The Routing Health tab showed only the iif and fwmark ip rules per interface. The unreachable rule, which prevents packets marked for a down interface from being silently misrouted via the main routing table, was not visible.

Add an unreachable rule row to the per-interface health card using the unreach_rule data now provided by the rpcd endpoint. Include the unreachable rule in the health calculation so a missing unreachable rule is flagged as degraded. Update the field guide to describe all three rule tiers and display the unreachable base priority dynamically.

luci-app-mwan3: add fwmark rule support and configurable rule base priorities

Adds an Fwmark field to the rule edit modal. The field takes a combined combined value/mask hex expression (e.g. 0x80000/0xff0000) and splits it across the fwmark and fwmask UCI options on write. Validation checks hex format and rejects masks that overlap mwan3's internal MMX_MASK bits. The Destination column in the rule list is extended to append a mark:value/mask token when fwmark/fwmask are set, so fwmark-only rules do not appear as wildcard entries.

Traffic Simulator tab is extended to match fwmark rules. An Fwmark input field is added to the simulator form; its value is compared against each rule's fwmark/fwmask pair using a masked comparison, with an empty or absent mark treated as zero. matchSummary is updated to include the fwmark expression in the match description, and a pre-existing bug is fixed where family=any was displayed as IPv6 rather than being omitted.

Status Overview tab is extended to display the fwmark/fwmask expression in the Match column of the status page rules grid. Previously, rules that match solely on packet mark displayed as '(all traffic)'.

Globals tab gains three new form fields: iif_rule_base, fwmark_rule_base, and unreachable_rule_base, exposing the ip rule base priority options added to mwan3. Each field carries live cross-field validation enforcing the two ordering constraints required by the backend:

iif_rule_base + MWAN3_INTERFACE_MAX < fwmark_rule_base fwmark_rule_base + MWAN3_INTERFACE_MAX + 1 < unreachable_rule_base

The tooltip for the fwmark and unreachable fields shows the minimum required offset above the preceding base, computed at render time from the current firewall mask setting so the displayed value reflects the actual interface capacity.

Changing any of the three base values requires a full mwan3 restart to delete and recreate all ip rules at the new priorities. handleSaveApply is overridden to detect base changes by comparing pre- and post-save UCI values and, if a change is found, registers a one-shot uci-applied event listener that calls mwan3 restart after the config is committed to disk. Pages that change only unrelated globals settings (logging, loglevel) do not trigger a restart.

Routing Health field guide displays the actual configured rule base priorities dynamically rather than hardcoded 1000/2000 labels, using the rule_bases object returned by the rpcd routing_health endpoint.

Configuration tab rule shadowing analysis is extended to account for fwmark/fwmask matching, and disabled rules (enabled=0) are filtered out before the analysis since mwan3 skips them at runtime.

luci-app-mwan3: simulator: skip disabled rules during matching

The introduction of the enabled flag for rules created a gap in the traffic path simulator: disabled rules were still evaluated and could appear as matching results, giving incorrect output.

Filter disabled rules from both the nftset collection pass and the rule matching loop so that the simulator reflects the active ruleset as mwan3 sees it at runtime.

luci-app-mwan3: ipset/rule: add mutual enable protection

An enabled rule referencing a disabled IP set silently misfires at runtime: mwan3 auto-creates the set as empty and the rule never matches any traffic. Three guards are added to prevent this misconfiguration.

On the IP sets tab, the Enable checkbox is rendered disabled (greyed out) when the set is currently enabled and referenced by at least one enabled rule. This prevents the user from disabling a set that is in active use.

On the rules tab, the Enable checkbox is rendered disabled when the rule is currently disabled and references at least one disabled IP set. This prevents the user from re-enabling a rule whose set dependency is not yet satisfied. The guard applies only when the rule is currently disabled; if the rule is already enabled the checkbox remains free so the user can uncheck it to correct the state.

On the rules tab, the Source NFT set and Destination NFT set dropdowns exclude any set that is explicitly disabled in the mwan3 UCI config. This prevents a disabled set from being assigned to an already-enabled rule via the modal form.

Both tabs also show an explanatory note in the section header describing when the checkbox will be greyed.

The implementation uses this.readonly set temporarily before calling the parent form.Flag renderWidget, which passes disabled: true to ui.Checkbox and sets the HTML disabled attribute on the input element. The enabled state is read via uci.get() rather than the cfgvalue argument, which is unreliable for options stored at their default value.

luci-app-mwan3: use `addr:port` format in Overview rule listing

The Status-->Overview rule grid was displaying source and destination match fields as separate tokens (src:, src ipset:, sport:, dst:, dst ipset:, dport:), inconsistent with the Network-->Rules grid which combines address/ipset and port into a single addr:port expression.

Add a fmtAddr() helper mirroring the textvalue logic used in rule.js: address (or ipset if no address) is joined with port as addr:port, or *:port when only a port is set. The overview now shows proto, src: and dst: in the same compact format as the Rules grid.

luci-app-mwan3: add Enable column to rules grid

Add an inline Enable checkbox to the rules GridSection. The checkbox uses o.editable = true so it is interactive directly in the grid row without requiring the modal to be opened.

To accommodate the new column without widening the grid, the separate Source port and Destination port columns are merged into the Source and Destination columns respectively. The merged textvalue format is ddress:port or ipset_name:port when both are set, address or ipset_name when only an address is present, *:port when only a port is set, and - when neither is set. src_port and dest_port are marked modalonly so they remain editable via the modal.

The Policy assigned column label is shortened to Policy.

18.14 Version 3.5.3

Summary: Version 3.5.3 adds two major LuCI features and a set of bug fixes and routing reliability improvements.

The Policy tab is rewritten with a tier-based policy builder that lets users specify failover tiers and per-interface load-balancing weights without manually creating member definitions. The builder handles member creation and garbage-collects orphaned members automatically, and updates the policy list to display IPv4 and IPv6 policies in symbolic form. The traffic simulator gains hostname resolution: Source and Destination fields now accept hostnames as well as IP addresses, resolved via the new resolve_host rpcd method against the local DNS server. The new Policy tab metaphor and layout should greatly improve how both new and existing users get to grips with mwan3, whose flexible but less than intuitive interface-member-policy definitions sometimes pose a substantial barrier to immediate use.

Three new rpcd methods - nftset_flush, nftset_reload, and nftset_resolve - allow the LuCI IP Sets tab to flush a set, repopulate it from UCI static entries and a loadfile, or trigger fresh DNS resolution by sending SIGHUP to dnsmasq before resolving configured domains. Matching Flush, Reload, and Resolve buttons appear on each set panel alongside the existing Expand button. get_nftset_members is fixed to unwrap the counter-decorated element wrapper that nft emits for counter-enabled sets, which previously caused all members of such sets to be silently dropped, breaking simulator rule matching for counter-enabled nftsets.

mwan3_create_iface_route is changed to use ip route replace instead of ip route add, making route installation unconditionally idempotent and eliminating spurious EEXIST errors when mwan3rtmon or a racing event had already inserted the route. A missing default argument in the config_get_bool call for the enabled option is added, suppressing cosmetic sh: out of range noise on startup.

mwan3: add nftset_flush, nftset_reload, and nftset_resolve RPC methods

nftset_flush empties a named nft set via nft flush set.

nftset_reload flushes the set then repopulates it from the UCI ipset config: static entries from the entry list and any entries read from the loadfile path.

nftset_resolve sends SIGHUP to dnsmasq to clear its cache, then iterates the domain list for the ipset and calls nslookup_resolve for each, triggering fresh DNS lookups that cause dnsmasq to populate the set via its nftset integration.

mwan3: fix missing default in config_get_bool enabled call

mwan3_update_dev_to_table iterates all mwan3 interface sections to build the device-to-routing-table map. The config_get_bool call for the enabled option had no default argument. When an interface section has no explicit option enabled set, get_bool receives an empty value and an empty default, falls through to the wildcard case, and returns an empty string. The subsequent [ "$enabled" -eq 0 ] comparison then receives an empty string as its left operand, causing busybox ash to emit sh: out of range (empty string has no digits consumed, triggering the error path in the test builtin's integer conversion).`

The error is cosmetic. The failed test exits non-zero, so the && return does not fire and the interface is treated as enabled, which is correct, but the error messages are misleading and concern users.

Fix by adding the missing default of 1, consistent with all other config_get_bool enabled calls in the file.

mwan3: use ip route replace in mwan3_create_iface_route

The previous ip route add would fail with EEXIST if the route was already present in the per-interface table -- for example when mwan3rtmon had already copied it from the main table, or when a connected event raced an ifup event. The string-match dedup check intended to prevent this was fragile: ip route list output for the same route can differ in text representation between table main and a per-interface table (field ordering, explicit metric 0, etc.), causing the check to miss a match and fall through to a duplicate add.

ip route replace uses NLM_F_CREATE | NLM_F_REPLACE at the kernel level, making the operation unconditionally idempotent. This is the same semantics mwan3rtmon already uses for its route copies. The duplicate-detection pre-check is retained as a cheap optimisation to avoid a redundant syscall when the route is already known to be present, but it is no longer safety-critical.

mwan3: add resolve_host RPC method for hostname resolution

Add nslookup_resolve() helper and resolve_host RPC method to allow the LuCI traffic simulator to accept hostnames in addition to IP addresses. The resolver invokes /bin/busybox nslookup against the local DNS server (127.0.0.1), populating any dnsmasq nftset entries configured for the queried domain as a side effect. Returns separate v4 and v6 address arrays; the family parameter restricts resolution to A or AAAA records. Input is validated against [a-zA-Z0-9._-]+ before use.

mwan3: fix get_nftset_members to unwrap counter-decorated elements

nft wraps set elements in {"elem":{"val":"...","counter":{...}}} objects when the set has the counter flag. The previous code passed these wrapper objects directly to parse_elem_val(), which returned null, silently dropping all members of counter-enabled sets. This caused the traffic simulator to fail to match rules that reference counter-enabled nftsets.

get_nftset_elements() already handled this case correctly; apply the same unwrap logic to get_nftset_members().

luci-app-mwan3: add Flush, Reload, and Resolve buttons to IP Sets tab

Each set panel gains Flush, Reload, and Resolve action buttons alongside the existing Expand button. Flush empties the nft set. Reload flushes and repopulates from UCI static entries and the loadfile. Resolve is shown only for sets with configured domains; it sends SIGHUP to dnsmasq to clear its cache and triggers fresh DNS lookups to repopulate the set via dnsmasq's nftset integration.

All three buttons disable during the async operation and refresh the element count on completion. A page-level description explains each button's behaviour.

ACL grants for nftset_flush, nftset_reload, and nftset_resolve are added to both read-only and read-write rpcd permission groups.

luci-app-mwan3: reimplement the Policy tab

mwan3's use of members, metrics and weights is counter-intuitive for new users and even a barrier to effective use for some.

Create a new policy builder modal that allows policies to be specified by choosing interfaces and percentages to create a load balanced policy and policy tiers to allow for failover configurations.

The policy builder automatically handles member definitions, meaning that users never have to be concerned with creating members and with calculating appropriate metrics and weights. The policy builder does it all according to the configured tiers (metrics) and weights.

Update the policy list to remove the member column and instead show the IPv4 and IPv6 policies in a symbolic form that makes the configuration immediately obvious to the user.

Policy builder will automatically garbage collect orphaned members; a "Delete unused member definitions" checkbox in the policy tab header controls whether orphaned members are removed on save or not.

The Member tab is kept visible for manual inspection and editing of metric/weight values, although with the advent of policy builder, it is entirely redundant. Preserved more as a comfort to users who've been using mwan3 for a long time than for any really functional need.

luci-app-mwan3: add hostname resolution to traffic simulator

Allow the Source IP/Name and Destination IP/Name fields in the traffic simulator to accept hostnames in addition to IP addresses.

When a hostname is entered, the simulator calls the resolve_host RPC method to resolve it via the local DNS server before running the simulation. The resolved address is displayed inline as Resolved: x.x.x.x (+N more). Resolution errors abort the simulation with an error message.

The address family selector controls A vs AAAA record resolution; IPv4 is preferred when the selector is set to both families. Update the ACL to permit the resolve_host method, update field labels and placeholder text to indicate hostname support, and widen the input fields.

18.15 Version 3.5.2

Summary: Version 3.5.2 is a bug-fix and maintenance release. It corrects a misrouting bug where kernel-generated NDP Neighbor Solicitation probes entered mwan3_output without a conntrack entry, fell through to mwan3_rules, and received a WAN policy mark that caused the kernel to probe the gateway via the wrong interface, cycling the NDP entry to FAILED state and breaking WRAP ping tracking for that interface. It updates the package dependency from ip to ip-full to ensure the full iproute2 implementation is always present, since the busybox ip is a minimal subset that does not support all options mwan3 requires. It adds mwan3-diag, a ucode diagnostic script installed to /usr/sbin/mwan3-diag that collects a comprehensive snapshot of mwan3 state -- interface status, policy routing rules, nftables ruleset, routing tables, conntrack summary and system log -- with all public IP addresses anonymised with stable placeholders so output can be shared safely.

mwan3: add mwan3-diag network diagnostic script

mwan3-diag is a ucode script that collects a comprehensive snapshot of mwan3 state. It gathers interface status, policy routing rules, nftables ruleset, routing tables, conntrack summary, system log and anonymises all public IP addresses with stable placeholders so output can be shared safely.

mwan3: depend on ip-full instead of ip

The busybox ip implementation is a minimal subset of iproute2 and does not support all options and subcommands that mwan3 requires for correct operation. Depend on ip-full to ensure the full iproute2 implementation is always present.

mwan3: bypass NDP in mwan3_output to prevent re-routing of NDP probes

Kernel-generated NDP Neighbor Solicitation probes start with mark=0 and enter mwan3_output. They have no conntrack entry, so the ct mark restore is a no-op. They are not matched by mwan3_connected, mwan3_custom, or mwan3_dynamic. They fall through to mwan3_rules, where the default IPv6 rule (ip6 daddr ::/0) applies a WAN policy mark -- the same mark that would be assigned to outbound user traffic. Because mwan3_output is type route, the mark change triggers a routing re-evaluation, which may route the probe to a different WAN interface than the one whose gateway the kernel is trying to resolve. The gateway NDP entry cycles to FAILED state, and subsequent WRAP ping probes are dropped because the kernel cannot resolve the gateway MAC address.

Add an icmpv6 NDP accept rule at the top of mwan3_output, mirroring the equivalent rule already present in mwan3_prerouting.

18.16 Version 3.5.1

Summary: Version 3.5.1 is a bug-fix and maintenance release. It corrects a silent failure in mwan3rtmon where route replication to per-interface routing tables was completely non-functional, adds nft set flag-change detection on reload so that changing a set's timeout, counter, or size options takes effect immediately without requiring a full service restart, suppresses spurious stderr noise from ip rule and ip route operations during upgrades and teardown, and removes version number references from comments.

mwan3: fix mwan3rtmon route replication broken by stale table name

refresh_active_chains() filtered for chains in table inet fw4 instead of table inet mwan3. Because mwan3's interface chains live in table inet mwan3, the active_chains cache was always empty. Every caller that depended on it -- is_iface_nft_active, get_active_tids, populate_iface_routes, and the route-replication path in handle_route_event -- silently did nothing. Route replication from the main routing table to per-interface routing tables was completely non-functional.

Most deployments did not notice because mwan3_create_iface_route in the hotplug script populates per-interface tables at ifup time, covering the static routing table case. The bug manifests when routes are added to or removed from the main table after mwan3 starts (VPN tunnels, PPPoE reconnection, etc.).

The connected set population path (populate_connected_set) was unaffected by the bug and continues to work correctly.

mwan3: detect nft set flag changes on reload and delete+recreate as needed

nft add set is idempotent on existence: if a set already exists it returns without error but does not update its flags (timeout, counters, size). A reload that changed any of these flags silently left the live set with the old configuration until the next full service restart.

Add _mwan3_nft_time_to_sec to parse nft time unit strings (1h, 5m, 300s) into a common integer-seconds representation for comparison. Add _mwan3_ipset_needs_delete which queries the live set via nft list set and returns true if the live flags differ from the desired spec.

_mwan3_render_one_ipset now calls _mwan3_ipset_needs_delete before the add set statement. If flags differ the set is deleted first, clearing its elements but ensuring the recreated set has the correct type, timeout, counter, and size flags.

When a set is deleted and recreated its dnsmasq-populated domain entries are lost. Set MWAN3_NEED_DNSMASQ_HUP when this occurs and call mwan3_dnsmasq_hup after the reload batch in reload_service to repopulate those entries.

Move the ipset and dnsmasq fragment functions from common.sh to mwan3.sh. They are only called from mwan3.sh or init.d/mwan3, and the new helpers (_mwan3_nft_time_to_sec, _mwan3_ipset_needs_delete) naturally belong with them.

mwan3: remove version number references from comments

Comments referencing specific version numbers become misleading as the codebase evolves. Replace all such references with descriptions of the actual state or behaviour they document.

mwan3: suppress stderr on unguarded ip rule/route operations

Four locations in mwan3.sh produced noise on stderr during package upgrades and edge-case teardown:

mwan3_create_iface_rules: ip rule add calls had no error suppression. mwan3_delete_iface_rules runs first, so a "File exists" error means the rule is already in the desired state - a valid outcome that should not be reported as an error.
mwan3_delete_iface_route: ip route flush on a never-populated table produces "FIB table does not exist". This is a normal teardown scenario when an interface was never brought online.
mwan3_extra_tables_routes: ip route list on a missing rt_table_lookup table produces the same error. Suppressed here without a warning since this is called per-interface per-connect; mwan3_set_custom_set provides the warning at a more appropriate point.
mwan3_set_custom_set: ip route list calls restructured to capture output and check exit code separately, so a missing rt_table_lookup table is suppressed on stderr but logged via LOG warn. This preserves the error as a diagnosable signal without printing raw kernel errors to the console.

18.17 Version 3.5

Summary: Version 3.5 is a major architectural release that moves mwan3 out of table inet fw4 and into its own table inet mwan3, eliminating the fw4 rebuild scaffold and the mwan3evtd debounce daemon entirely.

The reload path is replaced with a single atomic nft batch that commits the complete new ruleset while the old one is still serving traffic, with zero window of misrouted connections. User-declared nft sets are now configured directly in /etc/config/mwan3 with inline, file, and dnsmasq-populated modes, and per-element packet and byte counters are optionally available. The APK install lifecycle is hardened to eliminate RTNETLINK errors on both fresh install and upgrade. LuCI gains a full IP Sets configuration tab and a new IP Sets status view with paginated element display, and the overview layout is redesigned with CSS grid cards. Additional bug fix to debounce RTM_NEWROUTE calls in mwan3rtmon.

mwan3: update mwan3rtmon to debounce RTM_NEWROUTE calls

RTM_NEWROUTE events for connected routes were fast-pathed directly to nft_exec, bypassing the debounce timer that was only applied to deletes. On IPv6 systems with prefix delegation, the kernel appears to send repeated RTM_NEWROUTE updates for already-present connected routes, causing nft processes to be spawned multiple times per second and producing measurable CPU load.

Fix: route both RTM_NEWROUTE and RTM_DELROUTE for CIDR routes through the same 100ms debounce timer, calling populate_connected_set() once after the burst settles rather than once per event. Add a fingerprint (sorted, joined element list) to opulate_connected_set() so that calls where the connected set content has not changed skip the nft_batch call entirely. Also add ECMP deduplication (seen map) and link-local filtering to the element build loop in populate_connected_set().

mwan3: add conntrack as a hard dependency

mwan3 relies on the conntrack userspace tool in several places: flush_conntrack is called when interfaces go down or policies change to force existing connections to be re-evaluated under the new routing state. Without conntrack installed these operations silently fail, leaving stale connections pinned to a dead or reconfigured WAN interface. Making the dependency explicit ensures conntrack is always present when mwan3 is installed.

mwan3: add counters support, nftset_elements RPC, and orphaned set cleanup

Add option counters (bool, default 0) to config ipset sections. When set, enables per-element packet and byte count tracking via nft set counter statement.

Change maxelem default from 65536 to 0 (unlimited), matching fw4 behaviour where sets have no size limit unless explicitly configured.

Add mwan3_cleanup_orphaned_ipsets, called during reload_service after mwan3_render_config_ipsets. Queries nft for user-defined sets not prefixed mwan3_, compares against configured set names, and deletes any orphans. Prevents stale sets accumulating when a set is removed via LuCI and the config is applied.

rpcd ucode: refactor get_nftset_members to use a shared parse_elem_val helper; add get_nftset_elements which returns elements with optional per-element counter data (packets/bytes) and supports pagination via a max parameter; add count_nftset_elements for lightweight element counting; enhance nftset_info to include flags, counters, and count fields; add nftset_elements RPC method with a 5000-element hard cap.

mwan3: atomic non-destructive reload via single nft batch

Replaces the stop/start reload_service with an atomic single nft -f batch that rebuilds the entire ruleset while the old one serves traffic. The batch commits in one kernel transaction with zero window of misrouted traffic.

No conntrack flush in reload: existing connections keep their ct marks and current routing; new connections use new rules immediately.

Race condition immunity:

Reload: the entire rebuild is a single nft -f batch. The kernel commits the complete new ruleset atomically or rolls back to the old one. There is no intermediate state where prerouting exists but iface_in chains are absent. Adding or removing an interface requires a full restart because procd service instances for new trackers can only be registered during start_service. The tracker mismatch check at the end of reload_service detects the discrepancy and falls through to stop/start automatically.

Startup: the wait $hotplug_pids barrier in start_service ensures all background ifup jobs complete before mwan3_set_general_nft populates prerouting. Prerouting's vmap dispatch never references a chain that does not yet exist.

mwan3: update mwan3-lb-test for standalone table inet mwan3

mwan3-lb-test creates a temporary test set and inserts rules into mwan3_output, mwan3_rules, and fw4's forward chain. Moving mwan3 to its own table inet mwan3 requires two changes.

Rename TABLE from "inet fw4" to "inet mwan3" so that mwan3_output, mwan3_rules, and the test set creation all target the correct table.

Introduce FORWARD_TABLE="inet fw4" for the forward isolation rule. nft sets are table-scoped: the forward chain lives in table inet fw4, so the test set and the drop rule that references it must both exist in fw4. All forward chain operations (rule insertion, handle lookup, rule deletion, set creation/deletion, and stale-set cleanup) are updated to use FORWARD_TABLE.

mwan3: fix APK preinst and postinst for table inet mwan3

Fix the APK install lifecycle to ensure clean operation on both fresh install and upgrade from an installation that used table inet fw4.

Add a preinst script that stops mwan3 before APK replaces any files so that procd's inotify trigger does not auto-restart it during package installation. A safety-net stop at the start of postinst covers the case where preinst did not run or procd restarted the service between preinst and postinst.

Make the fw4 reload in postinst conditional on the firewall.mwan3_reload UCI section existing. On a fresh install that section is absent, so an unconditional reload triggers queued ifup hotplug events that cause a race with mwan3_create_iface_rules, producing RTNETLINK "File exists" errors. On upgrade the section exists, so the delete succeeds, and fw4 is reloaded exactly as before.

Add a second mwan3 stop immediately before the final mwan3 start in postinst. By this point the procd auto-start has completed and its ip rules are present. The second stop calls stop_service and removes them before start_service re-adds them, eliminating the remaining source of RTNETLINK "File exists" errors.

mwan3: remove mwan3evtd debounce daemon

mwan3evtd was introduced to debounce rapid-fire mwan3 rebuild events and deliver a single safe dnsmasq SIGHUP after each fw4 reload. Since mwan3 now lives in its own table inet mwan3 which fw4 reload does not touch, there are no fw4-triggered rebuilds, no dnsmasq SIGHUPs from mwan3, and no event storms to debounce. mwan3evtd is redundant.

Remove the daemon, its init script, config file, helper binary, example files, and ACL. Remove the ucode-mod-log dependency (used only by mwan3evtd). Remove /etc/config/mwan3evtd from conffiles. Remove the postinst enable/start and postrm stop/disable calls.

The postrm dnsmasq cleanup (removing mwan3-nftsets.conf fragments and restarting dnsmasq) is retained since mwan3 still writes those fragments for domain-based config ipset population; removal of the package should clean them up so dnsmasq stops attempting to populate sets that no longer exist.

mwan3: fix port-range rendering for nftables

UCI stores port ranges as x:y (e.g. '47813:47814') but nftables requires x-y (e.g. '47813-47814'). The colon is map-element syntax in nft; a set element like { 47813:47814 } is rejected with "mapping outside of map context", causing the entire mwan3_rules batch to fail atomically and leaving the chain empty.

Fix: add s/:/-/g to the sed transform in mwan3_set_user_nft_rule for both src_port and dest_port. Old configs with colon separators and new configs with dash separators both work correctly after this change.

mwan3: auto-flush mwan3-marked conntrack on service reload

Add mwan3_flush_marked_conntrack() in mwan3.sh and invoke it from a new reload_service override in init.d/mwan3. Flushes every conntrack entry whose mark has any MMX_MASK bit set, so UCI-driven reloads (including the uci-commit-trigger path via procd_add_reload_trigger) cause live flows to re-enter the classification chains and re-evaluate against the new rules instead of staying pinned to a previously saved ct mark.

Complements the existing mwan3_flush_stale_conntrack, which handles the distinct zero-mark case (flow slipped through unclassified during the fw4-rebuild window). The two cover orthogonal cleanup needs.

conntrack's -D --mark VALUE/MASK filter does exact-match on the masked bits; there is no "any bit set" predicate. The helper iterates the mwan3 id-space (default 6 bits => 63 ids) and issues one targeted -D per id. Bounded and fast.

Not called from start_service: service mwan3 restart should leave live TCP flows intact; only a config-driven reload reclassifies them. The mwan3_init at the top of reload_service ensures MMX_MASK is populated before the flush, since stop wipes the persisted status dir.

Fixes the stale-ct-mark class of issues where a rule change does not take effect on a live flow, leaving it pinned to the old interface even after the user adds a mwan3 rule that should reclassify it.

mwan3: add table inet mwan3 nft set support

Add support for user-declared nft sets via a new config ipset section type in /etc/config/mwan3 that matches fw4 syntax and replaces sets declared in /etc/config/firewall and which live within table inet fw4's namespace, which is not in scope in table inet mwan3.

Three population modes are supported: inline entries via list entry, file-based population via option loadfile, and dnsmasq-populated sets via list domain (mwan3 writes confdir fragments and signals dnsmasq only when fragment content changes).

Add mwan3-migrate-ipset-v4.sh, a one-shot idempotent migration helper that copies existing config ipset declarations from /etc/config/firewall to /etc/config/mwan3 on upgrade from v3.x. Called from postinst. Makefile gains the corresponding install line and a postrm cleanup block that removes dnsmasq confdir fragments and reloads dnsmasq on package removal.

Update rpcd nftset_info: nftset membership lookup was filtering on s.table == 'fw4' but sets now live in table inet mwan3, so the filter is updated to s.table == 'mwan3'.

mwan3: move to standalone table inet mwan3, remove fw4 rebuild scaffold

Move mwan3 from table inet fw4 to its own table inet mwan3.

Rename every inet fw4 literal to inet mwan3 across common.sh, mwan3.sh, init.d/mwan3, hotplug.d/iface/25-mwan3, usr/sbin/mwan3, mwan3rtmon, and usr/share/rpcd/ucode/mwan3. The Makefile postinst retains its inet fw4 references: those are legacy cleanup of v3.x-era chains that remain correct.

Add files/lib/mwan3/mwan3-skeleton.nft: a standalone nftables ruleset that creates (or atomically re-creates) table inet mwan3 with its base chains and sets using the delete+recreate idiom for idempotency. Wire the skeleton load into init.d/mwan3 start_service with an early return on failure so subsequent nft add calls cannot paper over a missing table.

Delete mwan3-fw-include.sh and mwan3-fw-rebuild.sh. fw4 reload no longer touches table inet mwan3, so there is nothing to detect or rebuild. The fw4-include UCI registration script mwan3-firewall-include is replaced with mwan3-remove-firewall-include, a one-shot uci-defaults script that removes the stale firewall.mwan3_reload section left by v3.x installs. Remove the fw4 reload detection block from 25-mwan3.

stop_service gains a final nft delete table inet mwan3 so a clean stop leaves no nft residue.

Add postinst cleanup of legacy mwan3 chains, sets, and sticky maps from table inet fw4 so upgrading from a v3.x install removes the old hooked chains that would otherwise remain after fw4 reload (fw4 uses flush-table not delete-table, so they survive indefinitely).

Simplify mwan3_dnsmasq_hup in mwan3.sh to a direct ubus call. The mwan3evtd dispatch path is no longer needed as mwan3 now lives in its own table and fw4 reload does not trigger rebuilds or SIGHUP storms. Call mwan3_dnsmasq_hup from start_service after all sets are created so dnsmasq clears its cache and re-populates the new nft sets.luci-app-mwan3: improve overview status layout

Switch the interface and policy card grids from flexbox to CSS grid with fixed 13em columns using repeat(auto-fill). Cards wrap gracefully as the browser is resized rather than forcing all items onto one row regardless of available width.

Replace the LuCI %t format with a custom formatDuration() that omits seconds, keeping the uptime display stable in width and removing the visual noise of a ticking seconds counter.

Add Interfaces and Policies section headings above each grid, matching the existing Rules heading, which also cleanly separates the two grids visually so their differing column counts do not appear misaligned.

Remove the indent from policy member entries and add white-space:nowrap to prevent wrapping within a member line on narrow columns.

luci-app-mwan3: add IP Sets status tab

New status view at Status > MultiWAN Manager > IP Sets (order 27, between Routing and Diagnostics).

Page load calls nftset_info to get all user-defined mwan3 set metadata (type, counters flag, runtime element count) and loads mwan3 UCI config for each set's static parameters (entries, domains, loadfile, maxelem, timeout); element counts are shown in the panel header without requiring any user interaction
Each set is shown as a collapsible panel with an Expand/Collapse toggle; on first expand, configured domain names (from list domain) are shown immediately from UCI, then the runtime member table is loaded via the new nftset_elements RPC; subsequent collapse/expand cycles reuse the already-loaded DOM without re-fetching
Members are displayed in a consistent 3-column table (Address / Packets / Bytes) regardless of whether counters are enabled; Packets and Bytes cells are empty for sets without counters
Large sets: elements load up to 200 by default; Load more (1000) and Load all (5000) buttons appear when the result is truncated
ACL: add nftset_info and nftset_elements to both luci-app-mwan3-status and luci-app-mwan3 ACL sections

luci-app-mwan3: improve IP Sets network configuration tab

Block deletion of a set that is referenced by mwan3 rules; show a warning notification naming the referencing rules so the user knows what to fix first
Suppress premature validation red on the Name field when a new set modal opens with an empty name; the field is marked pristine until the user interacts with it or clicks Save, at which point the red border appears if the field is still empty
Change maxelem description and placeholder from 65536 to "unlimited" to match fw4 behaviour (sets have no size limit unless explicitly configured)

luci-app-mwan3: add IP Sets configuration tab

Add a new IP Sets tab to the MultiWAN Manager network section (order 52, between Rule and Simulator). The tab provides a GridSection UI for managing config ipset sections in /etc/config/mwan3.

Fields: name (validated: safe chars, no mwan3_ prefix, unique across sections), family (IPv4/IPv6), entry (DynamicList, ipaddr datatype, family cross-check validation), domain (DynamicList, for dnsmasq nftset population), loadfile (FileUpload to /etc/luci-uploads), maxelem, timeout, counters, enabled.

IP address entry fields use blur-only validation matching rule.js behaviour, implemented via makeBlurOnlyList which attaches a capture-phase keyup suppressor to existing inputs at render time and to dynamically added inputs via MutationObserver.

ACL updated to add ubus file read/list (read) and file write/remove (write) permissions required by form.FileUpload.

luci-app-mwan3: update nftset references to table inet mwan3

nftset_dump in luci-mwan3 now lists sets from inet mwan3 rather than inet fw4; the mwan3_ prefix filter correctly hides internal skeleton sets and surfaces only user-declared config ipset sections.

The ipset dropdown placeholder text in rule.js is updated to show the inet#mwan3 nftset directive syntax and explain that fw4-side sets need a parallel config ipset declaration in /etc/config/mwan3 to be usable in mwan3 rules.

luci-app-mwan3: update port-range hint to use dash separator

mwan3 now renders port ranges using x-y (nft native format); the colon separator x:y is still accepted in UCI for backwards compatibility but is no longer the recommended input format. Update the src_port and dest_port help text from "1024:2048" to "1024-2048" so new entries match nft syntax directly.

18.18 Version 3.4.1 (Unreleased)

Summary: Builds the per-interface mwan3_iface_in_* chains before mwan3_set_general_nft() activates mwan3_prerouting to avoid a race condition that leads to a wrong interface mark being assigned. Fixes bugs in the nft list chains syntax in stop_service() and a grep expression that was causing a too-broad match and resulting in traffic for interface wan bypassing mwan3 marking.

mwan3: fix nft list chains syntax error in stop_service

nft list chains only accepts an optional family argument, not a table name. The two chain-enumeration loops in stop_service used nft list chains inet fw4, which is invalid syntax. The fw4 argument caused nft to exit with an error; 2>/dev/null suppressed it, producing no output. Both loops therefore silently iterated over nothing on every service mwan3 stop or service mwan3 restart.

Consequence: dynamic chains (mwan3_iface_in_*, mwan3_policy_*, mwan3_or_meta_*, mwan3_or_ct_*) were never flushed or deleted on stop. They persisted in table inet fw4 as empty orphan objects. The final hardcoded skeleton-chain flush at the end of stop_service was unaffected and continued to work correctly, so functional impact was limited to residual chain objects after stop.

Fixed by changing both occurrences to nft list chains inet. The grep "chain mwan3_" filter is sufficient to scope results to mwan3's chains, which only exist in table inet fw4.

mwan3: fix grep substring match in mwan3_ifaces_in chain wiring

mwan3_create_iface_nft used grep -q "jump mwan3_iface_in_$1" as an idempotency check before adding a jump from mwan3_ifaces_in to the per-interface chain. grep -q does substring matching, so for $1=wan the pattern also matches lines containing mwan3_iface_in_wan2, mwan3_iface_in_wan6, etc. If those interfaces had already added their jumps, wan's check returned a false positive and its jump was skipped, leaving wan absent from mwan3_ifaces_in and all wan traffic bypassing mwan3 marking entirely.

The same bug in mwan3_delete_iface_nft caused the handle lookup for wan's jump to return the handle of a different interface's rule.

The bug existed before this patch but was masked by the old sequential ordering, where wan (first in UCI order) always checked mwan3_ifaces_in before any other process had added jumps. The preceding commit changed to parallel execution, exposing the race.

Fixed with grep -qw (whole-word match) in both functions. Only affects configurations where one interface name is a prefix of another (e.g. wan/wan2, eth0/eth0b).

mwan3: build iface_in chains before activating prerouting

The per-interface mwan3_iface_in_* chains are now fully populated before mwan3_set_general_nft() activates mwan3_prerouting. Previously the ordering was reversed: prerouting became live first, then each mwan3_ifup background process added rules to mwan3_ifaces_in one by one. During that window, packets arriving on any interface whose chain was not yet wired got no ct mark. For DNAT traffic (fib daddr type local return with no mark save) the reply could hit numgen and acquire a random interface mark, which WireGuard PersistentKeepalive then locked in permanently by refreshing the conntrack entry every 25s.

Changed in all three rebuild code paths:

init.d/mwan3 start_service: moved config_foreach mwan3_ifup and wait $hotplug_pids before mwan3_set_general_nft
25-mwan3 fw4-reload detection block: config_foreach mwan3_rebuild_iface_nft before mwan3_set_general_nft
mwan3-fw-rebuild.sh: same reorder as 25-mwan3

Also removed the flush of mwan3_postrouting from mwan3_set_general_nft. That flush was a leftover from when general_nft ran before iface setup; with the new ordering it ran after mwan3_create_iface_nft had already written snat6 rules to postrouting, silently deleting them. mwan3_delete_iface_nft and the per-chain cleanup in mwan3_create_iface_nft already handle postrouting cleanup correctly.

18.19 Version 3.4

Summary: Version 3.4 introduces mwan3evtd, a generalised ucode debounce daemon that coalesces rapid-fire events - such as simultaneous interface flaps triggering multiple fw4 reloads - into a single handler execution after the activity settles. This prevents the repeated dnsmasq SIGHUPs that previously caused cache thrash and, in tight-timing scenarios, dnsmasq crashes during concurrent startup.

mwan3: mwan3evtd debounce daemon

Add mwan3evtd, a generalised ucode debounce daemon that debounces rapid-fire events and executes a single safe handler for all identical events received during the debounce window.

Background: fw4 recreates the entire inet fw4 nftables table on every reload, which flushes mwan3's nftsets. dnsmasq must be SIGHUPed after each rebuild to repopulate those sets. Without debouncing, simultaneous interface events (e.g. N-interface flap) would fire N independent mwan3 rebuilds in quick succession, each attempting to SIGHUP dnsmasq.

mwan3evtd coalesces all pushes within a window into a single fire: window_ms (default 5 s) resets on each push, and the HUP is delivered only once after activity has settled or on max_window_ms.

The debounce daemon exists to avoid unnecessary cache flushes and to make dnsmasq HUPs safe, since in some cases, and with tight timing, a HUP on a starting instance was observed to crash dnsmasq, resulting in loss of connectivity for clients.

mwan3evtd exposes a ubus object mwan3evtd with push/list/status/flush/reload/reset methods. Callers push named events via:

ubus call mwan3evtd push '{"event":"dnsmasq-hup"}'

or via the fast-path helper /usr/sbin/mwan3evtd-push (falls back to direct UCI config parsing when ubus is unavailable, e.g. early boot).

Each handler is configured in /etc/config/mwan3evtd with:

  window_ms:          debounce window reset on each push
  max_window_ms:      hard cap, fires even under continuous pressure
  command:            shell command to run on fire
  handler_timeout_ms: kill handler if it runs longer than this

Default handlers: dnsmasq-hup, dnsmasq-restart, rpcd-reload, rpcd-restart, firewall-reload, firewall-restart, unbound-restart, smartdns-restart, kresd-restart, named-restart.

Shell injection in the handler fire path is prevented by passing the command through EVTD_CMD env var and using eval "$EVTD_CMD", avoiding any brace-grouping that an adversarial } in command could escape.

mwan3evtd is a generalised debounce daemon, capable of accepting any event type and any handler and can be used by other packages if desired.

18.20 Version 3.3.5

Summary: Version 3.3.5 is a single-fix release that suppresses the per-deleted-entry output that conntrack -D writes to stdout, which was previously appearing on the console whenever mwan3 start or an fw4 reload triggered the zero-mark conntrack flush.

mwan3: suppress conntrack -D output to console

conntrack -D prints each deleted entry to stdout. The call in mwan3_flush_stale_conntrack() redirected only stderr, causing all deleted zero-mark conntrack entries to appear on the console whenever mwan3 start or fw4 reload triggered the flush.

Fix: redirect stdout to /dev/null alongside stderr.

18.21 Version 3.3.4

Summary: Version 3.3.4 closes a class of misrouting bugs caused by the brief window between fw4 flushing table inet fw4 and mwan3 completing its nft rebuild. Connections established during that window acquire ct mark=0; the new mwan3_flush_stale_conntrack helper removes all zero-mark conntrack entries after every rebuild and restart, preventing WireGuard persistent-keepalive and similar long-lived UDP from locking in a bad entry indefinitely. A double-rebuild race in mwan3-fw-rebuild.sh is also fixed by acquiring the procd lock before checking for empty chains.

mwan3: flush zero-mark conntrack entries after fw4 rebuild and restart

During the brief window between fw4 wiping table inet fw4 and the nft rebuild completing, new connections (DNAT, WireGuard) can be established with ct mark=0. The iface_in chains do not yet exist so incoming packets are not marked; the conntrack entry is created with ct mark=0.

For most protocols the bad entry expires within 120 seconds and self-heals on the next connection attempt. Long-lived UDP with persistent keepalives (WireGuard persistent-keepalive) refreshes the entry before expiry, keeping it alive indefinitely and causing persistent misrouting of DNAT replies.

Add mwan3_flush_stale_conntrack() to mwan3.sh. Flushes conntrack entries with no mwan3 mark (0x0/MMX_MASK) using the conntrack tool. Only zero-mark entries are removed; correctly-marked active connections are untouched. Even for users without the persistent-keepalive problem, the flush causes stale connections to re-establish immediately rather than waiting up to 120 seconds for natural conntrack expiry. If conntrack-tools is not installed, logs a notice suggesting installation.

Called from three sites:

mwan3-fw-rebuild.sh: covers fw4 reload triggered by any means
25-mwan3 fw4 rebuild detection block: covers ifup-triggered fw4 reload
start_service in init.d/mwan3: covers service mwan3 restart, which runs 25-mwan3 with MWAN3_STARTUP=init bypassing both guarded blocks and does not invoke 26-mwan3-user where a mwan3.user workaround would otherwise run

mwan3: fix double rebuild race in mwan3-fw-rebuild.sh

mwan3-fw-rebuild.shchecked for emptymwan3_preroutingbefore acquiringprocd_lock. If 25-mwan3was already holding the lock and rebuilding, thefw-rebuildscript could pass the check, then queue behind25-mwan3, and proceed to rebuild and call mwan3_dnsmasq_hup` a second time after the lock was released.

Fix by acquiring procd_lock first and re-checking under the lock, so only one of the two rebuild paths does the actual work.

18.22 Version 3.3.3

Summary: Version 3.3.3 is a broad bug-fix release addressing several correctness issues: DNAT reply routing was broken by a misplaced fib daddr type local return rule that fired before DNAT translation, causing replies to exit via a randomly load-balanced interface; IPv6 ip rules were silently leaked on ifdown because delete_iface_rules queried the IPv4 rule table; mwan3_dnsmasq_hup never sent SIGHUP because json_get_var stores booleans as integers not strings; and the numgen counter was contaminated by inbound and reply traffic. Additional fixes cover a grep substring false-positive in iface chain wiring, unquoted regex variables, a dead function stub, a duplicate function, and missing mwan3_postrouting in the stop_service chain lists. A new bypass_network UCI option populates the dynamic bypass sets from config, and mwan3-lb-test gains fw4 reload detection.

mwan3: add mwan3_postrouting to stop_service skeleton chain lists

mwan3_postrouting is defined in the static 10-mwan3.nft skeleton alongside the other named skeleton chains, but was absent from both the deletion exclusion list and the safety-flush list in stop_service(). This caused it to be deleted on stop rather than preserved, relying on start_service/mwan3_ensure_nft_framework to recreate it. Add it to both lists for consistency with the other skeleton chains.

mwan3: remove duplicate mwan3_count_one_bits from mwan3.sh

An identical copy of mwan3_count_one_bits() existed in both mwan3.sh and common.sh. Since mwan3.sh sources common.sh, the mwan3.sh copy shadowed the canonical definition without any functional difference. Remove the duplicate; all call sites continue to use the common.sh definition.

mwan3: remove dead mwan3_set_sticky_nft function

mwan3_set_sticky_nft() was an incomplete stub from an earlier sticky routing design that was superseded by the current per-member ip-only set + OR-immediate vmap-dispatch implementation in mwan3_set_user_nft_rule(). The stub had a no-op loop body and was never called. Remove it.

mwan3_get_policy_members_for_family() is retained - it is called by the active sticky implementation at line 1131.

mwan3: quote $cmdline in mwan3_get_mwan3track_status

$cmdline was unquoted in the [ $cmdline != ... ] test. If readfile returns nothing (narrow race where the tracked process exits between the PID file read and the /proc/$pid/cmdline read), the empty expansion produces a malformed two-argument test expression. In busybox ash this happens to evaluate correctly (falls through to export -n "$1=down"), but the unquoted form is fragile. Quote it.

mwan3: quote $IPv4_REGEX in mwan3_set_user_nft_rule

$IPv4_REGEX was unquoted in the grep -qE call on line 951 while the adjacent $IPv6_REGEX on line 950 was correctly quoted. The IPv4 regex contains characters (?, [, ]) that the shell interprets as glob patterns before passing to grep if unquoted. Quote it for consistency and correctness.

mwan3: fix IPv6 ip rule leak in mwan3_delete_iface_rules

mwan3_delete_iface_rules() sets IP="$IP6" for IPv6 interfaces but used bare ip rule list (which defaults to IPv4) to find rule priorities to delete. No IPv6 rules were found, so the for loop never executed and all three ip rules (at priorities IIF_BASE+id, FWMARK_BASE+id, and 3000+id) were leaked on every IPv6 interface ifdown.

On the next ifup, mwan3_create_iface_rules calls mwan3_delete_iface_rules before adding new rules, but the delete fails for the same reason, so stale rules accumulate on each ifdown/ifup cycle.

Fix: change ip rule list to $IP rule list so the correct address family is used.

mwan3: fix dnsmasq_hup running check - json_get_var returns 1 not "true" for boolean true

json_get_var stores JSON boolean values as integers (1 for true, 0 for false), not as strings. The previous check [ "$running" = "true" ] never matched, so mwan3_dnsmasq_hup never actually sent SIGHUP to dnsmasq after fw4 reloads. DNS cache was therefore not cleared, leaving stale nftset entries in place.

mwan3: fix DNAT reply routing broken by misplaced fib-local return rule

The v3.3.1 numgen contamination fix added an unguarded fib daddr type local return as the first substantive rule in mwan3_prerouting. Because mwan3_prerouting runs at priority mangle+1 (-149), before the nat/prerouting DNAT hook (-100), the rule fires on the original packet of a DNAT connection while its destination is still the router's own WAN IP. The packet is returned immediately with no ct mark saved. When the DNAT reply arrives from the internal host on the LAN interface, the ct mark restore finds zero, the packet falls to the policy chain, numgen assigns a random WAN mark, and the reply exits via whichever interface numgen picks rather than the one the original packet arrived on.

Fix: remove the unguarded fib-local return from its position before the ifaces_in dispatch and reinsert it after, guarded by meta mark & MMX_MASK == 0. Traffic arriving on a mwan3 WAN interface is already marked by the iface_in catchall before reaching this rule, so the guard makes it a no-op for WAN traffic while still blocking non-WAN (LAN, loopback) local-destined traffic from reaching the policy chain. DNAT original packets are now marked by the iface_in catchall, ct mark is saved correctly, and DNAT replies restore it and route back via the correct interface.

Also remove ct direction reply return from mwan3_output. That rule was added to compensate for ct mark being zero (a consequence of the misplaced fib-local return). With the fib-local return correctly placed and guarded, ct mark is non-zero for inbound WAN connections, the ct mark restore in mwan3_output works correctly, the policy chain guard (meta mark != 0) prevents numgen from firing, and router-level service replies route back via the correct WAN table rather than the main routing table.

mwan3: add bypass_network UCI option to populate dynamic sets from globals config

Add mwan3_set_dynamic_network() callback and update mwan3_set_dynamic_sets() to read the bypass_network list from UCI globals config and populate mwan3_dynamic_v4/v6 from it at startup, rather than just flushing the sets empty. Each entry is classified as IPv4 (contains .) or IPv6 (contains :) and added to the appropriate set via the existing nft batch.

Add mwan3_set_dynamic_sets to the fw4 reload rebuild sequence in both 25-mwan3 and mwan3-fw-rebuild.sh so that bypass_network entries are restored after fw4 reload alongside the connected and custom sets.

The mwan3_dynamic_v4/v6 sets and mwan3_dynamic chain already existed and were already referenced in mwan3_prerouting, mwan3_output, and mwan3_iface_in_* chains. This change makes the feature accessible via UCI rather than requiring direct nft element injection.

mwan3: add fw4 reload detection and timestamp to mwan3-lb-test

When fw4 reload occurs during the test wait period, the 25-mwan3 hotplug rebuilds the policy chain without the injected counter rules. Previously this produced TOTAL=unknown and silent per-member zeros with a generic FAIL.

Now: detect missing counter rules explicitly and report "fw4 reload likely occurred during the test". Also add a human-readable completion timestamp to the summary output.

luci-app-mwan3: add Bypass networks field to globals page and improve rt_table_lookup UI

Add a Bypass networks DynamicList field (UCI option bypass_network) to the globals page. Accepts IPv4 and IPv6 CIDRs directly; traffic to these networks bypasses mwan3 policy routing and uses the default route. Validation uses the cidr datatype with blur-only firing (suppress keyup) via a capture-phase event delegation listener on the container node, covering dynamically added items without a MutationObserver.

Improve the rt_table_lookup field: rename label from "Routing table lookup" to "Routing table bypass" with a description that explains the bypass effect and that both table numbers and names are accepted. Remove the uinteger datatype constraint which incorrectly rejected table names, since the backend (ip route list table) accepts both.

18.23 Version 3.3.2

Summary: Version 3.3.2 fixes a spurious tracked-IP entry in ubus status output caused by a naming collision between mwan3track's temporary output file and the TRACK_* glob used by rpcd. The mwan3-lb-test tool gains mandatory client isolation, a Windows test command, and a stale-artifact cleanup subcommand. LuCI receives cross-field family consistency validation in the rule editor, a fix for false "Present (unexpected)" health badges during interface bring-up, source nftset display in the overview rules column, and source nftset support in the traffic path simulator.

mwan3: rename TRACK_OUTPUT temp file to PING_OUTPUT

The rpcd status module globs TRACK_* files in each interface's status directory to discover tracked IPs, using the suffix as the IP address. The temporary output file used by mwan3track for ping/httping/nping/nslookup output was named TRACK_OUTPUT, causing the rpcd glob to yield OUTPUT as a spurious tracked IP address in ubus call mwan3 status.

Rename the variable and its backing file to PING_OUTPUT to avoid the collision with the TRACK_* namespace.

mwan3: enhance mwan3-lb-test with client isolation and Windows test command

Add mandatory -c <client_ip> parameter: inserts a forward chain drop rule blocking pings to the test destination set from all LAN clients except the nominated test client, and an mwan3_output return rule bypassing mwan3 marking for router processes pinging the same IPs. Both rules are scoped to the test set and removed by cleanup().

Add cleanup subcommand to remove stale rules and sets left by a SIGKILL-terminated run.

Add Windows cmd.exe test command output alongside the existing Linux shell loop. Windows ping uses a fixed ICMP identifier (id=1), causing conntrack entry reuse on repeated pings to the same destination. The generated Windows command uses a longer inter-ping delay (30/TRACK_COUNT + 3 seconds) so that the full cycle through all test IPs exceeds the 30s ICMP conntrack timeout, ensuring each revisit creates a new conntrack entry.

luci-app-mwan3: fix missing cross-field family consistency checks in rule editor

nftset_validate() only checked set type against the explicit family field and returned true immediately when family was unset. This allowed invalid combinations such as an IPv6 source NFT set paired with an IPv4 destination address to pass validation silently.

Add cross-field checks:

ipset_src type vs dest_ip address family
ipset type vs src_ip address family
ipset_src type vs ipset type (mixed families on the same rule)

Add ip_family() helper to derive ipv4/ipv6 from an IP address string.

luci-app-mwan3: fix routing health showing Present (unexpected) during interface bring-up

During interface bring-up, mwan3 adds ip rules on the connected hotplug event before mwan3track has confirmed STATUS=online. The routing health page was calling renderStatusBadge with expectedPresent=false (derived from online=false), causing rules to be labelled Present (unexpected) even though their presence is normal and correct in this state.

Fix by introducing a tri-state expectedPresent parameter: true - expected present: green Present / red Missing false - expected absent: orange Present (unexpected) / muted Absent null - no expectation: muted Present / muted Absent (neutral)

Pass true when online, null otherwise. The card border colour already conveys health for non-online interfaces; the individual badges no longer need to second-guess presence during transitional states.

luci-app-mwan3: add src ipset to overview rules match column

Display ipset_src (source NFT set) in the Match column of the rules table on the Overview tab. Rename the existing ipset: label to dst ipset: to distinguish it from the new src ipset: label.

luci-app-mwan3: add ipset_src support to traffic path simulator

Extend the rule matching simulation to handle the source NFT set option (ipset_src) added alongside the existing destination NFT set (ipset). Updates ruleMatches() to check src_ip membership in the source set, matchSummary() to display it, and the set fetch loop to collect ipset_src names alongside ipset names.

Family validation: src_ip and dest_ip now validate against the selected address family on blur, showing a clear error if an IPv4 address is entered with family set to IPv6 or vice versa.

nftset improvements: the nftset dropdown is now populated via the new mwan3.nftset_info rpcd method instead of luci-mwan3 nftset dump. Sets are annotated with their address family ((IPv4)/(IPv6)) in the dropdown label. Family consistency is validated on save. Combining a destination nftset with dest_ip or a source nftset with src_ip is flagged as an error since both match the same dimension.

Source nftset: new ipset_src field allows matching source addresses against an nftset, complementing the existing destination nftset. Both src and dst nftsets can be set on the same rule and are ANDed together.

Grid display: Source and Destination columns now show the nftset name when no IP address is configured, preventing the rule from appearing as a wildcard match in the listing.

18.24 Version 3.3.1

Summary: Version 3.3.1 adds source nftset matching (ipset_src) as a complement to the existing destination nftset, fixes three distinct numgen counter contamination bugs that caused load-balancing distributions to skew under inbound or reply traffic, sweeps orphaned policy chains that accumulate when policies are removed from UCI without an fw4 reload, and corrects IPv6 ip rule detection in the routing health check. The release also adds nftset_info as an rpcd ubus method and introduces the mwan3-lb-test CLI tool for verifying load-balancing weight distributions against configured policy members.

mwan3: add ipset_src option for source nftset matching in rules

User rules previously supported only a destination nftset match via the ipset UCI option (ip daddr @set). Add ipset_src as an independent source nftset match (ip saddr @set), allowing both src and dst nftsets to be specified on the same rule and ANDed together.

The existing ipset option is unchanged for backward compatibility. ipset_src follows the same pre-creation and family handling logic as the destination set. The nfproto guard condition is updated to include ipset_src as an implicit family qualifier.

mwan3: add nftset_info rpcd ubus method

Returns the name and address-family type of all non-mwan3 nftables sets in table inet fw4. Used by luci-app-mwan3 to annotate the nftset dropdown with family information and validate that a selected set is consistent with the rule's configured address family.

mwan3: translate icmp to ipv6-icmp for IPv6 family rules

In nftables inet tables, meta l4proto icmp matches protocol 1 (ICMPv4) only. A UCI rule with proto=icmp and family=ipv6 previously generated meta nfproto ipv6 meta l4proto icmp, a contradiction that silently matched nothing.

Translate proto icmp to ipv6-icmp (proto 58) when family is ipv6 in mwan3_set_user_nft_rule() so the generated rule correctly matches IPv6 ICMP traffic. All other proto/family combinations are unaffected.

mwan3: add mwan3-lb-test load balancing distribution verifier

CLI tool that verifies a load-balancing policy distributes traffic according to configured member weights. Inserts temporary nft counter rules into the policy chain, generates a test ping command, waits for completion, then reports per-member actual vs expected hit counts with PASS/FAIL verdict.

Key design points:

Computes N from member weights (LCM/GCD) so expected counts per member are always whole numbers
Temporary nft set + ICMP-only rule inserted into mwan3_rules ensures only test pings hit the policy chain; DNS and other traffic to the same IPs is not matched
Default destination pool excludes any IPs already configured as mwan3 tracking IPs (mwan3track pings those via mwan3_output, which would contaminate the counter)
-6 flag selects IPv6 mode (ip6 daddr + ping6 + IPv6 well-known IPs)
Optional IP override arguments for sites with non-standard reachability
Cleanup on exit, SIGINT, SIGTERM and SIGPIPE; startup sweep removes stale artifacts from aborted previous runs
Installed as /usr/sbin/mwan3-lb-test alongside mwan3track and mwan3rtmon

mwan3: fix numgen counter contamination from inbound and reply traffic

Three bugs caused the numgen load-balancing counter in policy chains to be incremented by traffic that should never reach a policy chain. The effect is non-strict alternation and a skewed distribution that does not match the configured member weights.

Bug 1: inbound internet traffic (port scanners, bots) destined for the router's own WAN IP traversed mwan3_prerouting with mark=0, found no matching user rule, fell to the default policy, and fired numgen. Fix: add fib daddr type local return in mwan3_prerouting after the ICMPv6 ND bypass.

Bug 2: the router's replies to inbound connections (ICMP echo replies, TCP responses) passed through mwan3_output with ct mark=0 -- because bug 1's fix caused the inbound packet to bypass prerouting without saving a ct mark -- and fell through to the policy chain firing numgen. On a public IPv6 address receiving frequent external pings this generates ~10-20 spurious hits per second. Fix: add ct direction reply return as the first rule in mwan3_output.

Bug 3: a UCI rule with family ipv4 or ipv6 but no src_ip, dest_ip, or ipset generates a bare meta mark ... jump policy rule with no IP version restriction. Rules with address-based criteria get an implicit family qualifier from the ip/ip6 keyword; rules without any address criteria do not. Combined with bug 2, the router's IPv6 ICMP replies fell through all ip saddr/ip daddr rules and hit the bare default rule, firing numgen at high rate. Fix: in mwan3_set_user_nft_rule(), if none of src_ip/dest_ip/ipset_name are set and family is explicitly ipv4 or ipv6, prepend meta nfproto ipv4/ipv6 to nft_match.

mwan3: sweep orphaned policy chains on startup

When a policy is removed from UCI config, its mwan3_policy_* chain persists in table inet fw4 until the next fw4 reload flushes the table. mwan3 restart only manages chains it knows about from current config, leaving orphans indefinitely.

mwan3_set_policies_nft() now enumerates all mwan3_policy_* chains in the live ruleset and deletes any that have no corresponding UCI policy config before rebuilding the policy chains from config.

mwan3: fix routing_health ip rule detection for IPv6 interfaces

get_ip_rules() called ip -j rule list which is IPv4-only. mwan3 adds ip rules for IPv6 interfaces via ip -6 rule add, making them invisible to the IPv4 rule list. routing_health() therefore reported iif and fwmark rules as missing for any IPv6 mwan3 interface, showing a false red card even when mwan3 was functioning correctly.

Fix by querying both -4 and -6 rule tables and merging the results, matching the same pattern already used by get_table_routes(). The stale rule detection also benefits as it now sees IPv6 stale rules too.

18.25 Version 3.3

Summary: Version 3.3 adds three major LuCI diagnostic tools - a traffic path Simulator, a static Configuration analyser, and a live Routing health view - backed by two new rpcd ubus methods (nftset_members and routing_health). The configuration analyser detects undefined references, orphaned sections, and rule shadowing including correct IPv6 CIDR containment checks. The routing health view colour-codes per-interface ip rule and routing table state against live kernel state. The apk info vs apk list -I version display bug is also fixed.

mwan3: add nftset_members and routing_health rpcd ubus methods

nftset_members { set: "<name>" } returns the current members of a named nft set in table inet fw4. The set name is validated against [a-zA-Z0-9_-]+ before being passed to nft. Used by the luci-app-mwan3 Simulator tab for ipset rule matching and connected-network bypass detection.

routing_health {} compares the UCI configuration against live kernel state. For each mwan3 interface (by 1-based UCI order index N) it checks for ip rules at priorities 1000+N (iif) and 2000+N (fwmark), checks routing table N for a default route, and reads the mwan3track STATUS file. Reports stale ip rules in mwan3's priority range that have no matching UCI interface. Built-in blackhole (FWMARK_BASE + MAX_IFACES - 2 = 2061) and unreachable (FWMARK_BASE + MAX_IFACES - 1 = 2062) policy rules are explicitly excluded from stale detection.

Both methods are declared in the luci-app-mwan3 ACL files.

mwan3: fix software version display in troubleshooting output

apk info mwan3 returns the version from the repository index, not the installed version. When the installed package version differs from the repo version, the Software-Version section of mwan3 internal output shows the wrong version.

Use apk list -I to query installed packages only, ensuring the correct installed version is displayed.

luci-app-mwan3: extend rule-shadowing check to cover IPv6 CIDRs

The Configuration tab's rule-shadowing analysis previously skipped all IPv6 addresses, treating every IPv6 pair as 'may shadow'. Add proper IPv6 CIDR containment using BigInt arithmetic (expandIPv6, ipv6ToBigInt, ipv6CidrContains) so shadowed IPv6 rules are correctly identified. Refactor the IPv4 path into ipv4CidrContains to match the new structure.

luci-app-mwan3: add Simulator, Configuration, and Routing diagnostic tabs

Three new tabs backed by mwan3 internals:

Simulator (Network > MultiWAN Manager > Simulator): simulates which mwan3 rule matches a described packet and shows live policy state. Supports IPv4 and IPv6 CIDR matching, port ranges, nft set membership via nftset_members ubus method, and connected-network bypass detection via mwan3_connected_v4/v6 sets. Enter key in any field triggers simulation.

Configuration (Network > MultiWAN Manager > Configuration): static analysis of the mwan3 UCI configuration. Detects undefined references, orphaned sections, policies with no members, all-same-interface policies, and rule shadowing via IPv4 and IPv6 CIDR containment.

Routing (Status > MultiWAN Manager > Routing): live comparison of ip rules and routing tables against UCI configuration via routing_health ubus method. Per-interface cards show iif rule, fwmark rule, and routing table state with colour-coded health badges. Includes a field reference explaining the ip rule priority scheme for non-expert users. Stale rule detection excludes built-in blackhole/unreachable policy priorities (2061/2062).

Also adds nftset_members and routing_health methods to the rpcd module with appropriate ACL entries in both luci-app-mwan3 and luci-app-mwan3-status.

18.26 Version 3.2.3

Summary: Version 3.2.3 improves tracking status visibility by adding per-IP latency and packet-loss detail to mwan3 status output and fixing the check_quality display to derive its state from mwan3track's runtime files rather than UCI, so changes to UCI without a restart no longer cause the status page to disagree with what is actually running. Stale gateway TRACK_*/LATENCY_*/LOSS_* files from previous PPPoE sessions are cleaned up on each probe list rebuild. The luci-app-mwan3 PKG_VERSION scheme is fixed to prevent apk upgrade from reverting to the official package, and the GitHub Actions APK rename step is corrected to avoid i18n sub-packages overwriting the main package.

mwan3: add per-IP tracking detail to interfaces status output

mwan3_report_iface_status now prints each tracking IP below the interface status line, showing its status and (when check_quality=1) latency and packet loss:

  interface wan is online and tracking is active
    track 8.8.4.4: up (12ms, 0% loss)
    track 8.8.8.8: down (-, 100% loss)
    track 192.168.1.1: ignored

check_quality is derived from LATENCY_* file presence (not UCI) to reflect actual mwan3track runtime state. The kernel status value "skipped" is displayed as "ignored" for consistency with the LuCI status page.

mwan3: fix check_quality display inconsistency when UCI is changed without restart

When check_quality is changed in UCI without restarting mwan3track, the rpcd status reported the new UCI value immediately while mwan3track continued running with the old setting, causing the status page to show "Not enabled" for latency/loss even though measurements were still being taken (or vice versa).

Fix in two parts:

rpcd: derive check_quality from LATENCY_* file content rather than UCI. mwan3track only writes LATENCY_* files when check_quality=1, so file presence with content is the authoritative indicator of actual runtime state. A fresh read of UCI is not needed on the rpcd side.

mwan3track: in mwan3_load_track_ips(), re-read check_quality from the current UCI config and remove any stale LATENCY_*/LOSS_* files when it is 0. This runs at startup and on every ifup event, so stale files from a previous check_quality=1run are cleaned up as soon as the interface next comes up withcheck_quality=0`, ensuring the rpcd file-based detection also returns the correct result after restart.

mwan3: remove stale gateway TRACK_/LATENCY_/LOSS_* files on probe list rebuild

When track_gateway=1 and the gateway IP changes (e.g. PPPoE reconnect), mwan3_load_track_ips builds a new probe list with the new gateway but leaves TRACK_*, LATENCY_*, and LOSS_* files from the old gateway IP on disk. The rpcd status function globs all TRACK_* files to build the tracking IP list, so stale files from previous gateways appear in ubus call mwan3 status alongside the current ones.

Fix: after building track_ips, scan the interface directory and delete any TRACK_*/LATENCY_*/LOSS_* files for IPs not in the current probe list. Runs at startup and on every ifup event, covering both initial cleanup of files left by a previous mwan3track instance and gateway changes during operation.

luci-app-mwan3: set explicit PKG_VERSION and fix release workflow

Without explicit versioning, luci.mk auto-generates the package version from git commit metadata. In the GitHub Actions SDK build environment the feed directory has no .git (excluded by rsync), so git walks up to the SDK repo and picks up the SDK commit hash and timestamp. The resulting version (e.g. 26.099.76803~8a085e7) is indistinguishable from the official OpenWrt package version format, causing apk to install the official repo version.

Set PKG_VERSION to $(PKG_SRC_PREFIX).$(PKG_SRC_SUFFIX) where PKG_SRC_PREFIX is derived from the build year and a fixed day value of 999 (not a valid calendar day), ensuring it is always higher than any real date-based official package version for that year. PKG_SRC_SUFFIX encodes the mwan3 version number for readability.

Also fix the APK rename step to only match the main luci-app-mwan3 package (pattern: PKG_NAME[_-][0-9]*.ext) instead of all APKs in the feed directory. The previous pattern matched all i18n sub-packages as well; since they all got renamed to the same target filename, the last one alphabetically clobbered the main package and was what actually got uploaded to the release.

luci-app-mwan3: improve tracking IP latency/loss display for down/skipped/disabled states

When check_quality=1, tracker latency/loss sentinel values (999999ms, 100%) are replaced with more informative indicators:

Tracker down: latency shows ∞ (scaled to be legible), packet loss retains 100% since it is accurate
Tracker skipped: both show - since no probe was run that round
Interface disabled: tracker status shows 'Disabled' (muted) instead of 'Down', and both latency and loss show -

18.27 Version 3.2.2

Summary: Version 3.2.2 fixes two misrouting bugs: duplicate jump rules accumulating from repeated fw4 reload cycles caused iface_in chain deletion to fail with "Resource busy", and the unguarded catchall rule in each mwan3_iface_in_* chain was stamping IPv6 packets with the IPv4 interface mark on dual-stack physical devices, breaking QUIC/HTTP3 streams that resumed after conntrack expiry. The gateway IP is moved to the front of the tracking probe list so it is always tested. LuCI receives a visual redesign replacing solid alert cards with bordered flex cards, and adds latency and packet-loss columns to the tracking IP table.

mwan3: fix iface_in jump rule deletion failing with multiple handles

mwan3_delete_iface_nft used a single-shot handle lookup to remove the jump rule for an interface from mwan3_ifaces_in. If repeated fw4 reload cycles caused duplicate jump rules to accumulate, the handle variable received multiple space-separated values and the resulting nft command was syntactically invalid. The chain delete that followed then failed with "Resource busy" since the jump rule was still present.

Replace the single-shot lookup with a while loop using head -n1, matching the pattern already used for SNAT rule deletion in the same function. This correctly removes all copies of the jump rule one at a time and self-heals any pre-existing duplicates on the next ifdown.

mwan3: fix iface_in catchall misclassifying IPv6 on shared-device interfaces

The catchall rule in each mwan3_iface_in_* chain lacked a meta nfproto filter. When an IPv4 and IPv6 mwan3 interface share the same physical device (e.g. a dual-stack PPPoE or L2TP WAN), mwan3_ifaces_in dispatches to the IPv4 chain first. The IPv4-specific ip saddr bypass rules do not match IPv6 packets, but the unguarded catchall does, stamping incoming IPv6 packets with the IPv4 interface mark instead of the IPv6 mark.

The ct mark is then saved with the wrong value. Subsequent packets for that connection restore the IPv4 mark, find no matching ip -6 rule, and fall to the main routing table. For established TCP connections the bug is invisible because ct mark is set on the first outbound packet before any inbound packet arrives. For QUIC (UDP), conntrack entries expire after ~120-180s of inactivity; when the server resumes first the entry has ct mark 0, the catchall fires, and the connection is disrupted - manifesting as stalled streams or a gray page on sites using HTTP/3.

Fix by adding meta nfproto ipv4/ipv6 to the catchall based on interface family, consistent with the protocol filters already applied to the connected/custom/dynamic bypass rules above it.

mwan3: ping gateway IP first when track_gateway=1

mwan3_load_track_ips() previously appended the gateway IP to the end of the track_ips list. Since mwan3track stops probing once host_up_count reaches the reliability threshold, the gateway would be skipped whenever an earlier IP responded.

Prepend the gateway instead so it is always probed first. This gives the gateway probe priority and ensures its reachability is tested on every round regardless of reliability setting.

mwan3: fix rpcd status to include gateway tracking IP and restore latency/loss

interfaces_status() built the track_ip list from UCI track_ip options only, so the gateway IP (added dynamically by mwan3track when track_gateway=1) was never visible in ubus status output.

Replace the UCI iteration with a glob scan of the TRACK_* files that mwan3track actually writes, so all tracked IPs appear regardless of whether they come from UCI config or the dynamic gateway.

Also correct get_mwan3track_status() to treat track_gateway=1 as an active tracking configuration (not disabled), and restore the latency/packetloss fields to the per-IP output (populated when check_quality=1 is configured).

luci-app-mwan3: visual redesign of status cards across overview and main status page

Replace solid alert-message background cards with bordered flex cards throughout: interface status boxes now use a 2px border coloured to match the interface status (green/red/orange/grey) with no background fill.

detail.js: status box border colour matches interface status; paused tracking state changed from warning orange to muted grey; tracking IP table rows sorted by status (up first, then down, then ignored).

overview.js: policy section replaced with per-policy flex cards matching the interface card style; interface card borders coloured by status.

90_mwan3.js: main LuCI status overview page cards converted from alert-message solid backgrounds to the same bordered card style.

luci-app-mwan3: add latency and packet loss columns to tracking IP status table

The rpcd mwan3 status now returns latency, packetloss and check_quality fields per interface. Extend the Status tab tracking IP table with Latency and Packet Loss columns.

When check_quality is disabled (the default), the columns display "Not enabled" in muted text rather than zeroed values.

18.28 Version 3.2.1

Summary: Version 3.2.1 fixes policy status reporting to include all members with their live traffic share percentages (not just the currently-routing member), replaces killall -HUP dnsmasq with a procd-aware targeted SIGHUP to avoid crashing instances still in the startup phase, adds the installed mwan3 package version to mwan3 internal output, and redesigns the LuCI status pages with structured collapsible sections and an IPv6 troubleshooting pane. LuCI also exposes the snat6 IPv6 SNAT option on interface configuration.

mwan3: show mwan3 package version in internal troubleshooting output

Replace the OpenWrt release string in the Software-Version section of mwan3 internal with the installed mwan3 package version from apk.

mwan3_report_policies() read the live nftables policy chain to determine members, which only contains the currently-routing member. Metric-2 standby members have no nft rules while metric-1 is up, so they were invisible in the status output.

The rpcd ucode get_policies() had the same problem, and additionally returned the raw nft mark value instead of the interface name.

Fix both by reading policy membership from UCI config and cross-referencing with mwan3track STATUS files to determine which metric is active and what the traffic share is for each member.

Every member now shows a percentage: 100% for a sole active member, the load-balanced share for equal-priority members, and 0% for standby/offline members. This makes failover and load-balancing configuration immediately readable at a glance.

The shell status command now calls ubus rather than duplicating the metric/weight/status logic in shell.

mwan3: fix dnsmasq SIGHUP race during concurrent startup

killall -HUP dnsmasq signals every process named dnsmasq, including instances that are mid-initialisation. A dnsmasq receiving SIGHUP before it has completed startup exits, which can happen when mwan3's fw4-reload recovery (25-mwan3 at position 25) fires concurrently with anything else restarting dnsmasq.

Replace killall -HUP dnsmasq with mwan3_dnsmasq_hup(), which queries procd via ubus for the dnsmasq service instances and sends SIGHUP only to PIDs that procd reports as running=true. Instances still in the startup phase are invisible to the function and are not signalled.

json_set_namespace is used to protect the caller's jshn state since mwan3.sh uses jshn elsewhere.

luci-app-mwan3: redesign status pages with structured views

Overview tab: replace simple interface cards with full operational view showing interface status cards (flex layout), policies table with per-member traffic share percentages, and rules table.

Status tab: replace static interface cards with per-interface tracking health panels showing tracking mode, score, and a table of probe IPs with up/down/ignored status using coloured text indicators.

Troubleshooting tab: replace raw pre-formatted text dump with collapsible sections (collapsed by default with expand arrow), IPv6 sections alongside IPv4, and vmap-dispatch boilerplate chains filtered from the nftables output. Add ACL entry for mwan3 internal ipv6.

luci-app-mwan3: expose mwan3 snat6 option for IPv6 interfaces

Adds an "IPv6 SNAT" form field to the interface configuration modal, visible only when family is set to ipv6. Mirrors the placement of the existing "Track gateway" field. Accepts unset/0 (off, default), 1 (SNAT to the interface's primary GUA), or a literal IPv6 address for NPTv6-style fixed-source pinning. The control is opt-in by design -- see the mwan3 package for the rationale (RFC 6724 source-address selection, NAT66 harm in PA/ULA designs, fixed-saddr upstream requirements).

18.29 Version 3.2

Summary: Version 3.2 adds two significant features. First, opt-in per-interface IPv6 SNAT via the snat6 UCI option, which corrects BCP38/uRPF drops for router-originated traffic rerouted by mwan3_output onto a different WAN than the kernel initially selected at sendto(). Second, non-destructive vmap-dispatch mark save/restore: 126 per-mark OR-immediate setter chains replace the previous unmasked connmark operations, making mwan3 fully order-independent with respect to pbr and other fwmark-using packages without requiring coordinated chain priority ordering.

mwan3: add opt-in IPv6 postrouting SNAT via snat6 UCI option

Adds a mwan3_postrouting base chain (type nat hook postrouting priority srcnat - 1) and opt-in per-interface IPv6 SNAT for router-originated traffic that mwan3_output has rerouted onto a different WAN.

The kernel binds saddr at sendto() using the unmarked initial route; mwan3_output then reroutes oif onto a different WAN; without SNAT the packet egresses with the wrong source prefix and is dropped upstream by BCP38/uRPF. fw4 does not masquerade IPv6 by default so there is no automatic fallback. mwan3track is unaffected because it uses SO_BINDTODEVICE. IPv4 router-originated traffic does not require explicit SNAT -- fw4's masquerade handles it.

IPv6 SNAT is opt-in per interface via the new snat6 UCI option, defaulting to off because RFC 6724 source-address selection / SADR routing can solve the same problem without translation, NAT66 is harmful in PA/ULA designs, and some upstreams require a specific saddr. snat6 values: unset/0 (off), 1 (SNAT to interface primary GUA via mwan3_get_src_ip), or a literal v6 address (NPTv6-style fixed-source pinning). Cleanup is via comment-tag matching ("mwan3_snat_<iface>").

mwan3: non-destructive vmap-dispatch save/restore

Restores iptables-equivalent non-destructive masked CONNMARK semantics by synthesising masked save/restore via vmap dispatch into 126 per-mark OR-immediate setter chains (mwan3_or_meta_*, mwan3_or_ct_*, 63 each). Each setter chain is a 2-statement skeleton "meta/ct mark set ... | <imm>; return". The kernel rejects the obvious compound form "(meta mark & ~M) | (ct mark & M)" because a set-statement may reference at most one runtime source register; vmap-dispatch routes around this by materialising the constant immediate at rule-emit time.

Because the save and restore operations are now non-destructive, mwan3's fwmark bits are ORed in and out of the conntrack mark without disturbing bits owned by other packages. This makes mwan3 order-independent with respect to other fwmark-using packages such as pbr, regardless of chain priority ordering.

The same vmap-dispatch primitive is reused by mwan3_create_policies_nft for load-balancing numgen rules and by the sticky implementation in mwan3_set_user_nft_rule, both of which were previously destructive in unmasked bits. Sticky routing is rebuilt around per-(rule, family, member) ip-only sets that pair with the per-mark setter chains, replacing the legacy ip->mark map. mwan3_delete_iface_map_entries and mwan3_report_policies are updated for the new sticky and numgen forms.

18.30 Version 3.1.4

Summary: Version 3.1.4 fixes interoperability with pbr by moving mwan3's prerouting and output chains from priority mangle + 1 to mangle - 1, so mwan3 restores and saves its ct mark bits before pbr injects its own marks at mangle priority. With the previous ordering pbr's marks were zeroed before the routing decision and its ip rules never matched.

mwan3: fix pbr interoperability by running at priority mangle - 1

Move mwan3_prerouting and mwan3_output from priority mangle + 1 to mangle - 1 so mwan3 runs before pbr, which injects into fw4's mangle_prerouting at priority mangle (-150).

With the old priority, pbr set marks in 0x00ff0000 first, then mwan3's unmasked restore (meta mark set ct mark & MMX_MASK) zeroed them before the routing decision, causing pbr's ip rules to never match.

With mangle - 1, mwan3 restores and saves its mark before pbr runs. pbr then adds its bits on top, and both sets of marks are present at the routing decision -- matching the coexistence behaviour of the original iptables implementation.

Add postinst migration to flush and delete the old chains on upgrade, since nftables rejects a chain redeclaration with a different priority.

18.31 Version 3.1.3

Summary: Version 3.1.3 fixes three status and policy rendering bugs: single-member policies were emitting spurious "unreachable" entries because the empty-string guard on mwan3_mark_to_name never matched; mixed IPv4/IPv6 policies lost one family's members because both shared a single reset list; and equal-weight load-balancing entries were invisible in mwan3 status because nft normalises single-element numgen ranges to plain values that the reporting regex did not match.

mwan3: fix status reporting for single-member and mixed-family policies

mwan3_report_policies used mwan3_mark_to_name on the mark from the single-member branch and tested [ -n "$iface_name" ] to skip special marks. However mwan3_mark_to_name never returns an empty string: it returns "unreachable", "blackhole", "default", or the raw hex value as a fallthrough for unrecognised marks. The empty-string test therefore never filtered anything, causing spurious "unreachable" output.

Replace the empty-string guard with a case statement that explicitly skips the known non-interface values (unreachable, blackhole, default, and raw 0x... hex fallthrough). Also iterate all "meta mark set" rules in the chain rather than only the first, so mixed-family policies with one IPv4 and one IPv6 member both appear in the status output.

mwan3: fix cross-family member reset in mixed IPv4/IPv6 policies

When a policy contains members from both IPv4 and IPv6 interfaces, the previous code used a single policy_members list that was reset on each new lowest-metric member regardless of address family. This caused IPv4 members to be erased when a lower-metric IPv6 member was processed (or vice versa), leaving the policy with only the last-processed family.

Fix by maintaining separate policy_members_v4 and policy_members_v6 lists, each reset only when a new lowest-metric member of the same family is encountered. When both families have members, emit per-family nft rules guarded with meta nfproto ipv4/ipv6 so traffic is only directed to members of the matching address family.

mwan3: fix load balancing policy not shown in status

nft normalizes single-element ranges (e.g. 0-0) to plain values (e.g. 0) when listing rules. The reporting regex only matched the range format N-M : 0xMARK, so equal-weight load balancing entries (weight=1, displayed as N : 0xMARK) were silently skipped, causing the policy to appear empty in mwan3 status despite working correctly.

Handle both N-M : 0xMARK (weight>1, range preserved by nft) and N : 0xMARK (weight=1, normalized by nft) formats.

18.32 Version 3.1.2

Summary: Version 3.1.2 improves mwan3rtmon with two fixes: an in-memory route cache replaces the per-event RTM_GETROUTE dump for O(1) ECMP path checks, and a ucode-mod-rtnl double-destructor bug that caused a reliable segfault on clean shutdown is eliminated by letting the GC collect the route listener rather than calling close() explicitly.

mwan3rtmon: replace route_still_exists() with an in-memory route cache

route_still_exists() issued a full RTM_GETROUTE dump on every route-delete event to check whether an ECMP path still existed before removing per-interface table entries. This is unnecessary overhead.

Replace with main_route_cache: a { route_key: count } map built from the initial route snapshot in populate_iface_routes() and maintained incrementally in handle_route_event(). The ECMP check becomes an O(1) cache lookup with no rtnl round-trip.

mwan3rtmon: fix segfault on exit caused by ucode-mod-rtnl double-destructor bug

Calling route_listener.close() explicitly zeroes the resource data pointer while the ucode variable still holds a live reference. When that reference is later released at scope exit, uc_nl_listener_free() fires a second time with arg=NULL and reads uc_nl_listener_t.index at NULL+0x10, producing a reliable "segfault at 10" on every clean shutdown.

Fix: omit the explicit close() call and let the GC collect the listener naturally. The destructor then fires exactly once with a valid pointer.

18.33 Version 3.1.1

Summary: Version 3.1.1 is a broad mwan3track hardening release: the disconnecting threshold is raised to suppress false alarms from single transient ping losses, an exclusive flock prevents ghost duplicate tracker processes per interface, sockopt_wrap replaces exit() with graceful error returns so a stale source IP or disappearing interface does not abruptly terminate the tracked process, per-host failure logs are suppressed when the reliability threshold is still met, and interface events are processed at the top of the main loop before pinging to avoid a spurious disconnecting state on wakeup from disabled. LuCI adds a track_gateway checkbox to the interface modal and clarifies the flush_conntrack help text.

mwan3track: raise disconnecting threshold and log recovery

Raise the threshold at which the disconnecting state fires from the first score drop to ceil(down/3) failures below the maximum score. This prevents single or double transient ping losses (e.g. to a public DNS server) from spuriously triggering the disconnecting state.

Log an explicit notice when the score recovers out of the disconnecting state, making the transition back to online visible in the log.

mwan3track: use flock to prevent ghost processes for same interface

Use an exclusive flock on a per-interface lock file to ensure only one mwan3track instance runs per interface at a time. The lock is held for the lifetime of the process and released unconditionally on exit.

mwan3: sockopt_wrap: replace exit() with graceful error returns

A LD_PRELOAD shim must not call exit() on recoverable errors; doing so terminates the tracked process abruptly with no log from mwan3track, making it indistinguishable from a genuine connectivity failure.

Three cases are fixed:

Source IP bind failure in dobind(): can occur when SRC_IP becomes stale after a DHCP address change that does not generate an ifup event. Close the socket and return; the subsequent sendto()/connect() call will fail with EBADF, causing the ping to exit with a normal error code that mwan3track records as a ping failure.
SO_BINDTODEVICE failure in socket(): can occur if the interface disappears between m### mwan3track: suppress per-host failure logs when reliability threshold is met

With multiple track IPs and reliability < number of IPs, a single host failure was logged as "Check failed for target X" even when a subsequent host met the reliability threshold and the round succeeded. 95% of all logged failures are false alarms of this kind, making it appear the interfaces are degrading when they are in fact healthy.

Fix the excessive log noise by accumulating failed host names during the probe loop and logging them in a single message after the loop, when host_up_count is still below the reliability threshold (round genuinely failed). Rounds that succeed via a later host produce no failure log. The success log during recovery remains per-host and unchanged.

For check_quality mode the accumulated entry includes per-host latency and loss: "target(s) \"1.2.3.4(999999ms/100%)\"".

mwan3track: process interface events before ping round on wakeup

When USR2 (ifup) wakes mwan3track from disabled state, the main loop resumes past the MAX_SLEEP wait and immediately starts a ping round before reaching the IFUP_EVENT handler at the bottom of the loop. At that point DEVICE is still stale (empty string at first start), so sockopt_wrap skips SO_BINDTODEVICE, the unbound ping fails, and a spurious "disconnecting" state is logged.

Fix this by checking IFDOWN_EVENT/IFUP_EVENT at the top of the loop, before any pinging, and using continue to restart the iteration cleanly after firstconnect() has refreshed DEVICE and SRC_IP. The existing handlers at the bottom of the loop are retained for events that arrive during a ping round or the inter-round sleep.

luci-app-mwan3: clarify flush_conntrack help text

Update the help text for the flush_conntrack option to clarify that it flushes the entire global conntrack table, and that per-interface conntrack entries are already flushed automatically on ifdown since commit ("mwan3: selectively flush conntrack entries for failed WAN interface on ifdown").

luci-app-mwan3: add track_gateway option to interface settings

Add a "Track gateway" checkbox to the interface configuration modal, visible only when the internet protocol is set to IPv4. This exposes the track_gateway UCI option added to the mwan3 backend for automatic point-to-point peer/gateway tracking.

19. Specific use-cases

19.1 Tailscale

Tailscale's specific routing architecture is incompatible with legacy mwan3, meaning that Tailscale had to explicitly bypass mwan3 and to that end, has specific detection code and a bypass. Tailscale cannot be managed with legacy mwan3 policies.

mwan3 nf tables, however, has the requisite functionality to enable peaceful co-existence between the two packages. If mwan3 is left in default configuration, Tailscale will continue to bypass mwan3 as it has always done and users will notice no difference.

However, mwan3 can be specifically configured to policy route Tailscale bypass-marked traffic. The rest of this section explains how to do that.

Configuring mwan3 failover for tailscale control traffic

Tailscale marks every socket it opens for its own outbound connections (DERP relay servers, coordination server, STUN) with SO_MARK = 0x80000 at socket creation time via SetsockoptInt. In normal operation with mwan3, it then installs an ip rule at priority 1310 that intercepts these marked packets and routes them to the main routing table (table 254, managed by netifd), bypassing mwan3's per-interface tables entirely:

1310: fwmark 0x80000/0xff0000 lookup main (254)
1330: fwmark 0x80000/0xff0000 lookup default (253)
1350: fwmark 0x80000/0xff0000 unreachable
1370: (unconditional) lookup tailscale (52)

Tailscale installs these rules at the 1300 base specifically because it detects OpenWrt with mwan3 active (checkOpenWRTUsingMWAN3() in wgengine/router/osrouter/router_linux.go) and shifts its default base from 5200 to 1300. The 1300 base is chosen to fit between mwan3's default iif rules (1001-1060) and mwan3's default fwmark lookup rules (2001-2060). The rules at 1310-1350 are an anti-loop mechanism: if tailscaled's own outbound traffic fell through to rule 1370 and was routed into tailscale0, it would re-enter the overlay indefinitely.

The problem is that rule 1310 routes tailscale's bypass-marked traffic via the main routing table, which is maintained exclusively by netifd and is not subject to mwan3's failover machinery. In the most common real-world WAN outage (upstream unreachable with carrier still up), netifd sees carrier and leaves the primary WAN route in the main table, so tailscale's control traffic continues trying to reach DERP servers via the dead upstream while mwan3 has already failed LAN traffic over to the backup WAN.

Tailscale requires this in order to prevent a routing loop, whereby bypass-marked packets enter the tailscale device via routing table 52 and was necessary when using Tailscale with legacy mwan3.

Making use of three features in mwan3 nftables, it is possible to get tailscale to benefit from mwan3 while also avoiding the aforementioned loop.

The solution has three parts:

A new mwan3 rule type that matches on a fwmark/fwmask.
The ability to shift the mwan3 rule priorities to a new base
Dynamic route monitoring by mwan3rtmon that mirrors routes from Tailscale's table 52 defined with list rt_table_lookup '52' in the globals section of /etc/config/mwan3. mwan3rtmon will dynamically mirror peer IP/CIDRs in table 52 to the mwan3 sets mwan3_custom_v4/v6. Anything present in these sets will thus bypass mwan3's rules, fall through to the next rules in the list and get correctly routed by Tailscale's table 52.

How the three parts work to integrate mwan3 and tailscale

The mwan3 rule matches on fwmark 0x80000/0xff0000 and assigns the tailscale bypass marked packets to an mwan3 policy
The mwan3 rule fires before tailscale's bypass because of the route ordering priority change and routes out whichever is the active WAN. This gives the failover: the bypass marked packets are the outer tunnel / control plane
Traffic destined for the tailscale inner tunnel, which is normally looked up in table 52, has matching IPs and CIDRs in the mwan3_custom_v4/6 sets put there by mwan3rtmon. Any destination IP in those sets will now bypass mwan3's specific handling and fall through to tailscale's rule that looks them up in table 52, routing them out the tailscale0 device.

mwan3's fwmark and tailscale's bypass mark do not interfere with each other

mwan3 uses bits 8-13 of the packet mark (0x3f00 default mask). Tailscale's bypass mark uses bits 16-23 (0xff0000). The two ranges do not overlap. mwan3's mark-set operation is non-destructive: it uses a bitwise OR via vmap-dispatch setter chains, so it writes only to the bits it owns and leaves all other bits intact.

After mwan3_output fires on a tailscale bypass packet, the packet carries both marks simultaneously in separate bit fields:

bits 16-23: 0x80000 (tailscale bypass mark, unchanged)
bits 8-13: 0x100 (mwan3 policy mark for interface id 1, ORed in)
final mark: 0x80100

Choosing the priority values

The only constraint is that mwan3's fwmark rules must fall outside of tailscale's mwan3 detection window. checkOpenWRTUsingMWAN3() looks for ip rules at priorities 2001-2004 with a non-zero fwmark. If any fwmark rule falls in that range, tailscale detects mwan3 and shifts its rules to a 1300 base, which would interleave with mwan3's fwmark rules. The constraint is:

fwmark_rule_base + MWAN3_INTERFACE_MAX < 2001

With the default mask 0x3f00 (6 bits, 60 maximum interfaces), this gives fwmark_rule_base < 1941. The recommended value is fwmark_rule_base = 1100, which places the highest-numbered fwmark lookup rule at 1160, well clear of the detection window. Tailscale therefore does not detect mwan3 and installs at its default 5210 base instead.

Because iif_rule_base defaults to 1000 and can stay there, only two UCI options need to be changed from their defaults:

Option	Default	Recommended for tailscale
`iif_rule_base`	1000	1000 (unchanged)
`fwmark_rule_base`	2000	1100
`unreachable_rule_base`	3000	1200

The ordering constraints validated by mwan3 at startup both hold:

1000 + 60 = 1060 < 1100 (iif tier clears fwmark tier)
1100 + 60 + 1 = 1161 < 1200 (fwmark tier including global blackhole/unreachable at 1161/1162 clears unreachable tier)

Why rule 5270 (table 52) still works correctly

Tailscale's rule 5270 is unconditional - it fires for every packet that reaches it. With mwan3's fwmark rules at 1101-1160, rule 5270 is reached only by packets whose mark & 0x3f00 did not match any live per-interface id mark. The relevant case is packets carrying MMX_DEFAULT (mark = 0x3f00).

mwan3_prerouting marks packets destined for networks in mwan3_connected_v4 and mwan3_custom_v4 with MMX_DEFAULT as an early-return bypass. MMX_DEFAULT does not match any per-interface fwmark rule (those require specific id marks, not all-bits-set), so these packets fall through to rule 5270, which looks up table 52 and routes them to tailscale0.

Ensuring tailscale peer destinations appear in mwan3_custom_v4 is therefore the prerequisite for correct tailnet routing after the fwmark_rule_base change. This is described in the next section.

Ensuring tailnet-destined traffic reaches `tailscale0`

Tailscale installs peer routes exclusively into table 52 via netlink. The main routing table receives no tailscale peer routes. With mwan3's fwmark rules moved to 1101-1160, the default catch-all mwan3 rule assigns a WAN policy mark to all LAN traffic including packets destined for tailscale peers. That marked traffic hits rule 1101 before it reaches rule 5270. The per-interface routing table has a default route to the internet gateway but no route to the tailscale peer. The packet exits via the WAN unencrypted.

Under the default mwan3 configuration (fwmark rules at 2001+), tailscale rule 1370 fires at priority 1370, before the fwmark lookup rules at 2001+. The WAN policy mark is shadowed by rule 1370 and the packet reaches tailscale0 correctly. Moving the fwmark base to 1100 removes that shadowing.

The fix does not require touching routing tables. mwan3 provides mwan3_custom_v4 and mwan3_custom_v6, nftables sets whose members receive MMX_DEFAULT in mwan3_prerouting and thereby bypass WAN policy selection entirely. The rt_table_lookup option in mwan3 globals instructs mwan3 to populate mwan3_custom_v4/v6 from the routes in a specified routing table at start. Adding table 52 to this list:

list rt_table_lookup '52'

causes mwan3 to read all current table 52 routes at start and add their destinations to mwan3_custom_v4/v6. Packets to tailscale peers receive MMX_DEFAULT, pass through all per-interface fwmark rules without matching, and reach rule 5270 (from all lookup tailscale), which routes them to tailscale0 via table 52. No per-interface routing table needs a tailscale route.

This is the correct treatment for the inner tailnet packets. tailscale0 is a locally-connected TUN device: the inner packets are delivered to the tailscale daemon, which wireguard-encapsulates them into UDP. Those outer packets carry SO_MARK=0x80000 and are subject to mwan3 WAN failover via the dual-mark mechanism. MMX_DEFAULT on the inner packets correctly expresses that WAN selection does not apply at this layer; it is applied at the outer encrypted layer instead.

Tailscale adds and removes peer routes from table 52 dynamically as peers join and leave the tailnet. mwan3rtmon watches for RTM_NEWROUTE and RTM_DELROUTE events on tables listed in rt_table_lookup and updates mwan3_custom_v4/v6 accordingly, so the set remains current without any external script or daemon.

If mwan3 starts before tailscale has populated table 52 (for example, on boot), the initial rt_table_lookup pass reads an empty or partial table 52 and mwan3_custom_v4 is initially partially complete. mwan3rtmon's event handler fills in the missing entries as tailscale adds routes after its own startup, so the set converges to the correct state within seconds of tailscale coming up.

The resulting rule layout

With fwmark_rule_base = 1100 and unreachable_rule_base = 1200, and four mwan3 interfaces for illustration. Tailscale's ip rules are shown in their steady-state position, explained below. The moving of mwan3's base priority means that tailscale won't actually detect mwan3 to be running and will revert its own rule priority to the default of 5210 instead of installing at 1310.

 0:       from all lookup local
 1001:    iif <wan1-dev> lookup 1                      # mwan3 iif
 1002:    iif <wan2-dev> lookup 2                      # mwan3 iif
 1003:    iif <wan3-dev> lookup 3                      # mwan3 iif
 1004:    iif <wan4-dev> lookup 4                      # mwan3 iif
 1101:    fwmark 0x100/0x3f00 lookup 1                 # mwan3 fwmark  
 1102:    fwmark 0x200/0x3f00 lookup 2                 # mwan3 fwmark
 1103:    fwmark 0x300/0x3f00 lookup 3                 # mwan3 fwmark  
 1104:    fwmark 0x400/0x3f00 lookup 4                 # mwan3 fwmark
 1161:    fwmark 0x3d00/0x3f00 blackhole               # mwan3 blackhole
 1162:    fwmark 0x3e00/0x3f00 unreachable             # mwan3 unreachable
 1201:    fwmark 0x100/0x3f00 unreachable              # mwan3 unreachable
 1202:    fwmark 0x200/0x3f00 unreachable              # mwan3 unreachable
 1203:    fwmark 0x300/0x3f00 unreachable              # mwan3 unreachable
 1204:    fwmark 0x400/0x3f00 unreachable              # mwan3 unreachable
 5210:    fwmark 0x80000/0xff0000 lookup main (254)    # tailscale anti-loop
 5230:    fwmark 0x80000/0xff0000 lookup default (253) # tailscale anti-loop
 5250:    fwmark 0x80000/0xff0000 unreachable          # tailscale anti-loop
 5270:    from all lookup tailscale (52)               # tailscale overlay routing
 32766:   from all lookup main
 32767:   from all lookup default

Tailscale's mwan3 detection (checkOpenWRTUsingMWAN3() in wgengine/router/osrouter/router_linux.go) identifies mwan3 by looking for ip rules at priorities 2001-2004 with a non-zero fwmark. With fwmark_rule_base = 1100 those rules are at 1101-1104, outside the detection window. Tailscale therefore does not detect mwan3 and installs its rules at its default base of 5200 (priorities 5210, 5230, 5250, 5270) .

This does not affect correctness. Tailscale bypass-marked packets (0x80000) acquire an mwan3 policy mark in mwan3_output and are caught at 1101-1104, well before tailscale's rules at 5210. The anti-loop concern that motivated rules 5210-5250 does not apply: the per-interface routing tables contain internet routes, not tailscale overlay routes, so there is no loop path. Tailnet-destined LAN traffic carries MMX_DEFAULT (via mwan3_custom_v4, described above) and passes through all fwmark rules without matching before reaching rule 5270, which routes via table 52 to tailscale0.

Configuration

In /etc/config/mwan3, in the config globals section, change the two rule base options and add the rt_table_lookup entry for tailscale's routing table (this can also be done on the Globals tab in Luci, as well as a bypass network list bypass_network '100.64.0.0/10':

config globals 'globals'
    option mmx_mask '0x3F00'
    option fwmark_rule_base '1100'
    option unreachable_rule_base '1200'
    list rt_table_lookup '52'
    list bypass_network '100.64.0.0/10'

Add a rule section for tailscale bypass traffic, placed before any catch-all default rules, preferably as the very first rule in the list:

config rule 'tailscale_bypass'
    option fwmark '0x80000'
    option fwmask '0xff0000'
    option family 'any'
    option use_policy '<my_failover_policy>'
    option enabled '1'

Replace <my_failover_policy> with whichever mwan3 policy should carry tailscale's control traffic. Recommend you use a failover policy and not a balanced policy for this purpose. After saving, run:

service mwan3 restart
service tailscale restart

A reload (service mwan3 reload) updates the nftables rules and the ip rules atomically via mwan3_delete_iface_rules (which finds and removes old rules by routing table id regardless of their previous priority), so old rules at the legacy 2000/3000 bases are correctly cleaned up.

After restart, verify:

# mwan3's fwmark rules at 1101-1160, tailscale's rules at 5210-5270
ip rule list

# The tailscale bypass rule appears in mwan3_rules
nft list chain inet mwan3 mwan3_rules | grep fwmark
# Expected: meta mark & 0x00ff0000 == 0x00080000 ... jump mwan3_policy_<name>

# Tailscale peer destinations are in mwan3_custom_v4
nft list set inet mwan3 mwan3_custom_v4

Forcing immediate failover of tailscale DERP connections

DERP connections that were open at the moment of WAN failover remain on the old WAN until they close naturally. Only sockets opened after tailscale reconnects pick up the new fwmark ip rule. They should failover quite quickly anyway, but to force immediate failover, put the following into /etc/mwan3.user

#!/bin/sh
[ "$ACTION" = "ifup" ] || exit 0

. /lib/functions.sh
. /lib/mwan3/common.sh
. /lib/mwan3/mwan3.sh
config_load mwan3
mwan3_init
mwan3_flush_marked_conntrack

README.md Unescape Escape

mwan3 nftables User and Developer Reference

mwan3 version: 3.6.12

Contents

1. Architecture Overview