cwx/op-packages
1
0
Fork 0
mirror of https://github.com/kiddin9/op-packages.git synced 2026-09-10 18:34:18 +08:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity
Files
8eaa48a2770c6184209b3efc999bfa9a389f05d5
op-packages/mwan3-nft/mwan3-nftables-internals.html
T

2749 lines
411 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>mwan3 nftables User and Developer Reference - Developer Reference</title>
<style>
:root {
--bg: #fdfdfd; --fg: #1a1a1a; --accent: #2563eb; --accent-light: #dbeafe;
--code-bg: #f3f4f6; --border: #d1d5db; --heading: #111827;
--note-bg: #fffbeb; --note-border: #f59e0b;
--warn-bg: #fef2f2; --warn-border: #ef4444;
--chain-bg: #ecfdf5; --chain-border: #10b981;
}
* { box-sizing: border-box; }
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto,
'Helvetica Neue', Arial, sans-serif;
color: var(--fg); background: var(--bg); max-width: 960px;
margin: 0 auto; padding: 2rem 1.5rem; line-height: 1.7; font-size: 15px;
}
h1 { font-size: 2rem; color: var(--heading); border-bottom: 3px solid var(--accent);
padding-bottom: .5rem; margin-top: 0; }
h2 { font-size: 1.5rem; color: var(--heading); border-bottom: 2px solid var(--border);
padding-bottom: .3rem; margin-top: 2.5rem; }
h3 { font-size: 1.2rem; color: var(--accent); margin-top: 2rem; }
h4 { font-size: 1.05rem; color: var(--heading); margin-top: 1.5rem; }
code {
font-family: 'SFMono-Regular', Consolas, 'Liberation Mono', Menlo, monospace;
font-size: 0.92em; background: var(--code-bg); padding: 0.15em 0.35em; border-radius: 3px;
}
pre {
background: var(--code-bg); border: 1px solid var(--border); border-radius: 6px;
padding: 1rem 1.2rem; overflow-x: auto; font-size: 0.88em; line-height: 1.55;
}
pre code { background: none; padding: 0; }
table { border-collapse: collapse; width: 100%; margin: 1rem 0; font-size: 0.93em; }
th, td { border: 1px solid var(--border); padding: 0.5rem 0.75rem;
text-align: left; vertical-align: top; }
th { background: var(--code-bg); font-weight: 600; }
tr:nth-child(even) td { background: #fafafa; }
.note, .warn, .chain-box {
border-left: 4px solid; padding: 0.75rem 1rem; margin: 1.2rem 0;
border-radius: 0 6px 6px 0;
}
.note { background: var(--note-bg); border-color: var(--note-border); }
.warn { background: var(--warn-bg); border-color: var(--warn-border); }
.chain-box { background: var(--chain-bg); border-color: var(--chain-border); }
.toc { background: var(--code-bg); border: 1px solid var(--border);
border-radius: 6px; padding: 1rem 1.5rem; margin: 1.5rem 0; }
.toc ul, .toc ol { margin: 0.3rem 0; padding-left: 1.5rem; }
.toc li { margin: 0.15rem 0; }
.toc a { text-decoration: none; color: var(--accent); }
.toc a:hover { text-decoration: underline; }
a { color: var(--accent); }
.file-path { font-weight: 600; color: var(--accent); }
.func-sig { font-family: monospace; font-size: 0.95em; font-weight: 600; }
.diagram { text-align: center; margin: 1.5rem 0; }
.diagram pre { display: inline-block; text-align: left; font-size: 0.82em; }
.tag { display: inline-block; font-size: 0.78em; padding: 0.1em 0.5em;
border-radius: 3px; font-weight: 600; vertical-align: middle; }
.tag-static { background: #dbeafe; color: #1e40af; }
.tag-dynamic { background: #fce7f3; color: #9d174d; }
.tag-unchanged { background: #e5e7eb; color: #374151; }
.tag-new { background: #dcfce7; color: #166534; }
hr { border: none; border-top: 1px solid var(--border); margin: 2.5rem 0; }
</style>
</head>
<body>
<h1 id="mwan3-nftables-user-and-developer-reference">mwan3 nftables User and Developer Reference</h1>
<h3 id="mwan3-version-369">mwan3 version: 3.6.9</h3>
<p>Covers the nftables port of the mwan3 multi-WAN policy routing framework.</p>
<hr>
<nav class="toc">
<strong>Contents</strong>
<ol>
<li><a href="#1-architecture-overview">Architecture Overview</a></li>
<li><a href="#2-the-mark-bitmask-system">The Mark Bitmask System</a></li>
<li><a href="#3-table-inet-mwan3-architecture">table inet mwan3 Architecture</a></li>
<li><a href="#4-packet-flow-through-chains">Packet Flow Through Chains</a></li>
<li><a href="#5-ip-rules-and-lookup-tables">IP Rules and Lookup Tables</a></li>
<ul>
<li>5.1 <a href="#51-per-interface-routing-tables">Per-Interface Routing Tables</a></li>
<li>5.2 <a href="#52-three-ip-rule-tiers-per-interface">Three ip Rule Tiers Per Interface</a></li>
<li>5.3 <a href="#53-global-policy-rules">Global Policy Rules</a></li>
<li>5.4 <a href="#54-mmxdefault-fall-through">MMX_DEFAULT Fall-Through</a></li>
<li>5.5 <a href="#55-configurable-rule-base-priorities">Configurable Rule Base Priorities</a></li>
<li>5.6 <a href="#56-example-live-ip-rule-output">Example: Live ip rule Output</a></li>
</ul>
<li><a href="#6-file-reference">File Reference</a></li>
<ul>
<li>6.1 <a href="#61-libmwan3mwan3-skeletonnft-static">mwan3-skeleton.nft</a></li>
<li>6.2 <a href="#62-libmwan3commonsh">common.sh</a></li>
<li>6.3 <a href="#63-libmwan3mwan3sh">mwan3.sh</a></li>
<li>6.4 <a href="#64-etcinitdmwan3">init.d/mwan3</a></li>
<li>6.5 <a href="#65-etchotplugdiface25-mwan3">25-mwan3 - hotplug</a></li>
<li>6.6 <a href="#66-usrsbinmwan3-cli">usr/sbin/mwan3 - CLI</a></li>
<li>6.7 <a href="#67-usrsbinmwan3rtmon">mwan3rtmon</a></li>
<li>6.8 <a href="#68-usrsharerpcducodemwan3">rpcd/ucode/mwan3</a></li>
<li>6.9 <a href="#69-makefile">Makefile</a></li>
<li>6.10 <a href="#610-usrsbinmwan3track">mwan3track</a></li>
<li>6.11 <a href="#611-usrsbinmwan3-lb-test">mwan3-lb-test</a></li>
<li>6.12 <a href="#612-usrsbinmwan3-diag">mwan3-diag</a></li>
<li>6.13 <a href="#613-usrsbinmwan3ct">mwan3ct</a></li>
<li>6.14 <a href="#614-libmwan3mwan3-get-addruc">mwan3-get-addr.uc</a></li>
<li>6.15 <a href="#615-libmwan3mwan3-manage-rulesuc">mwan3-manage-rules.uc</a></li>
<li>6.16 <a href="#616-libmwan3mwan3-list-routesuc">mwan3-list-routes.uc</a></li>
<li>6.17 <a href="#617-libmwan3mwan3-create-iface-routeuc">mwan3-create-iface-route.uc</a></li>
<li>6.18 <a href="#618-usrbinmwan3ipcheck">mwan3ipcheck</a></li>
</ul>
<li><a href="#7-function-reference">Function Reference</a></li>
<ul>
<li>7.1 <a href="#71-commonsh-functions">common.sh Functions</a></li>
<li>7.2 <a href="#72-set-management-functions">Set Management Functions</a></li>
<li>7.3 <a href="#73-general-rule-setup">General Rule Setup</a></li>
<li>7.4 <a href="#74-interface-management">Interface Management</a></li>
<li>7.5 <a href="#75-policy--load-balancing">Policy &amp; Load Balancing</a></li>
<li>7.6 <a href="#76-sticky-routing">Sticky Routing</a></li>
<li>7.7 <a href="#77-user-rules">User Rules</a></li>
<li>7.8 <a href="#78-user-defined-nft-set-management">User-defined nft Set Management</a></li>
<li>7.9 <a href="#79-status-reporting">Status Reporting</a></li>
<li>7.10 <a href="#710-lifecycle--hotplug">Lifecycle &amp; Hotplug</a></li>
</ul>
<li><a href="#8-load-balancing-with-numgen">Load Balancing with numgen</a></li>
<li><a href="#9-sticky-routing-detail">Sticky Routing Detail</a></li>
<li><a href="#10-service-lifecycle-and-conntrack-management">Service Lifecycle and Conntrack Management</a></li>
<ul>
<li>10.1 <a href="#101-start">Start</a></li>
<li>10.2 <a href="#102-reload">Reload</a></li>
<li>10.3 <a href="#103-interface-up-hotplug">Interface Up (hotplug)</a></li>
<li>10.4 <a href="#104-interface-down-hotplug">Interface Down (hotplug)</a></li>
<li>10.5 <a href="#105-stop">Stop</a></li>
<li>10.6 <a href="#106-conntrack-management">Conntrack Management</a></li>
</ul>
<li><a href="#11-atomic-non-destructive-reload">Atomic Non-destructive Reload</a></li>
<li><a href="#12-user-defined-nft-sets">User-defined nft Sets</a></li>
<li><a href="#13-unchanged-files">Unchanged Files</a></li>
<li><a href="#14-diagnostic-commands">Diagnostic Commands</a></li>
<li><a href="#15-luci-app-mwan3">luci-app-mwan3</a></li>
<ul>
<li>15.1 <a href="#151-network-app">Network App</a></li>
<ul>
<li>15.1.1 <a href="#1511-globals">Globals</a></li>
<li>15.1.2 <a href="#1512-interface">Interface</a></li>
<li>15.1.3 <a href="#1513-member">Member</a></li>
<li>15.1.4 <a href="#1514-policy">Policy</a></li>
<li>15.1.5 <a href="#1515-rule">Rule</a></li>
<li>15.1.6 <a href="#1516-ip-sets">IP Sets</a></li>
<li>15.1.7 <a href="#1517-simulator">Simulator</a></li>
<li>15.1.8 <a href="#1518-configuration">Configuration</a></li>
<li>15.1.9 <a href="#1519-notify">Notify</a></li>
</ul>
<li>15.2 <a href="#152-status-app">Status App</a></li>
<ul>
<li>15.2.1 <a href="#1521-overview">Overview</a></li>
<li>15.2.2 <a href="#1522-interface-status">Interface Status</a></li>
<li>15.2.3 <a href="#1523-routing-health">Routing Health</a></li>
<li>15.2.4 <a href="#1524-ip-sets-status">IP Sets (Status)</a></li>
<li>15.2.5 <a href="#1525-diagnostics">Diagnostics</a></li>
<li>15.2.6 <a href="#1526-troubleshooting">Troubleshooting</a></li>
</ul>
<li>15.3 <a href="#153-backend-helper-script-and-acl">Backend: Helper Script and ACL</a></li>
<li>15.4 <a href="#154-rpcd-methods">rpcd Methods</a></li>
</ul>
<li><a href="#16-iptables-to-nftables-porting-notes">Iptables-to-nftables Porting Notes</a></li>
<li><a href="#17-command-line-tools">Command Line Tools</a></li>
<ul>
<li>17.1 <a href="#171-mwan3-lb-test-load-balancing-distribution-verifier">mwan3-lb-test: Load Balancing Distribution Verifier</a></li>
<li>17.2 <a href="#172-mwan3-diag-network-diagnostic-report">mwan3-diag: Network Diagnostic Report</a></li>
</ul>
<li><a href="#18-changelog">Changelog</a></li>
<ul>
<li>18.1 <a href="#181-version-369">Version 3.6.9</a></li>
<li>18.2 <a href="#182-version-368">Version 3.6.8</a></li>
<li>18.3 <a href="#183-version-367">Version 3.6.7</a></li>
<li>18.4 <a href="#184-version-366">Version 3.6.6</a></li>
<li>18.5 <a href="#185-version-365">Version 3.6.5</a></li>
<li>18.6 <a href="#186-version-364">Version 3.6.4</a></li>
<li>18.7 <a href="#187-version-363">Version 3.6.3</a></li>
<li>18.8 <a href="#188-version-362">Version 3.6.2</a></li>
<li>18.9 <a href="#189-version-361">Version 3.6.1</a></li>
<li>18.10 <a href="#1810-version-36">Version 3.6</a></li>
<li>18.11 <a href="#1811-version-353">Version 3.5.3</a></li>
<li>18.12 <a href="#1812-version-352">Version 3.5.2</a></li>
<li>18.13 <a href="#1813-version-351">Version 3.5.1</a></li>
<li>18.14 <a href="#1814-version-35">Version 3.5</a></li>
<li>18.15 <a href="#1815-version-341-unreleased">Version 3.4.1 (Unreleased)</a></li>
<li>18.16 <a href="#1816-version-34">Version 3.4</a></li>
<li>18.17 <a href="#1817-version-335">Version 3.3.5</a></li>
<li>18.18 <a href="#1818-version-334">Version 3.3.4</a></li>
<li>18.19 <a href="#1819-version-333">Version 3.3.3</a></li>
<li>18.20 <a href="#1820-version-332">Version 3.3.2</a></li>
<li>18.21 <a href="#1821-version-331">Version 3.3.1</a></li>
<li>18.22 <a href="#1822-version-33">Version 3.3</a></li>
<li>18.23 <a href="#1823-version-323">Version 3.2.3</a></li>
<li>18.24 <a href="#1824-version-322">Version 3.2.2</a></li>
<li>18.25 <a href="#1825-version-321">Version 3.2.1</a></li>
<li>18.26 <a href="#1826-version-32">Version 3.2</a></li>
<li>18.27 <a href="#1827-version-314">Version 3.1.4</a></li>
<li>18.28 <a href="#1828-version-313">Version 3.1.3</a></li>
<li>18.29 <a href="#1829-version-312">Version 3.1.2</a></li>
<li>18.30 <a href="#1830-version-311">Version 3.1.1</a></li>
</ul>
<li><a href="#19-specific-use-cases">Specific use-cases</a></li>
<ul>
<li>19.1 <a href="#191-tailscale">Tailscale</a></li>
</ul>
</ol>
</nav>
<hr>
<h2 id="1-architecture-overview">1. Architecture Overview</h2>
<p>mwan3 is OpenWrt's multi-WAN policy routing framework. It classifies packets using <strong>firewall marks</strong>, then uses <code>ip rule</code> entries to route marked packets through per-interface routing tables. The nftables port replaces all iptables/ipset usage with nftables equivalents while keeping the ip rule/route management largely unchanged. <code>mwan3rtmon</code> is ported to a ucode implementation.</p>
<h3 id="key-design-decisions">Key Design Decisions</h3>
<ul>
<li><strong>Own standalone table</strong> - mwan3 lives in <code>table inet mwan3</code>, completely independent of <code>table inet fw4</code>. fw4 reload, restart, or reconfiguration has no effect on mwan3 rules.</li>
<li><strong>Static skeleton + dynamic rules</strong> - <code>mwan3-skeleton.nft</code> defines empty sets and chains loaded by <code>nft -f</code> at service start using the atomic table-replace idiom. All rules are added dynamically by shell scripts since they depend on the configurable <code>MMX_MASK</code>.</li>
<li><strong>Hook priority <code>mangle + 1</code></strong> - <code>mwan3_prerouting</code> and <code>mwan3_output</code> register at priority <code>-149</code>, placing them after fw4's mangle chains and any other packages registering at <code>-150</code>. Mark operations use masked OR-immediate setter chains via vmap-dispatch so they are non-destructive with respect to bits owned by other packages regardless of execution order.</li>
<li><strong>Non-destructive mark save/restore</strong> - Connmark save and restore are masked to mwan3's own bit-range (<code>MMX_MASK</code>) and never touch bits owned by other packages. The kernel rejects compound two-source bitwise expressions; mwan3 synthesises masked save/restore through a vmap-dispatch technique built from per-mark OR-immediate setter chains. See <a href="#connmark-operations">Section 2 - Connmark Operations</a>.</li>
<li><strong>inet family</strong> - Chains handle both IPv4 and IPv6 in a single pass. Sets remain type-specific (separate v4/v6 sets) since nftables requires a single address type per set.</li>
<li><strong>Atomic non-destructive reload</strong> - <code>reload_service</code> rebuilds the entire mwan3 ruleset in a single <code>nft -f</code> batch while the old ruleset serves traffic, committing atomically with zero downtime window. See <a href="#11-atomic-non-destructive-reload">Section 11</a>.</li>
<li><strong>User-defined nft sets</strong> - <code>config ipset</code> sections in <code>/etc/config/mwan3</code> create named nft sets in <code>table inet mwan3</code> supporting inline entries, file loading, and dnsmasq domain population. See <a href="#12-user-defined-nft-sets">Section 12</a>.</li>
</ul>
<h3 id="component-map">Component Map</h3>
<div class="diagram"><pre> UCI Config (/etc/config/mwan3)
|
+---------------------------------+---------------------------------+
| | |
init.d/mwan3 mwan3track mwan3rtmon
(service lifecycle) (health probes) (route replication)
| | |
+--------------------+--------------------+ | |
| | | | |
v v v | |
common.sh mwan3.sh mwan3-skeleton.nft | |
(helpers) (engine) (static skeleton) | |
| | | | |
v v | | |
mwan3-get-addr.uc mwan3ct | | |
| mwan3-manage-rules.uc | | |
| mwan3-list-routes.uc | | |
| mwan3-create-iface-route.uc | | |
| | | | |
v v v v v
nft / netlink netlink / conntrack nft writes STATUS files nft / netlink
to /var/run/mwan3track
Hotplug: 25-mwan3 --calls--&gt; mwan3.sh functions
Hotplug user: 26-mwan3-user --calls--&gt; /etc/mwan3.user
CLI: /usr/sbin/mwan3 --calls--&gt; mwan3.sh functions
RPC: rpcd/ucode/mwan3 --calls--&gt; nft -j (JSON output)
dnsmasq: mwan3_write_dnsmasq_fragments --writes--&gt; confdir nftset fragments
mwan3rtmon: --uses--&gt; ucode-mod-rtnl (netlink) + nft
mwan3ct: --uses--&gt; libnetfilter_conntrack + libmnl
mwan3.sh: --calls--&gt; mwan3ct (conntrack flush)
mwan3.sh: --calls--&gt; mwan3-manage-rules.uc (ip rule management)
mwan3.sh: --calls--&gt; mwan3-list-routes.uc (route enumeration)
mwan3.sh: --calls--&gt; mwan3-create-iface-route.uc (route install)
mwan3-get-addr.uc: --uses--&gt; ucode-mod-rtnl (netlink)
mwan3-manage-rules.uc: --uses--&gt; ucode-mod-rtnl (netlink)
mwan3-list-routes.uc: --uses--&gt; ucode-mod-rtnl (netlink)
mwan3-create-iface-route.uc: --uses--&gt; ucode-mod-rtnl (netlink)</pre></div>
<hr>
<h2 id="2-the-mark-bitmask-system">2. The Mark Bitmask System</h2>
<p>mwan3 uses a configurable bitmask (<code>MMX_MASK</code>, default <code>0x3F00</code>) within the 32-bit packet mark to encode routing decisions. The mask determines how many interfaces can be supported and which mark values are reserved.</p>
<h3 id="mark-layout-default-0x3f00">Mark Layout (default 0x3F00)</h3>
<table>
<thead><tr><th>Bits</th><th>Mask</th><th>Owner</th><th>Purpose</th></tr></thead><tbody>
<tr><td>0-7</td><td><code>0x000000FF</code></td><td>free</td><td>-</td></tr>
<tr><td>8-13</td><td><code>0x00003F00</code></td><td>mwan3</td><td>interface/policy marks</td></tr>
<tr><td>14-15</td><td><code>0x0000C000</code></td><td>free</td><td>-</td></tr>
<tr><td>16-23</td><td><code>0x00FF0000</code></td><td>pbr</td><td>policy routing marks</td></tr>
<tr><td>24-31</td><td><code>0xFF000000</code></td><td>free</td><td>-</td></tr>
</tbody></table>
<table>
<thead><tr><th>Value</th><th>Meaning</th><th>With 0x3F00</th></tr></thead><tbody>
<tr><td>0</td><td>Unmarked (needs classification)</td><td><code>0x0000</code></td></tr>
<tr><td>1 .. N</td><td>Interface marks (N = max interfaces)</td><td><code>0x0100</code> .. depends on mask</td></tr>
<tr><td>mmdefault-2</td><td>MM_BLACKHOLE</td><td>Routes to blackhole</td></tr>
<tr><td>mmdefault-1</td><td>MM_UNREACHABLE</td><td>Routes to unreachable</td></tr>
<tr><td>mmdefault (all bits set)</td><td>MMX_DEFAULT (= MMX_MASK)</td><td><code>0x3F00</code> = use default routing</td></tr>
</tbody></table>
<h3 id="bit-spreading-mwan3id2mask">Bit Spreading: <code>mwan3_id2mask()</code></h3>
<p>Interface IDs (sequential integers 1, 2, 3...) are mapped onto the mask bits using <code>mwan3_id2mask()</code>. This "spreads" the ID's bits into only the positions where the mask has a 1-bit. For example, with mask <code>0x3F00</code>:</p>
<pre><code>Interface 1 (binary 000001) -&gt; 0x0100 (bit 8 set)
Interface 2 (binary 000010) -&gt; 0x0200 (bit 9 set)
Interface 3 (binary 000011) -&gt; 0x0300 (bits 8+9)
Interface 5 (binary 000101) -&gt; 0x0500 (bits 8+10)</code></pre>
<h3 id="nftables-mark-manipulation">nftables Mark Manipulation</h3>
<p>The iptables operation <code>-j MARK --set-xmark VALUE/MASK</code> means <code>mark = (mark &amp; ~MASK) | VALUE</code>. In nftables this becomes:</p>
<pre><code>meta mark set meta mark &amp; COMPLEMENT | VALUE</code></pre>
<p>where <code>COMPLEMENT = ~MASK &amp; 0xFFFFFFFF</code>. The helper <code>mwan3_nft_mark_expr()</code> generates this expression.</p>
<div class="warn">
<p><strong>Operator syntax:</strong> Always use the <code>&amp;</code> and <code>|</code> <em>symbols</em>, not the <code>and</code>/<code>or</code> keywords. The nft parser treats keywords ambiguously after expressions like <code>meta mark set ct mark</code> - it cannot tell if <code>and</code> starts a new match or a bitwise operation. Symbols are unambiguous.</p>
</div>
<h3 id="connmark-operations">Connmark Operations</h3>
<p>mwan3's connmark save and restore are scoped to its own bit-range (<code>MMX_MASK</code>) so that mark bits owned by other packages are never disturbed in either direction.</p>
<p>The natural nftables expression for a masked restore would be:</p>
<pre><code>meta mark set (meta mark &amp; ~MMX_MASK) | (ct mark &amp; MMX_MASK)</code></pre>
<p>The kernel rejects this with "Operation not supported": an nft set-statement can reference at most one runtime source register on its right-hand side.</p>
<h4 id="vmap-dispatch-saverestore">vmap-dispatch save/restore</h4>
<p>mwan3 synthesises the masked-restore and masked-save from primitives the kernel does allow: OR-ing a <em>literal immediate</em> into a single register (restore), or masking and OR-ing in a single expression (save).</p>
<pre><code>meta mark set meta mark | &lt;constant&gt; # restore: only sets bits, never clears
ct mark set ct mark &amp; MMX_MASK_COMPLEMENT | &lt;constant&gt; # save: atomic clear+set in one expression</code></pre>
<p>The runtime source value is bridged to the constant immediate via a verdict map (<code>vmap</code>) that dispatches on the masked source bits into per-mark setter chains:</p>
<pre><code># Restore: copy mwan3 bits from ct mark into meta mark, non-destructively
meta mark &amp; MMX_MASK == 0 ct mark &amp; MMX_MASK vmap {
0x0100 : jump mwan3_or_meta_0x100,
0x0200 : jump mwan3_or_meta_0x200,
...
0x3f00 : jump mwan3_or_meta_0x3f00
}
chain mwan3_or_meta_0x0100 { meta mark set meta mark | 0x0100 ; return }
chain mwan3_or_meta_0x0200 { meta mark set meta mark | 0x0200 ; return }
...
# Save: atomic clear+set via vmap dispatch
meta mark &amp; MMX_MASK vmap {
0x0100 : jump mwan3_or_ct_0x0100,
...
}
chain mwan3_or_ct_0x0100 { ct mark set ct mark &amp; MMX_MASK_COMPLEMENT | 0x0100 ; return }
...</code></pre>
<p><strong>Properties:</strong></p>
<ul>
<li><strong>Restore</strong> is purely additive (<code>meta mark | &lt;imm&gt;</code>); bits in meta mark not owned by mwan3 are preserved across restore.</li>
<li><strong>Save</strong> atomically clears mwan3's own bits and sets the new value in a single nft expression (<code>ct mark &amp; MMX_MASK_COMPLEMENT | &lt;imm&gt;</code>, where <code>MMX_MASK_COMPLEMENT = ~MMX_MASK &amp; 0xFFFFFFFF</code>). The ct mark goes directly from old-value to new-value with no intermediate state visible to other CPUs. Bits owned by other packages survive the save unchanged.</li>
<li>The dispatch tables are bounded: with the default <code>MMX_MASK = 0x3F00</code>, there are 63 possible non-zero mwan3 mark values, so 63 setter chains per direction (126 total). All chains are 2-statement skeletons (a mark-set expression + <code>return</code>).</li>
<li>The technique is <em>order-independent</em>: whether mwan3's hook fires before or after another package's hook in the prerouting stack, both packages' mark bits arrive at the routing decision intact.</li>
</ul>
<p><strong>Why this works where a direct expression does not:</strong> the kernel constraint is on the expression, not the control flow. The kernel will not let one rule combine two register sources, but it will happily let a vmap dispatch on a runtime register value into a chain whose body uses a literal immediate. The dispatch chain materialises at runtime exactly the value you wanted to OR, baked into a constant when the mwan3 init scripts emit the rule.</p>
<p><strong>Cost:</strong> chain count, not packet-path overhead. Each packet traverses one extra <code>vmap</code> lookup (O(log) in the kernel's set lookup) and one extra <code>jump</code>/<code>return</code> per save and per restore. The 126 setter chains add no fast-path cost - they are visited via constant-time dispatch, and most never fire on any given packet.</p>
<div class="note">
<p>Compound two-source bitwise set expressions are rejected by the kernel. vmap-dispatch is built from primitives that are permitted and does not require any new kernel capability.</p>
</div>
<hr>
<h2 id="3-table-inet-mwan3-architecture">3. table inet mwan3 Architecture</h2>
<p>mwan3 operates in its own standalone nftables table, <code>table inet mwan3</code>. This table is completely independent of fw4: fw4 reloads, restarts, and reconfiguration do not affect it. The table is created at service start from <code>mwan3-skeleton.nft</code> using the atomic table-replace idiom, and torn down at service stop.</p>
<h3 id="static-objects-from-mwan3-skeletonnft">Static Objects (from mwan3-skeleton.nft)</h3>
<table>
<thead><tr><th>Object</th><th>Type</th><th>Purpose</th></tr></thead><tbody>
<tr><td><code>mwan3_connected_v4</code></td><td>set (ipv4_addr, interval, auto-merge)</td><td>Directly connected IPv4 networks</td></tr>
<tr><td><code>mwan3_connected_v6</code></td><td>set (ipv6_addr, interval, auto-merge)</td><td>Directly connected IPv6 networks</td></tr>
<tr><td><code>mwan3_custom_v4</code></td><td>set (ipv4_addr, interval, auto-merge)</td><td>Networks from routing tables in UCI <code>globals.rt_table_lookup</code></td></tr>
<tr><td><code>mwan3_custom_v6</code></td><td>set (ipv6_addr, interval, auto-merge)</td><td>Networks from routing tables in UCI <code>globals.rt_table_lookup</code></td></tr>
<tr><td><code>mwan3_dynamic_v4</code></td><td>set (ipv4_addr, interval, auto-merge)</td><td>IPv4 CIDRs from UCI <code>globals.bypass_network</code></td></tr>
<tr><td><code>mwan3_dynamic_v6</code></td><td>set (ipv6_addr, interval, auto-merge)</td><td>IPv6 CIDRs from UCI <code>globals.bypass_network</code></td></tr>
<tr><td><code>mwan3_prerouting</code></td><td>chain (filter, prerouting, mangle+1)</td><td>Entry point for forwarded/incoming traffic</td></tr>
<tr><td><code>mwan3_output</code></td><td>chain (route, output, mangle+1)</td><td>Entry point for locally-originated traffic</td></tr>
<tr><td><code>mwan3_postrouting</code></td><td>chain (nat, postrouting, srcnat-1)</td><td>Opt-in IPv6 SNAT for router-originated rerouted traffic - see <a href="#router-originated-traffic-and-source-address-rewriting">§4</a></td></tr>
<tr><td><code>mwan3_ifaces_in</code></td><td>chain (regular)</td><td>Dispatches to per-interface chains</td></tr>
<tr><td><code>mwan3_rules</code></td><td>chain (regular)</td><td>User-defined classification rules</td></tr>
<tr><td><code>mwan3_connected</code></td><td>chain (regular)</td><td>Marks traffic to connected networks as default</td></tr>
<tr><td><code>mwan3_custom</code></td><td>chain (regular)</td><td>Marks traffic to custom-table networks as default</td></tr>
<tr><td><code>mwan3_dynamic</code></td><td>chain (regular)</td><td>Marks traffic to dynamic networks as default</td></tr>
</tbody></table>
<div class="note">
<p><strong>auto-merge flag:</strong> All sets include the <code>auto-merge</code> flag in addition to <code>interval</code>. This allows nftables to merge overlapping elements (e.g., a host address and a containing CIDR) automatically, preventing insertion failures. The <code>nft add set</code> command is idempotent for creation but does <em>not</em> update flags on existing sets - <code>mwan3_ensure_nft_framework()</code> deletes and recreates the six internal sets at startup to guarantee the flag is present.</p>
</div>
<h3 id="dynamic-objects-created-at-runtime-in-table-inet-mwan3">Dynamic Objects (created at runtime, in table inet mwan3)</h3>
<table>
<thead><tr><th>Object Pattern</th><th>Type</th><th>Created By</th></tr></thead><tbody>
<tr><td><code>mwan3_iface_in_&lt;name&gt;</code></td><td>chain</td><td><code>mwan3_create_iface_nft()</code></td></tr>
<tr><td><code>mwan3_policy_&lt;name&gt;</code></td><td>chain</td><td><code>mwan3_create_policies_nft()</code></td></tr>
<tr><td><code>mwan3_rule_&lt;name&gt;</code></td><td>chain</td><td><code>mwan3_set_user_nft_rule()</code> (sticky rules only)</td></tr>
<tr><td><code>mwan3_or_meta_&lt;mark&gt;</code></td><td>chain (×63 with default <code>MMX_MASK</code>)</td><td><code>mwan3_build_or_chains_nft()</code> - non-destructive restore setter chains</td></tr>
<tr><td><code>mwan3_or_ct_&lt;mark&gt;</code></td><td>chain (×63 with default <code>MMX_MASK</code>)</td><td><code>mwan3_build_or_chains_nft()</code> - non-destructive save setter chains</td></tr>
<tr><td><code>mwan3_sticky_v4_&lt;rule&gt;_&lt;id&gt;</code></td><td>set (ipv4_addr, timeout)</td><td><code>mwan3_set_user_nft_rule()</code> - one set per policy member (id = interface id)</td></tr>
<tr><td><code>mwan3_sticky_v6_&lt;rule&gt;_&lt;id&gt;</code></td><td>set (ipv6_addr, timeout)</td><td><code>mwan3_set_user_nft_rule()</code> - one set per policy member (id = interface id)</td></tr>
<tr><td><code>&lt;name&gt;</code> (user-defined)</td><td>set (ipv4_addr or ipv6_addr, interval, auto-merge)</td><td><code>mwan3_render_config_ipsets()</code> from <code>config ipset</code> UCI sections</td></tr>
</tbody></table>
<div class="note">
<p><strong>Why <code>type route</code> for output?</strong> The output chain uses <code>type route</code> (not <code>type filter</code>) because changing a packet's mark on locally-originated traffic must trigger a routing re-lookup. This matches fw4's own <code>mangle_output</code> chain type.</p>
</div>
<div class="note">
<p><strong><code>fw4 reload</code> is a non-event for mwan3.</strong> <code>fw4 reload</code> rewrites only <code>table inet fw4</code>. <code>table inet mwan3</code> is untouched. No recovery, detection, or rebuild is needed on <code>fw4 reload</code>.</p>
</div>
<hr>
<h2 id="4-packet-flow-through-chains">4. Packet Flow Through Chains</h2>
<p>The same logical flow applies to both <code>mwan3_prerouting</code> and <code>mwan3_output</code>, with one difference: prerouting includes an IPv6 RA bypass at the top.</p>
<pre><code>Packet enters mwan3_prerouting (or mwan3_output)
|
|-- [prerouting only] ICMPv6 RA/NS/NA/redirect? --&gt; ACCEPT (bypass)
|
|-- mark &amp; MMX_MASK == 0?
| |
| +-- YES: Restore mwan3 bits from conntrack via vmap-dispatch
| | (ct mark &amp; MMX_MASK -&gt; jump mwan3_or_meta_&lt;mark&gt;;
| | non-destructive - preserves non-mwan3 bits in meta mark)
| |
| +-- Still mark == 0?
| |
| +-- jump mwan3_ifaces_in
| | Per-interface chains check source address:
| | - src in connected/custom/dynamic? -&gt; mark = MMX_DEFAULT
| | - otherwise -&gt; mark = interface mark
| |
| +-- [prerouting only] Still mark == 0? fib daddr type local? --&gt; RETURN
| |
| +-- Still mark == 0?
| | jump mwan3_custom (dst in custom sets? -&gt; MMX_DEFAULT)
| | jump mwan3_connected (dst in connected? -&gt; MMX_DEFAULT)
| | jump mwan3_dynamic (dst in dynamic? -&gt; MMX_DEFAULT)
| |
| +-- Still mark == 0?
| jump mwan3_rules (user classification rules)
| -&gt; jump to policy chain / set mark directly
|
|-- Save mwan3 bits to conntrack via vmap-dispatch:
| meta mark &amp; MMX_MASK -&gt; jump mwan3_or_ct_&lt;mark&gt;
| (atomic clear+set in one expression; preserves non-mwan3 bits in ct mark)
|
|-- mark &amp; MMX_MASK != MMX_DEFAULT?
| (Traffic that got a specific interface mark, not "default")
| Re-check against custom/connected/dynamic destinations
| This allows connected-destination traffic to be overridden
| back to default routing even if it was marked by user rules
|
+-- ACCEPT (policy accept; packet continues to routing decision)</code></pre>
<h3 id="the-three-bypass-set-groups">The Three Bypass Set Groups</h3>
<p>mwan3 maintains three parallel groups of destination-bypass sets. Each group has a v4 and v6 set, a corresponding regular chain (jumped from both <code>mwan3_prerouting</code> and <code>mwan3_output</code>), and matching rules in every <code>mwan3_iface_in_*</code> chain. They differ only in how their sets are populated:</p>
<table>
<thead><tr><th>Set group</th><th>Populated by</th><th>Source</th></tr></thead><tbody>
<tr><td><code>mwan3_connected_v4/v6</code></td><td><code>mwan3rtmon</code> (continuous) and <code>mwan3_set_connected_sets()</code> at startup</td><td>CIDR routes in the kernel's main routing table</td></tr>
<tr><td><code>mwan3_custom_v4/v6</code></td><td><code>mwan3_set_custom_sets()</code> at startup</td><td>Prefixes from routing tables listed in UCI <code>globals.rt_table_lookup</code></td></tr>
<tr><td><code>mwan3_dynamic_v4/v6</code></td><td><code>mwan3_set_dynamic_sets()</code> at startup; also writable at runtime via <code>nft add element</code></td><td>UCI <code>globals.bypass_network</code> CIDR list</td></tr>
</tbody></table>
<p>Each group appears in <strong>two distinct contexts</strong> with different match directions:</p>
<p><strong>Destination match - <code>mwan3_connected</code> / <code>mwan3_custom</code> / <code>mwan3_dynamic</code> chains:</strong> These chains are jumped from <code>mwan3_prerouting</code> and <code>mwan3_output</code> before the user rules chain. Rules match <code>ip daddr @set</code> / <code>ip6 daddr @set</code>. If the packet's destination falls in one of the sets, the packet is stamped <code>MMX_DEFAULT</code> (mwan3 mark bits cleared to zero) and returned - the user rules chain is never reached. Traffic destined for directly-connected, custom-table, or explicitly listed bypass networks is not policy-routed.</p>
<p><strong>Source match - <code>mwan3_iface_in_*</code> chains:</strong> Each per-interface chain includes source-match rules against all three set groups (<code>ip saddr @set</code>, scoped to the interface's address family). If an inbound WAN packet's source address is in any of the sets, it is stamped <code>MMX_DEFAULT</code>. Traffic arriving on a WAN interface from a connected or bypassed address does not get the WAN's interface mark, so replies to it use the main routing table rather than the WAN-specific policy route.</p>
<h3 id="per-interface-chain-detail">Per-Interface Chain Detail</h3>
<p>Each <code>mwan3_iface_in_&lt;name&gt;</code> chain handles packets arriving on a specific WAN device:</p>
<ol>
<li><strong><code>iifname</code> and <code>meta nfproto</code> match</strong> - only processes packets arriving on the correct physical device and address family. The <code>meta nfproto ipv4</code>/<code>meta nfproto ipv6</code> guard is critical when two mwan3 interfaces share the same physical device (e.g. dual-stack PPPoE): without it, an IPv4 interface's catchall would misclassify incoming IPv6 packets.</li>
</ol>
<ol>
<li><strong>Source in bypass sets → <code>MMX_DEFAULT</code></strong> - if the arriving packet's source is in <code>mwan3_connected_v4/v6</code>, <code>mwan3_custom_v4/v6</code>, or <code>mwan3_dynamic_v4/v6</code>, the packet is marked <code>MMX_DEFAULT</code>. This connection will use the main routing table for replies rather than being pinned to this WAN's policy route.</li>
</ol>
<ol>
<li><strong>Otherwise → interface mark</strong> - the packet is stamped with the interface's unique fwmark. The conntrack save step in the calling prerouting/output chain then records this mark so subsequent packets in the same connection have their mark restored from ct mark.</li>
</ol>
<ol>
<li><strong>Address-family-scoped catchall</strong> - the rule that marks unmatched packets with the interface fwmark carries a <code>meta nfproto</code> guard matching the interface's configured family, completing the dual-stack isolation begun in step 1.</li>
</ol>
<h3 id="router-originated-traffic-and-source-address-rewriting">Router-Originated Traffic and Source Address Rewriting</h3>
<p>When the router itself originates a packet (via <code>mwan3_output</code>), the kernel binds the source address at <code>sendto()</code> time using the initial, unmarked route lookup. mwan3's mark is not set at that point, so the kernel picks the source address corresponding to whichever WAN the unmarked default route points to - call it WAN-A. Later in the egress path, <code>mwan3_output</code> sets a mark and the kernel performs a re-lookup (because the chain is <code>type route</code>) that may move the outgoing interface to WAN-B. The re-lookup updates <code>oif</code> but does not rewrite the source address - that was already committed. The packet leaves WAN-B carrying WAN-A's source address and is dropped upstream by BCP38 or uRPF filtering.</p>
<p>mwan3track is unaffected by this problem: it sets <code>SO_BINDTODEVICE</code> at socket creation, which forces the correct source address at bind time before any of this occurs.</p>
<p><strong>IPv4:</strong> fw4's <code>srcnat_wan</code> masquerade rule applies to all outgoing traffic including locally-originated packets. When a rerouted packet reaches the <code>srcnat</code> hook, masquerade picks the primary IP of the actual outgoing interface and rewrites the source address correctly. No per-interface SNAT rule is needed from mwan3 - fw4 handles it automatically.</p>
<p><strong>IPv6:</strong> fw4 does not masquerade IPv6 by default, so there is no automatic safety net. mwan3 provides an opt-in per-interface UCI option <code>snat6</code> to address this. Blanket NAT66 is deliberately not the default for several reasons: RFC 6724 source-address selection with SADR routing tables can solve the problem without NAT in correctly-configured dual-stack deployments; NAT66 is harmful in ULA+delegated-PA topologies and breaks address-embedding protocols; some upstreams (tunnel brokers, fixed-address WireGuard endpoints) require a specific source address; and the IPv6 community treats address translation as an explicit opt-in.</p>
<p>The <code>snat6</code> option accepts three values:</p>
<table>
<thead><tr><th>Value</th><th>Meaning</th></tr></thead><tbody>
<tr><td>unset / <code>0</code></td><td>No IPv6 SNAT (default)</td></tr>
<tr><td><code>1</code></td><td>SNAT to the interface's primary global address, looked up via <code>mwan3_get_src_ip</code></td></tr>
<tr><td><code>&lt;v6 addr&gt;</code></td><td>SNAT to the literal address - used for fixed-source pinning from a delegated prefix</td></tr>
</tbody></table>
<p>When <code>snat6</code> is set, mwan3 installs a rule in the <code>mwan3_postrouting</code> base chain:</p>
<pre><code>oifname "&lt;dev&gt;" meta nfproto ipv6
meta mark &amp; MMX_MASK == &lt;iface_mark&gt;
fib saddr type local
ip6 saddr != &lt;iface_src_ip&gt;
snat to &lt;iface_src_ip&gt;</code></pre>
<p>The <code>fib saddr type local</code> guard limits the rule to router-originated traffic. Stale rules are cleaned up by tag (<code>mwan3_snat_&lt;iface&gt;</code>) in <code>mwan3_create_iface_nft()</code> and <code>mwan3_delete_iface_nft()</code>. The literal-address form is not validated against the egress interface - some deployments deliberately pin a source from a delegated prefix not directly configured on the device.</p>
<p><code>snat6</code> only addresses the router-originated rerouted case. It does not extend mwan3's IPv6 capability to forwarded LAN traffic, SADR integration, or NPTv6 prefix translation.</p>
<p>The LuCI control for <code>snat6</code> is described in <a href="#1512-interface">Section 15.1.2</a>.</p>
<hr>
<h2 id="5-ip-rules-and-lookup-tables">5. IP Rules and Lookup Tables</h2>
<p>mwan3 uses Linux policy routing as the actual packet-steering mechanism. The nftables chains (Section 4) write a firewall mark onto each packet; the kernel's <code>ip rule</code> database then selects the right routing table based on that mark. The nftables side and the policy-routing side are independent: nftables writes marks, the kernel routes according to them.</p>
<h3 id="51-per-interface-routing-tables">5.1 Per-Interface Routing Tables</h3>
<p>Each mwan3 <code>config interface</code> section is assigned a sequential integer ID by <code>mwan3_update_iface_to_table()</code>. Interface sections are enumerated in UCI declaration order; the first enabled interface is ID 1, the second is ID 2, and so on. The ID is both the routing table number and the input to <code>mwan3_id2mask()</code> that generates the interface's mark value (see Section 2). Routing tables 1 through <code>MWAN3_INTERFACE_MAX</code> (derived from <code>MMX_MASK</code>) are reserved for mwan3 and should not be used for other purposes.</p>
<p>Each per-interface routing table is built by <code>mwan3_create_iface_route()</code> from two sources:</p>
<ul>
<li>CIDR routes from the main routing table (host routes are excluded)</li>
<li>Routes from any additional tables listed in UCI <code>globals.rt_table_lookup</code></li>
</ul>
<p>The <code>MWAN3_ROUTE_LINE_EXP</code> sed expression strips transient attributes (<code>linkdown</code>, <code>offload</code>, expiry timestamps, <code>error</code> codes) before copying routes. Routes whose device matches the mwan3 interface's network device keep their original <code>dev</code> clause; the interface's default gateway (<code>default via ... dev ...</code>) is the critical entry that directs all policy-routed traffic to the correct WAN path.</p>
<h3 id="52-three-ip-rule-tiers-per-interface">5.2 Three ip Rule Tiers Per Interface</h3>
<p><code>mwan3_create_iface_rules()</code> installs three <code>ip rule</code> entries for each mwan3 interface. For IPv4 interfaces these are installed via <code>ip rule</code>; for IPv6 via <code>ip -6 rule</code>. All three tiers are installed together when the interface comes up (<code>ifup</code> hotplug event) and removed together when it goes down (<code>ifdown</code>).</p>
<p><strong>Tier 1 - iif lookup</strong> (priority <code>id + iif_rule_base</code>, default <code>id + 1000</code>):</p>
<pre><code>ip rule add pref &lt;id+1000&gt; iif &lt;device&gt; lookup &lt;id&gt;</code></pre>
<p>Matches packets arriving on this interface's network device, regardless of mark. Routes them by looking up the interface's own routing table. This handles traffic that arrived on a WAN interface and needs to exit the router via the same WAN - the interface's table contains the WAN's default gateway. Without this rule, reply packets from the router (such as responses to health-check probes from the WAN's remote host) would reach the routing decision without a mark, fall through to the main table, and potentially exit via a different WAN.</p>
<p><strong>Tier 2 - fwmark lookup</strong> (priority <code>id + fwmark_rule_base</code>, default <code>id + 2000</code>):</p>
<pre><code>ip rule add pref &lt;id+2000&gt; fwmark &lt;mark&gt;/&lt;MMX_MASK&gt; lookup &lt;id&gt;</code></pre>
<p>The primary steering rule. Matches packets whose mark, after masking with <code>MMX_MASK</code>, equals this interface's mark value (computed by <code>mwan3_id2mask(id, MMX_MASK)</code>). Routes them through the interface's per-interface routing table, which contains the WAN's default gateway. This is the rule that causes policy-classified traffic to exit via the correct WAN.</p>
<p><strong>Tier 3 - fwmark unreachable</strong> (priority <code>id + unreachable_rule_base</code>, default <code>id + 3000</code>):</p>
<pre><code>ip rule add pref &lt;id+3000&gt; fwmark &lt;mark&gt;/&lt;MMX_MASK&gt; unreachable</code></pre>
<p>Matches the same fwmark as Tier 2 but returns ICMP unreachable instead of routing. Sits below Tier 2 in priority (higher priority number = lower precedence). Serves as a safety net: if the Tier 2 lookup finds an empty or incomplete routing table (e.g. the default route is transiently absent during route installation), the packet receives an explicit unreachable response instead of silently falling through to the main routing table and potentially leaking out a different WAN.</p>
<h3 id="53-global-policy-rules">5.3 Global Policy Rules</h3>
<p><code>mwan3_set_general_rules()</code> installs two rules that are not tied to any specific interface. These handle the mark values reserved for policy <code>last_resort</code> actions:</p>
<p><strong>Blackhole rule</strong> (priority <code>MM_BLACKHOLE + fwmark_rule_base</code>):</p>
<pre><code>ip rule add pref &lt;priority&gt; fwmark &lt;MMX_BLACKHOLE&gt;/&lt;MMX_MASK&gt; blackhole</code></pre>
<p>Silently drops packets marked with <code>MMX_BLACKHOLE</code>. Applied when a policy chain marks traffic with this value via a <code>last_resort blackhole</code> policy option.</p>
<p><strong>Unreachable rule</strong> (priority <code>MM_UNREACHABLE + fwmark_rule_base</code>):</p>
<pre><code>ip rule add pref &lt;priority&gt; fwmark &lt;MMX_UNREACHABLE&gt;/&lt;MMX_MASK&gt; unreachable</code></pre>
<p>Returns ICMP unreachable for packets marked with <code>MMX_UNREACHABLE</code>. This is the default <code>last_resort</code> for policies with no available members.</p>
<p>With <code>MMX_MASK = 0x3F00</code> (6 mask bits): <code>MM_BLACKHOLE = 61</code>, priority <code>61 + 2000 = 2061</code>; <code>MM_UNREACHABLE = 62</code>, priority <code>62 + 2000 = 2062</code>. Both fall within the fwmark tier, above all per-interface Tier 2 entries.</p>
<h3 id="54-mmxdefault-fall-through">5.4 MMX_DEFAULT Fall-Through</h3>
<p>Packets marked with <code>MMX_DEFAULT</code> (= <code>MMX_MASK</code>, all mask bits set) do not match any per-interface fwmark rule or the global blackhole/unreachable rules. They fall through to the kernel's standard policy database, which includes the <code>main</code> routing table at priority 32766. Traffic stamped <code>MMX_DEFAULT</code> is routed normally via the main table without any mwan3 policy steering. This is the intended path for traffic destined for directly-connected, custom-table, or dynamically bypassed networks (see the bypass set groups in Section 4).</p>
<h3 id="55-configurable-rule-base-priorities">5.5 Configurable Rule Base Priorities</h3>
<p>The three rule-base offsets are configurable in UCI <code>config globals</code>:</p>
<table>
<thead><tr><th>UCI option</th><th>Default</th><th>Priority range used (60 interfaces, default MMX_MASK)</th></tr></thead><tbody>
<tr><td><code>iif_rule_base</code></td><td>1000</td><td>1001 - 1060 (one per interface)</td></tr>
<tr><td><code>fwmark_rule_base</code></td><td>2000</td><td>2001 - 2062 (per-interface + 2 global)</td></tr>
<tr><td><code>unreachable_rule_base</code></td><td>3000</td><td>3001 - 3060 (one per interface)</td></tr>
</tbody></table>
<p>Two ordering constraints must hold at startup:</p>
<pre><code>iif_rule_base + MWAN3_INTERFACE_MAX &lt; fwmark_rule_base
fwmark_rule_base + MWAN3_INTERFACE_MAX + 1 &lt; unreachable_rule_base</code></pre>
<p>The <code>+ 1</code> in the second constraint accounts for the global blackhole and unreachable entries at the top of the fwmark tier (<code>MM_BLACKHOLE + fwmark_rule_base</code> and <code>MM_UNREACHABLE + fwmark_rule_base</code>). If either constraint is violated, <code>mwan3_init</code> logs a warning and reverts all three to 1000/2000/3000.</p>
<p><code>mwan3_delete_face_rules()</code> uses content-based matching to find and delete rules: it scans <code>ip rule list</code> for entries referencing <code>lookup &lt;id&gt;</code> to identify iif rules, and derives the fwmark value from the fwmark lookup rule before deleting the matching unreachable rule. This approach remains correct if the bases or <code>MMX_MASK</code> are changed between service restarts.</p>
<h3 id="56-example-live-ip-rule-output">5.6 Example: Live ip rule Output</h3>
<pre><code>0: from all lookup local
1001: from all iif eth1 lookup 1
1002: from all iif eth2 lookup 2
2001: from all fwmark 0x100/0x3f00 lookup 1
2002: from all fwmark 0x200/0x3f00 lookup 2
2061: from all fwmark 0x3d00/0x3f00 blackhole
2062: from all fwmark 0x3e00/0x3f00 unreachable
3001: from all fwmark 0x100/0x3f00 unreachable
3002: from all fwmark 0x200/0x3f00 unreachable
32766: from all lookup main
32767: from all lookup default</code></pre>
<p>The corresponding per-interface routing table (<code>ip route show table 1</code>) contains the WAN gateway and a copy of connected routes:</p>
<pre><code>default via 192.0.2.1 dev eth1
192.0.2.0/24 dev eth1 proto kernel scope link src 192.0.2.2
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1</code></pre>
<p>The priority gap between 2002 and 2061 is intentional: per-interface Tier 2 entries occupy priorities 2001-2060 (with the default 60-interface capacity) and the global entries occupy 2061-2062 at the top of the fwmark tier.</p>
<hr>
<h2 id="6-file-reference">6. File Reference</h2>
<h3 id="61-libmwan3mwan3-skeletonnft">6.1 <span class="file-path">lib/mwan3/mwan3-skeleton.nft</span> <span class="tag tag-static">static</span></h3>
<p>The static nftables skeleton file. Loaded by <code>start_service()</code> via <code>nft -f /lib/mwan3/mwan3-skeleton.nft</code> before any dynamic rule installation. Uses the canonical atomic table-replace idiom:</p>
<pre><code>table inet mwan3
delete table inet mwan3
table inet mwan3 { ... }</code></pre>
<p>The three statements execute as one atomic <code>nft -f</code> transaction: the first ensures the table exists so the delete can succeed, the second wipes it, and the third recreates it fresh. This is idempotent: safe to run whether the table already exists or not.</p>
<p>Defines 6 named sets (all empty, <code>flags interval</code> and <code>auto-merge</code>) and 8 skeleton chains (all empty). No rules are present - all rules are added dynamically because they depend on the configurable <code>MMX_MASK</code> value.</p>
<p>The hook chains:</p>
<ul>
<li><code>mwan3_prerouting</code> - type <code>filter</code> at priority <code>mangle + 1</code></li>
<li><code>mwan3_output</code> - type <code>route</code> at priority <code>mangle + 1</code> (<code>type route</code> is required so mark mutations trigger a routing re-lookup for locally-originated traffic)</li>
<li><code>mwan3_postrouting</code> - type <code>nat</code> at priority <code>srcnat - 1</code>. Opt-in IPv6 SNAT chain. See <a href="#router-originated-traffic-and-source-address-rewriting">§4</a>.</li>
</ul>
<h3 id="62-libmwan3commonsh">6.2 <span class="file-path">lib/mwan3/common.sh</span></h3>
<p>Shared helper library sourced by all mwan3 shell scripts. Provides:</p>
<ul>
<li><strong>Tool variables</strong>: <code>$IP4</code>, <code>$IP6</code>, <code>$NFT</code></li>
<li><strong>IPv6 detection</strong>: Checks <code>/proc/sys/net/ipv6</code> existence (instead of the old <code>command -v ip6tables</code>)</li>
<li><strong>MWAN3_BATCH_DEPTH counter</strong>: Integer depth counter (default 0). <code>mwan3_nft_batch_start</code> increments it, truncating the batch file only at depth 0-&gt;1. <code>mwan3_nft_batch_commit</code> decrements it, committing to the kernel only at depth 1-&gt;0. <code>mwan3_nft_exec</code> routes to <code>mwan3_nft_push</code> when depth &gt; 0, accumulating all operations into the global batch.</li>
<li><strong>MWAN3_NEED_DNSMASQ_HUP flag</strong>: integer flag that gets set to 1 if a user nft set has been deleted and recreated as a result of a change to one of the flags. After <code>mwan3_write_dnsmasq_fragments</code>, if the flag is set then a call will be made to <code>mwan3_dnsmasq_hup()</code>.</li>
<li><strong>Batch file</strong>: Per-process temp file <code>/tmp/mwan3_nft_batch.$$</code> (PID-scoped, avoids conflicts between concurrent mwan3 instances).</li>
<li><strong>nft batch helpers</strong>: <code>mwan3_nft_batch_start()</code>, <code>mwan3_nft_push()</code>, <code>mwan3_nft_batch_commit()</code></li>
<li><strong><code>mwan3_nft_reload_start()</code></strong>: Opens the outermost batch level (depth 0-&gt;1), then writes a preamble that (1) flushes all 8 skeleton chains, (2) two-pass flush+delete all dynamic chains (<code>mwan3_iface_in_*</code>, <code>mwan3_policy_*</code>, <code>mwan3_rule_*</code>, <code>mwan3_or_meta_*</code>, <code>mwan3_or_ct_*</code>), (3) deletes the 6 internal mwan3 sets so <code>mwan3_ensure_nft_framework</code> can recreate them. User-defined sets and sticky sets are never in the delete path.</li>
<li><strong><code>mwan3_nft_reload_commit()</code></strong>: Thin wrapper around <code>mwan3_nft_batch_commit</code>. At depth 1-&gt;0 this commits the entire accumulated batch atomically. On failure the kernel rolls back and the old ruleset continues serving.</li>
<li><strong><code>mwan3_nft_exec()</code></strong>: Wrapper that runs <code>nft</code> commands with error logging; routes to <code>mwan3_nft_push</code> when <code>MWAN3_BATCH_DEPTH &gt; 0</code>.</li>
<li><strong><code>mwan3_nft_mark_expr()</code></strong>: Generates nftables mark-set expressions equivalent to iptables <code>--set-xmark</code>. Outputs <code>meta mark set meta mark &amp; COMPLEMENT | VALUE</code> using <code>&amp;</code>/<code>|</code> symbols (not <code>and</code>/<code>or</code> keywords).</li>
<li><strong><code>mwan3_ensure_nft_framework()</code></strong>: Guarantees all mwan3 nftables objects exist with correct flags. When called inside a batch (<code>MWAN3_BATCH_DEPTH &gt; 0</code>), skips the direct delete loop (deletes already in the preamble batch). Recreates all 6 internal sets with <code>interval</code> + <code>auto-merge</code> flags and all skeleton chains in <code>table inet mwan3</code>.</li>
<li><strong><code>mwan3_or_chain_suffix()</code></strong>: Converts a numeric mark value to the canonical lowercase <code>0x%x</code> hex string used as the suffix for OR-immediate setter chain names (e.g. <code>0x100</code> for interface 1 with default mask). Called by <code>mwan3_build_or_chains_nft()</code>, <code>mwan3_or_vmap_body()</code>, <code>mwan3_all_marks()</code>, and <code>mwan3_create_policies_nft()</code>.</li>
<li><strong><code>mwan3_or_vmap_body()</code></strong>: Builds the body string for a vmap statement dispatching on masked mark values into OR-immediate setter chains. Called by <code>mwan3_set_general_nft()</code>.</li>
<li><strong><code>mwan3_all_marks()</code></strong>: Enumerates every mark value that needs to appear in the restore/save vmaps: all per-interface marks plus <code>MMX_DEFAULT</code>, <code>MMX_BLACKHOLE</code>, and <code>MMX_UNREACHABLE</code>. Echoes a space-separated list. Used by <code>mwan3_set_general_nft()</code>.</li>
<li><strong><code>mwan3_build_or_chains_nft()</code></strong>: Materialises the per-mark setter chains used by the non-destructive vmap-dispatch save/restore. Iterates all 63 possible non-zero values within <code>MMX_MASK</code> and emits two chains per value: <code>mwan3_or_meta_&lt;imm&gt;</code> (non-destructive restore from ct mark to meta mark) and <code>mwan3_or_ct_&lt;imm&gt;</code> (atomic clear+set save from meta mark to ct mark, non-destructive to bits outside MMX_MASK). Called from <code>mwan3_set_general_nft()</code>. Always flushes and re-populates the chain bodies - an earlier idempotency check that returned early on chain <em>existence</em> alone could leave the chain bodies empty after a partial-failure first run, which is fatal to packet flow.</li>
<li><strong><code>mwan3_init()</code></strong>: Loads config, computes mask constants (<code>MMX_DEFAULT</code>, <code>MMX_BLACKHOLE</code>, <code>MMX_UNREACHABLE</code>, <code>MMX_MASK_COMPLEMENT</code>)</li>
<li><strong><code>mwan3_id2mask()</code></strong>: Bit-spreading function that maps interface IDs onto the mask</li>
<li><strong><code>mwan3_count_one_bits()</code></strong>: Counts set bits in a value</li>
<li><strong>Utility functions</strong>: <code>LOG()</code>, <code>readfile()</code>, <code>mwan3_get_src_ip()</code>, <code>mwan3_get_true_iface()</code>, <code>mwan3_get_mwan3track_status()</code>, <code>get_uptime()</code>, <code>get_online_time()</code></li>
</ul>
<div class="note">
<p><strong>Shell scoping note:</strong> Functions like <code>mwan3_id2mask</code> and <code>mwan3_count_one_bits</code> receive <em>variable names</em> as arguments (e.g., <code>mwan3_id2mask mmdefault MMX_MASK</code>) and use arithmetic expansion <code>$(($1))</code> to resolve them. This works in busybox ash (OpenWrt's default shell) because it uses dynamic scoping - local variables from the caller are visible in called functions.</p>
</div>
<h3 id="63-libmwan3mwan3sh">6.3 <span class="file-path">lib/mwan3/mwan3.sh</span></h3>
<p>The core engine. Contains all functions for managing nftables chains/sets/maps, ip rules, ip routes, policy creation, user rule classification, and status reporting. This is the largest file and the heart of the implementation. Sourced by init.d, hotplug, CLI, and rtmon scripts.</p>
<p>See <a href="#7-function-reference">Section 7</a> for detailed function reference.</p>
<h3 id="64-etcinitdmwan3">6.4 <span class="file-path">etc/init.d/mwan3</span></h3>
<p>procd service script. Handles:</p>
<ul>
<li><strong><code>start_service()</code></strong>: Loads <code>mwan3-skeleton.nft</code> via <code>nft -f</code> first (aborts if this fails), then runs the full init sequence: ensure framework, render ipsets, dnsmasq fragments, trackers, sets, general rules, ifup hotplug loop, nft chains, policies, user rules, conntrack flush (if flow offloading), dnsmasq HUP, rtmons.</li>
<li><strong><code>stop_service()</code></strong>: Guards with <code>service_running || exit 0</code>. Shuts down interfaces, flushes ip routes/rules, flushes all mwan3 nft chains, deletes dynamic chains (keeps skeleton chains), flushes sets, flushes and deletes sticky maps, then <code>$NFT delete table inet mwan3</code> (complete removal).</li>
<li><strong><code>reload_service()</code></strong>: Atomically rebuilds the entire mwan3 ruleset in a single kernel transaction via <code>mwan3_nft_reload_start</code> / all build functions / <code>mwan3_nft_reload_commit</code>. Then updates ip rules/routing tables (outside nft), updates dnsmasq fragments if changed, and checks tracker count - falls through to <code>stop; start</code> if tracker count mismatches (procd cannot add/remove service instances in reload).</li>
<li><strong><code>start_tracker()</code></strong>: Launches a <code>mwan3track</code> procd instance per enabled interface with track IPs or <code>track_gateway</code>.</li>
<li><strong><code>service_running()</code></strong>: Returns true if <code>$MWAN3_STATUS_DIR</code> exists.</li>
</ul>
<h4 id="startup-sequence">Startup Sequence</h4>
<pre><code>nft -f /lib/mwan3/mwan3-skeleton.nft # load standalone table (abort if fails)
mwan3_init() # load UCI config, compute mask constants
mwan3_ensure_nft_framework() # recreate 6 internal sets, ensure chains
mwan3_render_config_ipsets() # create user-defined sets from config ipset
mwan3_write_dnsmasq_fragments() # write nftset confdir fragments; restart dnsmasq if changed
config_foreach start_tracker interface # launch health probes
mwan3_update_iface_to_table() # build iface-&gt;table mapping
mwan3_set_dynamic_sets() # populate dynamic sets
mwan3_set_connected_sets() # populate connected sets
mwan3_set_custom_sets() # populate custom sets
mwan3_set_general_rules() # ip rule add (blackhole/unreachable)
config_foreach mwan3_ifup interface "init" # trigger ifup hotplug per interface
wait $hotplug_pids
mwan3_set_general_nft() # populate hook chain rules
mwan3_set_policies_nft() # create policy chains
mwan3_set_user_rules() # populate user rules chain
mwan3_flush_stale_conntrack() # flush zero-mark conntrack entries
mwan3_dnsmasq_hup() # HUP dnsmasq to re-populate nftset domains
[if flow_offloading=1] flush conntrack # force flow re-establishment under new policy
start rtmon_ipv4 + rtmon_ipv6 # route monitor daemons</code></pre>
<h3 id="65-etchotplugdiface25-mwan3">6.5 <span class="file-path">etc/hotplug.d/iface/25-mwan3</span></h3>
<p>Handles interface state change events from netifd. Triggered on <code>ifup</code>, <code>ifdown</code>, <code>connected</code>, and <code>disconnected</code> actions.</p>
<h4 id="guard-checks">Guard Checks</h4>
<ol>
<li>Valid action and interface name</li>
<li>Not first-connect or shutdown</li>
<li>Device present for ifup/connected</li>
<li>procd lock (unless called from init)</li>
<li>Service is running (<code>$MWAN3_STATUS_DIR</code> exists)</li>
<li>nft framework is loaded (<code>nft list chain inet mwan3 mwan3_prerouting</code> succeeds)</li>
<li>Interface is enabled in UCI</li>
</ol>
<p>There is no <code>fw4 reload</code> detection. <code>table inet mwan3</code> is unaffected by fw4 reloads; no recovery path is needed.</p>
<h4 id="actions">Actions</h4>
<table>
<thead><tr><th>Action</th><th>Operations</th></tr></thead><tbody>
<tr><td class="func-sig"><code>ifup</code></td><td>Update peer track IP (if <code>track_gateway</code> enabled), create interface nft chain, create ip rules, set hotplug state, create routes, set general rules (if not init), rebuild policies (if online and not init). Signal tracker with USR2.</td></tr>
<tr><td class="func-sig"><code>ifdown</code></td><td>Set offline state, delete map entries, delete ip rules, delete routes, delete interface nft chain. Signal tracker with USR1. Rebuild policies.</td></tr>
<tr><td class="func-sig"><code>connected</code></td><td>Set online state, create interface nft chain, rebuild policies.</td></tr>
<tr><td class="func-sig"><code>disconnected</code></td><td>Set offline state, rebuild policies.</td></tr>
</tbody></table>
<p>All actions call <code>mwan3_flush_conntrack</code> at end.</p>
<div class="note">
<p><strong>ifup conditional policy rebuild:</strong> During init (<code>MWAN3_STARTUP=init</code>), the ifup action skips general rules and policy rebuild because the init sequence handles those after all interfaces are up. Route creation runs unconditionally on every ifup, including during init. During normal operation, policies are only rebuilt if the interface state is "online" (not for interfaces with <code>initial_state=offline</code>).</p>
</div>
<h3 id="66-usrsbinmwan3-cli">6.6 <span class="file-path">usr/sbin/mwan3</span> (CLI)</h3>
<p>User-facing command-line tool. Provides <code>start</code>/<code>stop</code>/<code>restart</code>/<code>ifup</code>/<code>ifdown</code> commands plus status reporting: <code>interfaces</code>, <code>policies</code>, <code>connected</code>, <code>rules</code>, <code>status</code> (all combined), and <code>internal</code> (detailed dump).</p>
<p>The <code>use</code> command runs an arbitrary command bound to a specific interface using <code>LD_PRELOAD</code> with <code>libwrap_mwan3_sockopt.so</code>.</p>
<p>The <code>internal</code> command shows <code>nft list table inet mwan3</code> output instead of the old iptables dump.</p>
<h3 id="67-usrsbinmwan3rtmon">6.7 <span class="file-path">usr/sbin/mwan3rtmon</span></h3>
<p>Route monitor daemon, reimplemented in <strong>ucode</strong>. Runs one instance per address family (ipv4/ipv6) as a procd service. Uses <code>ucode-mod-rtnl</code> for direct netlink access and <code>ucode-mod-uloop</code> for the event loop, eliminating all <code>ip</code> command fork+exec overhead from the original shell implementation.</p>
<p>Key improvements over the shell version:</p>
<ul>
<li><strong>Direct netlink route monitoring</strong> via <code>rtnl.listener()</code> instead of <code>ip monitor route</code> piped to a shell read loop</li>
<li><strong>Structured route data</strong> from netlink messages instead of text parsing with sed/awk</li>
<li><strong>O(1) per-event cost</strong> - the refactored handler avoids per-interface ubus calls, popen subprocesses, and UCI cursor creation on each route event. Device-to-table mapping and interface state are cached and refreshed only when needed</li>
<li><strong>Debounced connected set rebuild</strong> - route add and delete events trigger a 100ms debounce timer rather than an immediate full set rebuild, coalescing bursts of route changes (e.g., during interface flap) into a single rebuild. Add events for the connected set additionally check to see if any change has occurred.</li>
<li><strong>Proper event loop</strong> via <code>uloop.run()</code> instead of a shell pipe+read loop</li>
</ul>
<p>On startup, it performs an initial synchronization: dumps the current routing table via netlink, populates the connected set, and replicates routes into active per-interface tables. It then enters the uloop event loop to process route change notifications asynchronously.</p>
<ul>
<li><strong>New route</strong>: Adds CIDR networks to the connected set via <code>nft add element</code>, then replicates the route into active per-interface tables. Host routes (bare IPs without prefix length) are skipped as they are remote destinations. Schedules via the debouncer.</li>
<li><strong>Deleted route</strong>: Schedules a debounced connected set rebuild, then removes the route from per-interface tables.</li>
</ul>
<h3 id="68-usrsharerpcducodemwan3">6.8 <span class="file-path">usr/share/rpcd/ucode/mwan3</span></h3>
<p>ucode RPC service exposing ubus methods under the <code>mwan3</code> object. Used by LuCI for the web interface.</p>
<p>Uses <code>nft -j</code> (JSON output mode) for reliable parsing. All set/chain queries are scoped to <code>table inet mwan3</code>. Rule and route queries use <code>ucode-mod-rtnl</code> (RTM_GETRULE, RTM_GETROUTE) directly rather than spawning <code>ip -j</code> subprocesses.</p>
<ul>
<li><strong><code>mwan3.status</code></strong>: Returns JSON data for interfaces, connected networks, and policies.</li>
<ul>
<li><strong>Connected IPs</strong>: Parses <code>nft -j list set inet mwan3 mwan3_connected_v4/v6</code>.</li>
<li><strong>Policies</strong>: Reads membership from UCI config and cross-references mwan3track <code>STATUS</code> files. Every member is always reported with traffic share percentage.</li>
<li><strong>Interfaces</strong>: Reads status from <code>/var/run/mwan3track/</code> files and queries procd/netifd via ubus. Tracking IPs discovered by globbing <code>TRACK_*</code> files. Per-IP <code>latency</code> and <code>packetloss</code> populated when <code>check_quality=1</code>.</li>
</ul>
<li><strong><code>mwan3.nftset_members { set: "&lt;name&gt;" }</code></strong>: Returns members of a named nft set in <code>table inet mwan3</code> as a flat array of strings. Correctly unwraps counter-decorated elements (<code>{"elem":{"val":"...","counter":{...}}}</code>) that nft emits for sets with the <code>counter</code> flag. Used by the Simulator tab for nftset rule matching and connected-network bypass detection.</li>
<li><strong><code>mwan3.nftset_info {}</code></strong>: Returns the name, address-family type, counters flag, and runtime element count of all non-mwan3 sets in <code>table inet mwan3</code>. Used by the rule editor and IP Sets status tab.</li>
<li><strong><code>mwan3.nftset_elements { set: "&lt;name&gt;", max: N }</code></strong>: Returns paginated elements of a named user-defined set with per-element packet/byte counters (when the set has <code>counter</code> enabled). Default maximum 200 elements; supports up to 5000. Used by the IP Sets status tab for runtime member display.</li>
<li><strong><code>mwan3.resolve_host { host: "&lt;name&gt;", family: "" }</code></strong>: Resolves a hostname to IP addresses by invoking <code>/bin/busybox nslookup</code> against the local DNS server (127.0.0.1). Returns <code>{ v4: [...], v6: [...] }</code>. The <code>family</code> parameter can be set to <code>ipv4</code> or <code>ipv6</code> to restrict resolution to A or AAAA records respectively; omit or leave empty for both. Querying via the local DNS server has the side effect of populating any dnsmasq nftset entries configured for the domain. Used by the Simulator tab to support hostname input in the source and destination fields.</li>
<li><strong><code>mwan3.nftset_flush { set: "&lt;name&gt;" }</code></strong>: Flushes all elements from the named set via <code>nft flush set inet mwan3 &lt;name&gt;</code>. Returns <code>{}</code> on success or <code>{ error: "..." }</code> on failure. Used by the Flush button on the IP Sets status tab.</li>
<li><strong><code>mwan3.nftset_reload { set: "&lt;name&gt;" }</code></strong>: Flushes the set then re-adds all static entries from the UCI <code>config ipset</code> section (<code>list entry</code> values and the <code>loadfile</code> if configured). Returns <code>{}</code> on success or <code>{ error: "..." }</code> on failure. Used by the Reload button on the IP Sets status tab.</li>
<li><strong><code>mwan3.nftset_resolve { set: "&lt;name&gt;" }</code></strong>: Sends SIGHUP to dnsmasq to clear its cache, then queries each domain configured under the set's <code>list domain</code> option via the local dnsmasq instance (127.0.0.1). The DNS queries trigger dnsmasq's <code>nftset=</code> population mechanism as a side effect. Returns <code>{ resolved: N }</code> where N is the count of domains that resolved successfully. Only applicable for sets with <code>list domain</code> entries. Used by the Resolve button on the IP Sets status tab.</li>
<li><strong><code>mwan3.routing_health {}</code></strong>: Compares UCI configuration against live kernel state. Reads <code>iif_rule_base</code>, <code>fwmark_rule_base</code>, and <code>unreachable_rule_base</code> from UCI globals (with the same defaults and ordering constraint validation as <code>mwan3_init</code>) and derives <code>iface_max</code> dynamically from <code>mmx_mask</code>. Per interface, checks presence of the iif rule, fwmark lookup rule, and unreachable rule, and reports routing table default route presence. Reports stale ip rules across all three priority tiers. Returns a <code>rule_bases</code> object so the frontend can display configured priorities dynamically rather than assuming fixed offsets.</li>
</ul>
<h3 id="69-makefile">6.9 <span class="file-path">Makefile</span></h3>
<p>Package build recipe. Key dependency changes:</p>
<table>
<thead><tr><th>Old Dependency</th><th>New Dependency</th></tr></thead><tbody>
<tr><td class="func-sig"><code>+ip</code></td><td><code>+ip-full</code></td></tr>
<tr><td class="func-sig"><code>+ipset</code></td><td><code>+kmod-nft-core</code></td></tr>
<tr><td class="func-sig"><code>+iptables</code></td><td><code>+nftables-json</code></td></tr>
<tr><td class="func-sig"><code>+IPV6:ip6tables</code></td><td><code>+ucode</code></td></tr>
<tr><td class="func-sig"><code>+iptables-mod-conntrack-extra</code></td><td><code>+ucode-mod-rtnl</code></td></tr>
<tr><td class="func-sig"><code>+iptables-mod-ipopt</code></td><td><code>+ucode-mod-uloop</code></td></tr>
<tr><td class="func-sig"></td><td><code>+ucode-mod-uci</code></td></tr>
<tr><td class="func-sig"></td><td><code>+ucode-mod-ubus</code></td></tr>
<tr><td class="func-sig"></td><td><code>+ucode-mod-fs</code></td></tr>
<tr><td class="func-sig"></td><td><code>+ucode-mod-socket</code></td></tr>
<tr><td class="func-sig"></td><td><code>+libnetfilter-conntrack</code></td></tr>
</tbody></table>
<p><code>PKG_BUILD_DEPENDS</code> adds <code>libnetfilter_conntrack</code> and <code>libmnl</code> for the headers and libraries needed to compile <code>mwan3ct.c</code>. The runtime <code>+libnetfilter-conntrack</code> dependency provides the shared library that <code>mwan3ct</code> links against on the target.</p>
<p><code>mwan3ipcheck.c</code> is also compiled in the same build step. It links only against libc (<code>inet_pton</code>, <code>strtol</code>, standard string functions) and requires no additional build or runtime dependencies.</p>
<p>The ucode dependencies are required by the reimplemented <code>mwan3rtmon</code> route monitor daemon. <code>ucode-mod-socket</code> is required by <code>mwan3-diag</code> for address normalisation.</p>
<p>Also installs:</p>
<ul>
<li><code>mwan3-skeleton.nft</code> to <code>$(1)/lib/mwan3/</code></li>
<li><code>mwan3-migrate-ipset-v4.sh</code> to <code>$(1)/lib/mwan3/</code> (one-shot migration helper, deleted from the router after postinst runs it)</li>
<li><code>mwan3-remove-firewall-include</code> UCI defaults to <code>$(1)/etc/uci-defaults/</code> (removes legacy <code>firewall.mwan3_reload</code> UCI section from v3.x)</li>
<li><code>mwan3-lb-test</code> to <code>$(1)/usr/sbin/</code></li>
<li><code>mwan3ct</code> to <code>$(1)/usr/sbin/</code></li>
<li><code>mwan3ipcheck</code> to <code>$(1)/usr/bin/</code></li>
</ul>
<p>The <code>preinst</code> script stops mwan3 before APK replaces any files. This handles the upgrade case where procd watches <code>/etc/init.d/</code> via inotify and auto-starts mwan3 when the init script is replaced.</p>
<p>The <code>postinst</code> script:</p>
<ol>
<li>Removes any mwan3 chains and sets still in <code>table inet fw4</code> (from v3.x upgrades)</li>
<li>Conditionally reloads fw4 only if the legacy <code>firewall.mwan3_reload</code> UCI section exists (v3.x only; avoids triggering queued hotplug events on a clean v3.5 install)</li>
<li>Runs <code>mwan3-migrate-ipset-v4.sh</code> to copy <code>config ipset</code> sections from <code>/etc/config/firewall</code> to <code>/etc/config/mwan3</code>, then removes the migration script</li>
<li>Restarts rpcd</li>
<li>Stops mwan3 a second time (by this point procd's auto-start has completed, so this stop removes auto-start's ip rules)</li>
<li>Starts mwan3 cleanly</li>
</ol>
<h3 id="610-usrsbinmwan3track">6.10 <span class="file-path">usr/sbin/mwan3track</span></h3>
<p>Interface health probe daemon. One procd service instance is launched per enabled mwan3 interface that has tracking IPs configured. Runs as a shell script; largely unchanged from the iptables version except for the addition of <code>track_gateway</code> and <code>check_quality</code> support.</p>
<h4 id="probe-methods">Probe methods</h4>
<p>Configured via the <code>track_method</code> UCI option. Supported values: <code>ping</code> (default), <code>arping</code>, <code>httping</code>, <code>nping-tcp</code>/<code>nping-udp</code>/etc., <code>nslookup</code>. All probes are wrapped via <code>LD_PRELOAD</code> with <code>libwrap_mwan3_sockopt.so</code> (the <code>WRAP</code> helper), which intercepts <code>setsockopt</code> to set <code>SO_BINDTODEVICE</code> on the probe socket. This binds the probe to the physical interface device regardless of mwan3 routing marks, ensuring the probe exits on the correct WAN.</p>
<h4 id="score-based-hysteresis">Score-based hysteresis</h4>
<p>mwan3track maintains a score counter for each interface:</p>
<ul>
<li>Each probe round: if <code>host_up_count &gt;= reliability</code> (enough IPs responded), score increments; otherwise score decrements.</li>
<li>When score reaches <code>up</code> threshold from below: fires <code>connected</code> hotplug event.</li>
<li>When score reaches <code>up</code> threshold from above (on decline): fires <code>disconnecting</code> then <code>disconnected</code> hotplug events.</li>
<li>Score is clamped between 0 and <code>down + up</code>.</li>
</ul>
<p>Once the reliability threshold is met in a round, remaining unprobed IPs are marked <code>skipped</code> - they are not probed further that round.</p>
<h4 id="status-files">Status files</h4>
<p>Written to <code>$MWAN3TRACK_STATUS_DIR/&lt;iface&gt;/</code> (default <code>/var/run/mwan3track/&lt;iface&gt;/</code>):</p>
<table>
<thead><tr><th>File</th><th>Content</th></tr></thead><tbody>
<tr><td class="func-sig"><code>STATUS</code></td><td>Current state: <code>online</code>, <code>offline</code>, <code>connecting</code>, <code>disconnecting</code>, <code>disabled</code></td></tr>
<tr><td class="func-sig"><code>SCORE</code></td><td>Current score counter</td></tr>
<tr><td class="func-sig"><code>TURN</code></td><td>Number of probe rounds completed</td></tr>
<tr><td class="func-sig"><code>LOST</code>, <code>ONLINE</code>, <code>OFFLINE</code>, <code>TIME</code></td><td>Loss count, uptime timestamps</td></tr>
<tr><td class="func-sig"><code>TRACK_&lt;ip&gt;</code></td><td>Per-IP probe result: <code>up</code>, <code>down</code>, or <code>skipped</code>. Always a status string, regardless of <code>check_quality</code>.</td></tr>
<tr><td class="func-sig"><code>LATENCY_&lt;ip&gt;</code></td><td>[check_quality=1 only] Latency in ms for this IP from the most recent probe round.</td></tr>
<tr><td class="func-sig"><code>LOSS_&lt;ip&gt;</code></td><td>[check_quality=1 only] Packet loss as a percentage for this IP from the most recent probe round.</td></tr>
<tr><td class="func-sig"><code>GATEWAY</code></td><td>Gateway IP written by <code>mwan3_update_peer_track_ip()</code> when <code>track_gateway=1</code>. Read by <code>mwan3_load_track_ips()</code> and prepended to the probe list.</td></tr>
</tbody></table>
<h4 id="checkquality">check_quality</h4>
<p>When <code>check_quality=1</code>, probes capture latency (ms) and packet loss (%) per IP. Three-state evaluation: fail (<code>loss &gt;= failure_loss</code> OR <code>latency &gt;= failure_latency</code>), pass (<code>loss &lt;= recovery_loss</code> AND <code>latency &lt;= recovery_latency</code>), grey zone (neither - probe result neither increments nor decrements score). Default thresholds (1000ms/500ms/40%/10%) are conservative, suited for monitoring without frequent false failovers.</p>
<h4 id="signal-handling">Signal handling</h4>
<p>procd sends <code>SIGUSR1</code> (ifdown event) and <code>SIGUSR2</code> (ifup event) to trigger immediate state transitions without waiting for the next probe interval.</p>
<h3 id="611-usrsbinmwan3-lb-test">6.11 <span class="file-path">usr/sbin/mwan3-lb-test</span></h3>
<p>Load balancing distribution verifier.</p>
<pre><code>mwan3-lb-test [-6] -c &lt;client_ip&gt; &lt;policy_name&gt; [ip1 ip2 ...]
mwan3-lb-test cleanup</code></pre>
<p>Verifies that numgen-based load balancing produces the expected traffic distribution across policy members. Key design:</p>
<ul>
<li><strong>Iteration count</strong> (<code>NITER</code>): computed from member weights as <code>base_N = total_weight / GCD(weights)</code>, <code>NITER = base_N * ceil(30 / base_N)</code>. Ensures per-member expected counts are whole numbers and <code>NITER &gt;= 30</code> always.</li>
<li><strong>Test rule</strong>: inserts a temporary ICMP-only rule into <code>mwan3_rules</code> matching a nft address set. Using ICMP prevents TCP/UDP traffic to the same IPs (DNS forwarders, Android clients bypassing local DNS, etc.) from contaminating the counter.</li>
<li><strong>Client isolation</strong> (<code>-c &lt;client_ip&gt;</code>, required): inserts a <code>forward</code> chain drop rule blocking pings to the test set from all LAN clients except the nominated test client, and an <code>mwan3_output</code> return rule bypassing mwan3 marking for any router process pinging the same IPs. Both rules are scoped to the test set and removed by cleanup.</li>
<li><strong>Destination pool</strong>: well-known public IPs. Excludes any IPs already configured as mwan3 <code>track_ip</code> values - mwan3track pings those via <code>mwan3_output -&gt; mwan3_rules</code>, which would match the test rule and inflate counts.</li>
<li><strong>IPv6 mode</strong> (<code>-6</code>): uses <code>meta l4proto ipv6-icmp ip6 daddr @set</code>, <code>ping6</code>, and a separate pool of well-known public IPv6 IPs.</li>
<li><strong>IP overrides</strong>: <code>mwan3-lb-test -c &lt;client_ip&gt; &lt;policy&gt; ip1 ip2 ...</code> for sites where defaults are unreachable or fully tracked.</li>
<li><strong>Windows command</strong>: outputs a <code>cmd.exe</code> <code>for</code> loop alongside the Linux shell loop. Windows <code>ping</code> uses a fixed ICMP identifier (id=1), causing conntrack entry reuse on repeated pings to the same destination; the Windows command uses a longer inter-ping delay (<code>30/TRACK_COUNT + 3</code> seconds) so the full cycle exceeds the 30s ICMP conntrack timeout. The IP list is formatted with <code>^</code> line continuation at 4 IPs per line.</li>
<li><strong><code>cleanup</code> subcommand</strong>: removes stale <code>mwan3_lb_test_*</code> sets and rules left by a run that was killed before cleanup could execute.</li>
<li><strong>Cleanup on exit</strong>: removes test set and rules on normal exit, SIGINT, SIGTERM, and SIGPIPE. Startup sweep removes stale artifacts from aborted prior runs.</li>
</ul>
<p>See <a href="#171-mwan3-lb-test-load-balancing-distribution-verifier">§17.1</a> for context on why this tool was added and the numgen contamination issues it was designed to detect.</p>
<hr>
<h3 id="612-usrsbinmwan3-diag">6.12 <span class="file-path">usr/sbin/mwan3-diag</span></h3>
<p>Network diagnostic report generator.</p>
<pre><code>mwan3-diag</code></pre>
<p>A ucode script that collects a comprehensive snapshot of the network state relevant to mwan3 operation and prints it to stdout. Intended to produce a complete, shareable report without manual redaction.</p>
<p>Collects: interface addresses, routing tables (including all per-WAN tables), policy rules, neighbour cache, mwan3 interface status, mwan3 UCI configuration, the complete mwan3 nftables ruleset, the fw4 mangle chains that interact with mwan3 packet marking, and the last 200 lines of the mwan3 log.</p>
<p>Before printing any output the script builds a map of every public routable IPv4 and IPv6 address present in the collected data and replaces each one with a stable placeholder -- <code>PUB4_1</code>, <code>PUB4_2</code>, <code>PUB6_1</code> and so on -- throughout the entire report. The same address always receives the same placeholder, so cross-references between sections remain consistent. Private addresses (RFC1918, link-local <code>fe80::</code>, ULA <code>fc00::/7</code>, loopback) are left unchanged. The elements of user-defined nftables sets are replaced with <code>{ ... }</code>.</p>
<p>See <a href="#172-mwan3-diag-network-diagnostic-report">§17.2</a> for context.</p>
<hr>
<h3 id="613-usrsbinmwan3ct">6.13 <span class="file-path">usr/sbin/mwan3ct</span></h3>
<p>Targeted conntrack entry flush helper.</p>
<p>mwan3 needs to flush conntrack entries at multiple places in the code. These flushes are designed to be as targeted as possible in order to leave other unrelated conntrack entries unmolested.</p>
<p>In some cases, the conntrack CLI tool is unable to perform the necessary targeted flush. For example, mwan3 needs to flush only UNREPLIED conntrack entries when an interface comes online to break the cycle where a stale mark pins traffic to the wrong WAN. <code>conntrack -D</code> silently ignores the <code>-u</code> status filter because the delete path in conntrack-tools omits the status filter from the kernel request, making UNREPLIED-only deletion impossible via the CLI.</p>
<p>The workaround is to parse <code>conntrack -L</code> output in shell, extract 5-tuple fields from each matching entry, and issue a separate <code>conntrack -D</code> call per entry. However, this is sensitive to output format changes and requires multiple subprocesses per entry.</p>
<p>Additionally, flushing all mwan3-marked entries with conntrack requires a shell loop spawning one conntrack process per mark value (up to 63 for the default 6-bit mask).</p>
<p>mwan3ct is a small C binary that calls libnetfilter_conntrack's <code>NFCT_Q_FLUSH_FILTER</code> directly, passing mark and status filters to the kernel in a single netlink message. The kernel performs the filtered delete atomically - no dump, no iteration, no per-entry delete is needed.</p>
<pre><code>mwan3ct flush [--mark &lt;val&gt;/&lt;mask&gt;] [--mark-any &lt;mask&gt;] [--status &lt;val&gt;/&lt;mask&gt;]</code></pre>
<ul>
<li><strong><code>--mark &lt;val&gt;/&lt;mask&gt;</code></strong>: delete entries with an exact mark match. Single kernel call.</li>
<li><strong><code>--mark-any &lt;mask&gt;</code></strong>: delete entries with any non-zero mark value within the mask. Internally loops over all possible mark IDs (up to 63 for the default 6-bit mask), issuing one kernel call per ID. All within a single process and netlink socket.</li>
<li><strong><code>--status &lt;val&gt;/&lt;mask&gt;</code></strong>: filter by conntrack status bits. For example, <code>--status 0/0x2</code> matches entries where IPS_SEEN_REPLY (0x2) is not set, i.e. UNREPLIED entries.</li>
<li><strong><code>--mark</code> and <code>--mark-any</code> are mutually exclusive.</strong> At least one filter option is required.</li>
<li><strong><code>--mark</code></strong> and <strong><code>--status</code></strong> filters can be combined.</li>
</ul>
<p>Exit code 0 on success, including when no entries were matched. Errors are printed to stderr.</p>
<hr>
<h3 id="614-libmwan3mwan3-get-addruc">6.14 <span class="file-path">lib/mwan3/mwan3-get-addr.uc</span></h3>
<p>Address lookup helper.</p>
<p>Queries the kernel via RTM_GETADDR (netlink) to find an IP address in one of two modes:</p>
<ul>
<li><strong>By device</strong>: given <code>&lt;family&gt; &lt;device&gt;</code>, returns the first suitable address on that device. For IPv6, skips link-local (fe80::/10) and non-global-scope addresses. For IPv4, accepts scope 0 (global) and scope 253 (link).</li>
<li><strong>By prefix</strong>: given <code>&lt;family&gt; "" &lt;prefix&gt;</code>, returns the first global-scope address whose string representation begins with the prefix. Used for IPv6 prefix matching.</li>
</ul>
<p>Prints the address stripped of any prefix-length to stdout and exits 0, or exits 1 if no matching address is found. Replaces <code>ip -f inet[6] addr show dev &lt;device&gt;</code> shell parsing.</p>
<hr>
<h3 id="615-libmwan3mwan3-manage-rulesuc">6.15 <span class="file-path">lib/mwan3/mwan3-manage-rules.uc</span></h3>
<p>Routing policy rule manager.</p>
<p>Manages ip rules (policy routing entries) via netlink (RTM_GETRULE, RTM_NEWRULE, RTM_DELRULE). Supports four modes:</p>
<ul>
<li><strong><code>check &lt;family&gt; &lt;prio1&gt; [prio2...]</code></strong>: returns a decimal bitmask where bit N is set if the Nth listed priority is absent from the live rule table. Used by mwan3.sh to decide which rules need to be created.</li>
<li><strong><code>check-route &lt;family&gt; &lt;table_id&gt; &lt;device&gt;</code></strong>: exits 0 if a default route for the given output device exists in the specified table, 1 otherwise. Used before populating per-interface tables to detect whether the route is already present.</li>
<li><strong><code>delete-iface &lt;id&gt; &lt;iif_base&gt; &lt;fwmark_base&gt; &lt;unreachable_base&gt; &lt;mmx_mask&gt;</code></strong>: removes the iif lookup, fwmark lookup, and unreachable rules for a single interface (identified by its sequential index). Each rule is located by priority and verified by type before deletion; rules not present are silently skipped.</li>
<li><strong><code>add-general &lt;family&gt; &lt;bh_prio&gt; &lt;bh_mark&gt; &lt;ur_prio&gt; &lt;ur_mark&gt; &lt;mask&gt;</code></strong>: adds the shared blackhole and unreachable rules common to all interfaces. Rules already present at those priorities are not re-added.</li>
</ul>
<p>Replaces <code>ip rule add/del</code> shell invocations that required fork-exec per operation and text parsing for presence checks.</p>
<hr>
<h3 id="616-libmwan3mwan3-list-routesuc">6.16 <span class="file-path">lib/mwan3/mwan3-list-routes.uc</span></h3>
<p>Route table lister.</p>
<p>Dumps routes from a given routing table via RTM_GETROUTE and prints destination prefixes to stdout, one per line. Accepts a numeric table ID or the special value <code>main</code>. Filters out: default routes (0.0.0.0/0, ::/0), host routes (prefix-length 32/128), link-local prefixes (fe80::/10, 169.254.0.0/16), IPv4 multicast (first octet &gt;= 224), and duplicate destinations.</p>
<p>Used by mwan3.sh to enumerate routes that need to be copied into per-interface routing tables.</p>
<hr>
<h3 id="617-libmwan3mwan3-create-iface-routeuc">6.17 <span class="file-path">lib/mwan3/mwan3-create-iface-route.uc</span></h3>
<p>Per-interface routing table populator.</p>
<p>Copies routes from the main routing table (and any additional tables listed in the <code>rt_table_lookup</code> UCI global) into a single per-interface routing table via RTM_NEWROUTE with NLM_F_CREATE | NLM_F_REPLACE. Uses ubus (<code>network.interface dump</code>) to map interface names to their kernel devices and table IDs; the <code>_4</code>/<code>_6</code> suffix convention for dual-stack interface pairs is recognised. Routes already present in the target table (matched by dst/oif/gateway/priority) are skipped. When <code>source_routing</code> is set, the route's source address is included in the copy.</p>
<p>Replaces <code>ip route add table &lt;id&gt;</code> shell invocations that required fork-exec and text parsing of <code>ip route show</code> output.</p>
<h3 id="618-usrbinmwan3ipcheck">6.18 <span class="file-path">usr/bin/mwan3ipcheck</span></h3>
<p>IP address and CIDR validation binary. Accepts a single argument and prints one of four classification strings to stdout, exiting 0 on success or 1 on failure.</p>
<table>
<thead><tr><th>Output</th><th>Exit</th><th>Meaning</th></tr></thead><tbody>
<tr><td class="func-sig"><code>ipv4</code></td><td>0</td><td>All elements are valid IPv4 addresses or IPv4 CIDR prefixes</td></tr>
<tr><td class="func-sig"><code>ipv6</code></td><td>0</td><td>All elements are valid IPv6 addresses or IPv6 CIDR prefixes</td></tr>
<tr><td class="func-sig"><code>mixed</code></td><td>1</td><td>Comma-separated list contains a mix of IPv4 and IPv6 elements</td></tr>
<tr><td class="func-sig"><code>invalid</code></td><td>1</td><td>Any element failed validation, input is empty, or argument count is wrong</td></tr>
</tbody></table>
<p>The argument may be a single address, a single CIDR prefix, or a comma-separated list of addresses and prefixes. Leading and trailing whitespace around each comma-separated token is stripped before classification.</p>
<p><strong>Address validation</strong> uses <code>inet_pton()</code> - first trying <code>AF_INET</code>, then <code>AF_INET6</code>. Any address accepted by <code>mwan3ipcheck</code> is guaranteed to be accepted by nft, which uses the same parser internally.</p>
<p><strong>CIDR validation:</strong> when a <code>/</code> is present, the prefix length is validated against the family maximum (32 for IPv4, 128 for IPv6). Out-of-range values and leading zeros in the prefix length are rejected. A trailing <code>/</code> with no digits is rejected.</p>
<p><strong>Comma-separated lists:</strong> each token is classified individually. If all tokens are the same family the list returns that family. If tokens span both families the result is <code>"mixed"</code>, which the rule-building path rejects since a single nft anonymous set cannot contain both IPv4 and IPv6 elements.</p>
<p>The binary has no dependencies beyond libc (<code>inet_pton</code>, <code>strtol</code>, and standard string functions only).</p>
<hr>
<h2 id="7-function-reference">7. Function Reference</h2>
<h3 id="71-commonsh-functions">7.1 common.sh Functions</h3>
<table>
<thead><tr><th>Function</th><th>Purpose</th></tr></thead><tbody>
<tr><td><code>LOG facility message...</code></td><td>Logs to syslog. Suppresses <code>debug</code> level by default.</td></tr>
<tr><td><code>mwan3_nft_exec args...</code></td><td>Runs <code>nft</code> with arguments, logs errors. Returns 1 on failure. When <code>MWAN3_BATCH_DEPTH &gt; 0</code>, routes to <code>mwan3_nft_push</code> instead.</td></tr>
<tr><td><code>mwan3_nft_batch_start</code></td><td>Increments <code>MWAN3_BATCH_DEPTH</code>; truncates <code>/tmp/mwan3_nft_batch.$$</code> only at depth 0-&gt;1.</td></tr>
<tr><td><code>mwan3_nft_push line</code></td><td>Appends a line to the batch file.</td></tr>
<tr><td><code>mwan3_nft_batch_commit</code></td><td>Decrements <code>MWAN3_BATCH_DEPTH</code>; executes <code>nft -f /tmp/mwan3_nft_batch.$$</code> and removes the temp file only at depth 1-&gt;0.</td></tr>
<tr><td><code>mwan3_nft_reload_start</code></td><td>Opens outermost batch (depth 0-&gt;1) and writes preamble: flush 8 skeleton chains, two-pass flush+delete all dynamic chains, delete 6 internal sets. User-defined sets and sticky sets are untouched.</td></tr>
<tr><td><code>mwan3_nft_reload_commit</code></td><td>Thin wrapper around <code>mwan3_nft_batch_commit</code>. Commits entire accumulated batch atomically at depth 1-&gt;0. On failure, kernel rolls back and old ruleset continues.</td></tr>
<tr><td><code>mwan3_nft_mark_expr value mask</code></td><td>Outputs <code>meta mark set meta mark &amp; COMPLEMENT | VALUE</code>. Uses <code>&amp;</code> and <code>|</code> symbols (not keywords). Equivalent to iptables <code>--set-xmark VALUE/MASK</code>.</td></tr>
<tr><td><code>mwan3_ensure_nft_framework</code></td><td>Recreates the 6 internal mwan3 sets with <code>interval</code> + <code>auto-merge</code> flags (skipping the direct delete loop when inside a batch - deletes already in the preamble). Adds all 8 skeleton chains in <code>table inet mwan3</code>. Idempotent for chains.</td></tr>
<tr><td><code>mwan3_build_or_chains_nft</code></td><td>Builds the 126 per-mark setter chains used by vmap-dispatch save/restore (63 <code>mwan3_or_meta_&lt;imm&gt;</code> restore chains + 63 <code>mwan3_or_ct_&lt;imm&gt;</code> atomic clear+set save chains). Always flushes and re-populates chain bodies. See <a href="#connmark-operations">§2 Connmark Operations</a>.</td></tr>
<tr><td><code>mwan3_or_chain_suffix mark</code></td><td>Converts a numeric mark value to the canonical lowercase <code>0x%x</code> hex string used as the suffix for OR-immediate setter chain names (e.g. <code>0x100</code> for interface 1 with default mask). Called by <code>mwan3_build_or_chains_nft</code>, <code>mwan3_or_vmap_body</code>, <code>mwan3_all_marks</code>, and <code>mwan3_create_policies_nft</code>.</td></tr>
<tr><td><code>mwan3_or_vmap_body reg mark...</code></td><td>Builds the body string for a vmap statement dispatching on masked mark values into OR-immediate setter chains. Called by <code>mwan3_set_general_nft()</code>.</td></tr>
<tr><td><code>mwan3_all_marks</code></td><td>Enumerates every mark value that needs to appear in the restore/save vmaps: all per-interface marks (IDs 1..<code>MWAN3_INTERFACE_MAX</code> bit-spread through <code>MMX_MASK</code>) plus <code>MMX_DEFAULT</code>, <code>MMX_BLACKHOLE</code>, and <code>MMX_UNREACHABLE</code>. Echoes a space-separated list. Used by <code>mwan3_set_general_nft()</code> to build the vmap body.</td></tr>
<tr><td><code>mwan3_init</code></td><td>Loads UCI config, creates status dirs, computes all mask constants (<code>MMX_MASK</code>, <code>MMX_DEFAULT</code>, <code>MMX_BLACKHOLE</code>, <code>MMX_UNREACHABLE</code>, <code>MMX_MASK_COMPLEMENT</code>, <code>MWAN3_INTERFACE_MAX</code>), and reads <code>iif_rule_base</code>, <code>fwmark_rule_base</code>, <code>unreachable_rule_base</code> from UCI globals into <code>MWAN3_IIF_RULE_BASE</code>, <code>MWAN3_FWMARK_RULE_BASE</code>, <code>MWAN3_UNREACHABLE_RULE_BASE</code>, reverting all three to defaults (1000/2000/3000) if either ordering constraint is violated.</td></tr>
<tr><td><code>mwan3_id2mask id mask</code></td><td>Bit-spreads <code>id</code>'s bits into positions where <code>mask</code> has 1-bits. Arguments are variable names (indirect evaluation).</td></tr>
<tr><td><code>mwan3_count_one_bits var</code></td><td>Counts 1-bits in the value named by <code>var</code> (indirect evaluation). Uses <code>n&amp;(n-1)</code> trick.</td></tr>
<tr><td><code>mwan3_get_true_iface out_var iface</code></td><td>Resolves virtual interface names (appends _4 or _6 suffix if that interface exists in netifd).</td></tr>
<tr><td><code>mwan3_get_src_ip out_var iface</code></td><td>Gets the source IP for an interface, with fallbacks for IPv6-PD prefixes.</td></tr>
<tr><td><code>readfile var path</code></td><td>Reads entire file into variable. Returns 1 if file doesn't exist.</td></tr>
<tr><td><code>mwan3_get_mwan3track_status out_var iface</code></td><td>Returns tracker status: <code>disabled</code>, <code>down</code>, <code>paused</code>, or <code>active</code>.</td></tr>
<tr><td><code>get_uptime [out_var]</code></td><td>Returns system uptime in seconds (integer).</td></tr>
<tr><td><code>get_online_time out_var iface</code></td><td>Returns how long the interface has been online.</td></tr>
</tbody></table>
<h3 id="72-set-management-functions">7.2 Set Management Functions</h3>
<table>
<thead><tr><th>Function</th><th>Purpose</th></tr></thead><tbody>
<tr><td><code>mwan3_set_connected_ipv4</code></td><td>Flushes and repopulates <code>mwan3_connected_v4</code> from main routing table. Adds <code>224.0.0.0/3</code> for multicast. Self-contained: starts and commits its own nft batch.</td></tr>
<tr><td><code>mwan3_set_connected_ipv6</code></td><td>Same for IPv6. Skips if <code>$NO_IPV6</code>. Self-contained batch.</td></tr>
<tr><td><code>mwan3_set_connected_sets</code></td><td>Calls both <code>_ipv4</code> and <code>_ipv6</code> functions.</td></tr>
<tr><td><code>mwan3_set_custom_set table_id</code></td><td>Callback for <code>config_list_foreach</code>. Adds routes from the given table to custom sets. Pushes to an existing batch (does not start/commit).</td></tr>
<tr><td><code>mwan3_set_custom_sets</code></td><td>Flushes and repopulates custom sets from all <code>rt_table_lookup</code> entries in globals config.</td></tr>
<tr><td><code>mwan3_set_dynamic_network</code></td><td>Callback for <code>config_list_foreach</code>. Classifies a single <code>bypass_network</code> CIDR as IPv4 (contains <code>.</code>) or IPv6 (contains <code>:</code>) and pushes the appropriate <code>add element</code> command to the current batch.</td></tr>
<tr><td><code>mwan3_set_dynamic_sets</code></td><td>Flushes <code>mwan3_dynamic_v4/v6</code> then repopulates from <code>bypass_network</code> entries in globals UCI config. Called at startup and in both fw4 reload recovery paths.</td></tr>
</tbody></table>
<div class="note">
<p><strong>Why are connected functions self-contained?</strong> <code>mwan3_set_connected_ipv4/ipv6</code> each manage their own batch because they may be called from contexts that are not already inside a larger batch (init.d, hotplug). <code>mwan3rtmon</code> does not call these shell functions; it has its own independent <code>populate_connected_set()</code> that operates via direct netlink calls and writes to the same sets using its own <code>nft_batch</code>.</p>
</div>
<h3 id="73-general-rule-setup">7.3 General Rule Setup</h3>
<table>
<thead><tr><th>Function</th><th>Purpose</th></tr></thead><tbody>
<tr><td><code>mwan3_set_general_rules</code></td><td>Adds <code>ip rule</code> entries for blackhole and unreachable marks (both IPv4 and IPv6) at priorities <code>MWAN3_FWMARK_RULE_BASE + MM_BLACKHOLE</code> and <code>MWAN3_FWMARK_RULE_BASE + MM_UNREACHABLE</code>. These are ip policy rules, not nftables rules.</td></tr>
<tr><td><code>mwan3_set_general_nft</code></td><td>Builds the per-mark OR setter chains via <code>mwan3_build_or_chains_nft()</code>, then populates all hook chain rules: IPv6 RA bypass (prerouting), vmap-dispatched connmark restore (if mark unset), jump to <code>mwan3_ifaces_in</code>, <code>fib daddr type local return</code> (prerouting, after ifaces_in), jumps to custom/connected/dynamic/rules chains, connmark save, and post-rules connected re-check. Idempotent: checks if rules already exist before adding. Uses a single batch for all operations.</td></tr>
</tbody></table>
<h4 id="rules-added-by-mwan3setgeneralnft">Rules added by <code>mwan3_set_general_nft()</code></h4>
<p>For each of connected/custom/dynamic chains, adds mark-setting rules that match against the corresponding sets and apply <code>MMX_DEFAULT</code>.</p>
<p>For the prerouting and output hook chains, adds (in order):</p>
<ol>
<li>[prerouting only] IPv6 RA bypass (accept ICMPv6 types: nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, nd-redirect)</li>
<li><strong>Non-destructive connmark restore</strong> (if <code>meta mark &amp; MMX_MASK == 0</code>): <code>ct mark &amp; MMX_MASK vmap { ... -&gt; jump mwan3_or_meta_&lt;imm&gt; }</code>. The setter chains contain <code>meta mark set meta mark | &lt;imm&gt;</code>, so non-mwan3 bits in meta mark are preserved across the restore.</li>
<li>Jump to <code>mwan3_ifaces_in</code> (if still 0)</li>
<li>[prerouting only] <code>fib daddr type local return</code> (if still 0): packets destined for the router's own IP return immediately. Placed after <code>mwan3_ifaces_in</code> so that WAN traffic arriving on a mwan3 interface is already marked by the iface_in catchall (mark != 0) and never reaches this check. Only traffic on non-WAN interfaces (LAN, loopback) with mark still zero reaches the fib local check. See <a href="#mwan3-fix-numgen-counter-contamination-from-inbound-and-reply-traffic">§c4bcd951e</a>.</li>
<li>Jump to custom, connected, dynamic chains (if still 0)</li>
<li>Jump to <code>mwan3_rules</code> (if still 0)</li>
<li><strong>Atomic connmark save</strong> (always): <code>meta mark &amp; MMX_MASK vmap { ... -&gt; jump mwan3_or_ct_&lt;imm&gt; }</code> dispatches into setter chains that clear mwan3's bits and set the new value in a single nft expression (<code>ct mark set ct mark &amp; MMX_MASK_COMPLEMENT | &lt;imm&gt;</code>). No intermediate state is visible to other CPUs. Bits in ct mark owned by other packages survive unchanged.</li>
<li>Post-rules: jump to custom/connected/dynamic again for non-default marks</li>
</ol>
<p>See <a href="#connmark-operations">§2 Connmark Operations</a> for the rationale and the kernel-limitation context.</p>
<h3 id="74-interface-management">7.4 Interface Management</h3>
<table>
<thead><tr><th>Function</th><th>Purpose</th></tr></thead><tbody>
<tr><td><code>mwan3_update_iface_to_table</code></td><td>Populates <code>mwan3_iface_tbl</code>: a space-separated string of <code>name=id</code> pairs for all configured interfaces, where <code>id</code> is the sequential table number. Called lazily (on first use) by <code>mwan3_get_iface_id()</code>. Also called explicitly during <code>start_service()</code> to prime the cache before interface chains are created.</td></tr>
<tr><td><code>mwan3_get_iface_id out_var iface</code></td><td>Looks up <code>mwan3_iface_tbl</code> for the given interface name and writes its table ID into the named output variable. Calls <code>mwan3_update_iface_to_table()</code> on first use if the cache is empty.</td></tr>
<tr><td><code>mwan3_create_iface_nft iface device</code></td><td>Creates (or flushes) <code>mwan3_iface_in_&lt;iface&gt;</code> chain. Adds rules matching on <code>iifname</code> and address family: source in connected/custom/dynamic → MMX_DEFAULT; otherwise → interface mark. Adds jump from <code>mwan3_ifaces_in</code> if not already present. Also installs a per-interface <code>mwan3_postrouting</code> SNAT rule for IPv6 interfaces when the <code>snat6</code> UCI option is set. Stale <code>mwan3_snat_&lt;iface&gt;</code>-tagged rules from a prior incarnation are removed first. See <a href="#router-originated-traffic-and-source-address-rewriting">§4</a>.</td></tr>
<tr><td><code>mwan3_rebuild_iface_nft iface</code></td><td>Rebuilds a single interface's nft chain if the interface is enabled, the correct family is available, and the interface is currently up (verified via ubus). Used during <code>reload_service</code> to rebuild all interface chains within the atomic batch. Calls <code>mwan3_create_iface_nft()</code> after resolving the L3 device from netifd.</td></tr>
<tr><td><code>mwan3_delete_iface_nft iface</code></td><td>Removes the jump rule from <code>mwan3_ifaces_in</code> (by handle lookup), removes any <code>mwan3_snat_&lt;iface&gt;</code>-tagged rules from <code>mwan3_postrouting</code> (comment-tag match), then flushes and deletes the interface chain.</td></tr>
<tr><td><code>mwan3_delete_iface_map_entries iface</code></td><td>Iterates all <code>mwan3_sticky_v[46]_*</code> sets in the <code>inet</code> family, finds sets whose name ends with <code>_&lt;id&gt;</code> (this interface's id), and flushes them. Sets are flushed rather than deleted because rule chains may still reference the set name.</td></tr>
<tr><td><code>mwan3_create_iface_rules iface device</code></td><td>Adds <code>ip rule</code> entries: pref <code>id+MWAN3_IIF_RULE_BASE</code> (iif lookup), pref <code>id+MWAN3_FWMARK_RULE_BASE</code> (fwmark lookup), pref <code>id+MWAN3_UNREACHABLE_RULE_BASE</code> (fwmark unreachable).</td></tr>
<tr><td><code>mwan3_delete_iface_rules iface</code></td><td>Removes ip rules for this interface (both IPv4 and IPv6) by delegating to <code>mwan3-manage-rules.uc delete-iface</code>. The ucode script locates the iif lookup rule, fwmark lookup rule, and unreachable rule by priority offset and table ID, and deletes each via RTM_DELRULE. Handles rule-base changes and <code>mmx_mask</code> changes transparently.</td></tr>
<tr><td><code>mwan3_create_iface_route iface</code></td><td>Copies routes from the main routing table (and any <code>rt_table_lookup</code> extra tables) into the per-interface routing table by invoking <code>mwan3-create-iface-route.uc</code>. The ucode script reads source routes via rtnl, skips routes already present in the target table, and writes each qualifying route with RTM_NEWROUTE. Passes the <code>source_routing</code> UCI flag to control whether source addresses are copied.</td></tr>
<tr><td><code>mwan3_delete_iface_route iface</code></td><td>Flushes the per-interface routing table. <em>Unchanged.</em></td></tr>
</tbody></table>
<h4 id="handle-based-rule-deletion">Handle-Based Rule Deletion</h4>
<p>nftables doesn't support deleting rules by match criteria (like iptables <code>-D chain match...</code>). Instead, <code>mwan3_delete_iface_nft()</code> uses:</p>
<pre><code>handle=$($NFT -a list chain inet mwan3 mwan3_ifaces_in | \
grep "jump mwan3_iface_in_$1" | sed -n 's/.*# handle \([0-9]*\)/\1/p')
$NFT delete rule inet mwan3 mwan3_ifaces_in handle "$handle"</code></pre>
<p>The <code>-a</code> flag shows rule handles in comments, which can then be used for targeted deletion.</p>
<h3 id="75-policy--load-balancing">7.5 Policy &amp; Load Balancing</h3>
<table>
<thead><tr><th>Function</th><th>Purpose</th></tr></thead><tbody>
<tr><td><code>mwan3_set_policy member_config</code></td><td>Callback per policy member. Tracks lowest metric and accumulates online members as <code>iface:id:weight</code> tuples into <code>$policy_members</code>. Tracks offline devices into <code>$policy_offline_devices</code>. Uses caller's variables (dynamic scoping).</td></tr>
<tr><td><code>mwan3_create_policies_nft policy</code></td><td>Creates/flushes the <code>mwan3_policy_&lt;name&gt;</code> chain. Iterates members via <code>mwan3_set_policy</code>, then builds the chain: single member gets a direct mark-set rule; multiple members get a <code>numgen</code> rule; offline devices get out-device fallback rules; last-resort rule (unreachable/blackhole/default) is appended.</td></tr>
<tr><td><code>mwan3_set_policies_nft</code></td><td>Before creating policy chains, enumerates all existing <code>mwan3_policy_*</code> chains in <code>inet mwan3</code> and deletes any whose name is not in the current UCI policy config (orphaned chain sweep). Then iterates all policy configs and calls <code>mwan3_create_policies_nft</code> for each.</td></tr>
</tbody></table>
<h3 id="76-sticky-routing">7.6 Sticky Routing</h3>
<table>
<thead><tr><th>Function</th><th>Purpose</th></tr></thead><tbody>
<tr><td><code>mwan3_get_policy_members_for_family policy family</code></td><td>Iterates the members of a policy config via <code>config_list_foreach</code>. For each member whose interface matches the requested family, resolves the interface id and computes its mark, and accumulates <code>&lt;id&gt;:&lt;mark&gt;</code> tuples into <code>$_policy_member_marks</code>. Used by the sticky path in <code>mwan3_set_user_nft_rule()</code> to enumerate per-member sets.</td></tr>
</tbody></table>
<h3 id="77-user-rules">7.7 User Rules</h3>
<table>
<thead><tr><th>Function</th><th>Purpose</th></tr></thead><tbody>
<tr><td><code>mwan3_set_user_nft_rule rule ipv</code></td><td>Builds and adds a single nft rule to <code>mwan3_rules</code>. Translates UCI config options (proto, src_ip, dest_ip, src_port, dest_port, src_iface, ipset, ipset_src) into nft match expressions. For sticky rules, calls <code>mwan3_get_policy_members_for_family()</code> and builds a <code>mwan3_rule_&lt;name&gt;</code> chain with per-member address sets and OR-immediate restore/save (see §8). Handles logging rules. Pre-creates missing nft sets (e.g., dnsmasq nftsets that haven't started yet) to prevent batch failures. When <code>family</code> is explicitly <code>ipv4</code> or <code>ipv6</code> but none of <code>src_ip</code>, <code>dest_ip</code>, <code>ipset</code>, or <code>ipset_src</code> are set, prepends <code>meta nfproto ipv4/ipv6</code> to the nft match to prevent a bare rule from matching both address families. Also translates <code>proto icmp</code> to <code>meta l4proto ipv6-icmp</code> when <code>family</code> is <code>ipv6</code> (in an inet table, proto 1 is ICMPv4 only). <code>src_ip</code> and <code>dest_ip</code> are validated via <code>mwan3ipcheck</code>; a result of <code>"invalid"</code> or <code>"mixed"</code> causes the rule to be skipped with a warning log. When a comma is present in <code>src_ip</code> or <code>dest_ip</code>, the value is wrapped in nft anonymous set syntax (<code>{ addr1, addr2 }</code>) for the generated match expression.</td></tr>
<tr><td><code>mwan3_set_user_rules</code></td><td>Flushes <code>mwan3_rules</code> chain, then iterates all rule configs for both ipv4 and ipv6, calling <code>mwan3_set_user_nft_rule</code> for each. Uses a single batch.</td></tr>
<tr><td><code>mwan3_set_user_iface_rules iface device</code></td><td>Called on ifup to check if the rules chain needs rebuilding (if this interface is a <code>src_iface</code> in any rule).</td></tr>
</tbody></table>
<h4 id="uci-to-nft-match-translation">UCI-to-nft Match Translation</h4>
<table>
<thead><tr><th>UCI Option</th><th>nft Expression</th></tr></thead><tbody>
<tr><td><code>proto tcp</code></td><td><code>meta l4proto tcp</code></td></tr>
<tr><td><code>src_ip 10.0.0.0/8</code></td><td><code>ip saddr 10.0.0.0/8</code> (or <code>ip6 saddr</code>)</td></tr>
<tr><td><code>src_ip 10.0.0.1,10.0.0.2</code></td><td><code>ip saddr { 10.0.0.1, 10.0.0.2 }</code></td></tr>
<tr><td><code>dest_ip 1.2.3.4</code></td><td><code>ip daddr 1.2.3.4</code></td></tr>
<tr><td><code>dest_ip 1.2.3.4,5.6.7.8</code></td><td><code>ip daddr { 1.2.3.4, 5.6.7.8 }</code></td></tr>
<tr><td><code>src_iface lan</code></td><td><code>iifname "br-lan"</code></td></tr>
<tr><td><code>src_port 80,443</code></td><td><code>th sport { 80, 443 }</code></td></tr>
<tr><td><code>dest_port 8080</code></td><td><code>th dport { 8080 }</code></td></tr>
<tr><td><code>ipset my_set</code></td><td><code>ip daddr @my_set</code></td></tr>
<tr><td><code>ipset_src my_set</code></td><td><code>ip saddr @my_set</code></td></tr>
<tr><td><code>use_policy balanced</code></td><td><code>jump mwan3_policy_balanced</code></td></tr>
<tr><td><code>use_policy default</code></td><td><code>meta mark set ... | MMX_DEFAULT</code></td></tr>
</tbody></table>
<h4 id="missing-nft-set-pre-creation">Missing nft Set Pre-creation</h4>
<p>When a user rule references an <code>ipset</code> (destination nft set) or <code>ipset_src</code> (source nft set) that doesn't exist yet in <code>table inet mwan3</code>, <code>nft -f</code> would roll back the entire batch atomically. To prevent this, <code>mwan3_set_user_nft_rule()</code> pre-creates any missing referenced sets with the appropriate type. The same logic applies to both <code>ipset</code> and <code>ipset_src</code>.</p>
<h3 id="78-user-defined-nft-set-management">7.8 User-defined nft Set Management</h3>
<table>
<thead><tr><th>Function</th><th>Purpose</th></tr></thead><tbody>
<tr><td><code>mwan3_render_config_ipsets</code></td><td>Iterates all <code>config ipset</code> UCI sections and calls <code>_mwan3_render_one_ipset</code> for each. Creates user-defined sets in <code>table inet mwan3</code>. Called from <code>start_service</code> and <code>reload_service</code>.</td></tr>
<tr><td><code>_mwan3_nft_time_to_sec str</code></td><td>Converts an nft human-unit timeout string (e.g. <code>1h</code>, <code>5m</code>, <code>300s</code>) to an integer number of seconds. Recognised suffixes: <code>d</code> (days), <code>h</code> (hours), <code>m</code> (minutes), <code>s</code> (seconds, default if no suffix). Used by <code>_mwan3_ipset_needs_delete</code> to compare the live kernel timeout against the desired spec in a common unit.</td></tr>
<tr><td><code>_mwan3_ipset_needs_delete name type counter timeout maxelem</code></td><td>Returns 0 (true) if the named set exists in <code>table inet mwan3</code> AND its flags (type, counter, timeout, size) differ from the desired spec; returns 1 if the set is absent or all flags match. Called by <code>_mwan3_render_one_ipset</code> during reload to decide whether to queue a <code>delete set</code> before recreating the set with new flags. Timeout values are compared in seconds after converting nft's human-unit display (e.g. <code>1h</code>, <code>5m</code>) via <code>_mwan3_nft_time_to_sec</code>.</td></tr>
<tr><td><code>_mwan3_render_one_ipset name</code></td><td>Creates one user-defined nft set. Builds a <code>type ipv4_addr/ipv6_addr; flags interval; auto-merge;</code> declaration, optionally appending <code>counter;</code> (standalone statement, NOT a flag keyword) and <code>timeout Ns;</code>. On the start path, unconditionally deletes any existing set before creating. On the reload path, calls <code>_mwan3_ipset_needs_delete</code> and only queues a <code>delete set</code> when flags differ; otherwise <code>add set</code> is idempotent and dnsmasq-populated elements survive. Sets <code>MWAN3_NEED_DNSMASQ_HUP</code> if a domain-populated set is deleted. Adds inline <code>list entry</code> elements and <code>loadfile</code> contents.</td></tr>
<tr><td><code>mwan3_write_dnsmasq_fragments</code></td><td>For each <code>config ipset</code> with <code>list domain</code> entries, writes a dnsmasq confdir fragment enabling <code>nftset=/domain/FAMILY#inet#mwan3#setname</code>. Compares against previously written fragments; restarts dnsmasq only when content changes. Silently does nothing if no domain sets are configured.</td></tr>
<tr><td><code>mwan3_cleanup_orphaned_ipsets</code></td><td>Queries <code>nft list table inet mwan3</code> for user-defined sets (those not prefixed <code>mwan3_</code>), compares against configured set names, and pushes <code>delete set</code> for any orphans. Called in <code>reload_service</code> after <code>mwan3_render_config_ipsets</code>. Prevents stale sets accumulating when a set is removed from UCI config.</td></tr>
</tbody></table>
<h3 id="79-status-reporting">7.9 Status Reporting</h3>
<table>
<thead><tr><th>Function</th><th>Purpose</th></tr></thead><tbody>
<tr><td><code>mwan3_report_iface_status iface</code></td><td>Shows interface online/offline status with uptime. Checks ip rules, nft chain existence in <code>table inet mwan3</code>, and default route presence.</td></tr>
<tr><td><code>_mwan3_report_policies_for_family family</code></td><td>Calls ubus <code>mwan3 status</code> for the policies section, then iterates the policy/member hierarchy for the given family (<code>ipv4</code> or <code>ipv6</code>), printing each policy name with its member interfaces and current traffic percentages. Called by <code>mwan3_report_policies_v4()</code> and <code>mwan3_report_policies_v6()</code>.</td></tr>
<tr><td><code>mwan3_report_policies_v4/v6</code></td><td>Delegates to <code>_mwan3_report_policies_for_family()</code> for IPv4 and IPv6 respectively.</td></tr>
<tr><td><code>_mwan3_report_connected_set set_name</code></td><td>Parses the named connected set using <code>nft -j list set inet mwan3 &lt;set_name&gt;</code> and <code>jshn.sh</code>. Iterates the JSON element array and prints prefix elements as <code>addr/len</code> and range elements as <code>start-end</code>. Handles both address families. Called by <code>mwan3_report_connected_v4()</code> and <code>mwan3_report_connected_v6()</code>.</td></tr>
<tr><td><code>mwan3_report_connected_v4/v6</code></td><td>Thin wrappers that call <code>_mwan3_report_connected_set</code> with <code>mwan3_connected_v4</code> and <code>mwan3_connected_v6</code> respectively. <code>mwan3_report_connected_v6</code> returns early if <code>$NO_IPV6</code> is set.</td></tr>
<tr><td><code>mwan3_report_rules_v4/v6</code></td><td>Parses <code>nft list chain inet mwan3 mwan3_rules</code>.</td></tr>
<tr><td><code>mwan3_mark_to_name mark</code></td><td>Resolves a numeric mark value to an interface name (or "default"/"blackhole"/"unreachable").</td></tr>
</tbody></table>
<h3 id="710-lifecycle--hotplug">7.10 Lifecycle &amp; Hotplug</h3>
<table>
<thead><tr><th>Function</th><th>Purpose</th></tr></thead><tbody>
<tr><td><code>mwan3_ifup iface caller</code></td><td>Resolves interface status via ubus, then triggers the 25-mwan3 hotplug script with <code>ACTION=ifup</code>. When called from init, runs in background.</td></tr>
<tr><td><code>mwan3_interface_hotplug_shutdown iface [ifdown]</code></td><td>Triggers ifdown or disconnected hotplug event for an interface.</td></tr>
<tr><td><code>mwan3_interface_shutdown iface</code></td><td>Calls hotplug shutdown then cleans track state files.</td></tr>
<tr><td><code>mwan3_set_iface_hotplug_state iface state</code></td><td>Writes state (<code>online</code>/<code>offline</code>) to status file.</td></tr>
<tr><td><code>mwan3_get_iface_hotplug_state iface</code></td><td>Reads state from status file (defaults to <code>offline</code>).</td></tr>
<tr><td><code>mwan3_flush_conntrack iface action</code></td><td>Two-path conntrack flush. First, iterates the <code>flush_conntrack</code> UCI list for this interface: for each configured action that matches the current hotplug action, writes <code>f</code> to <code>$CONNTRACK_FILE</code> to flush the entire global conntrack table. Second, on <code>ifdown</code>, uses <code>conntrack -D --mark MARK/MMX_MASK</code> to selectively delete only the conntrack entries for this interface's fwmark. Both paths run; the global flush path runs first.</td></tr>
<tr><td><code>mwan3_flush_marked_conntrack</code></td><td>Flushes all conntrack entries that have any <code>MMX_MASK</code> bit set, iterating the full mwan3 id-space (IDs 1..<code>MWAN3_INTERFACE_MAX</code> plus default/blackhole/unreachable marks). Called from <code>reload_service</code> so that live flows re-enter the classification chains and are re-evaluated against new rules rather than staying pinned to a previously saved ct mark.</td></tr>
<tr><td><code>mwan3_update_peer_track_ip iface</code></td><td>If <code>track_gateway</code> is enabled, queries <code>ifstatus</code> for the point-to-point peer address and writes it to <code>$MWAN3TRACK_STATUS_DIR/&lt;iface&gt;/GATEWAY</code>.</td></tr>
<tr><td><code>mwan3_track_clean iface</code></td><td>Removes track status directory for the interface.</td></tr>
<tr><td><code>mwan3_dnsmasq_hup</code></td><td>Sends SIGHUP to running dnsmasq instances via <code>ubus call service signal '{"name":"dnsmasq","signal":1}'</code>. Called once from <code>start_service</code> after sets are created so dnsmasq resolves domain entries into nft sets. No longer uses mwan3evtd (which was removed ).</td></tr>
<tr><td><code>mwan3_flush_stale_conntrack</code></td><td>Flushes conntrack entries with no mwan3 mark (<code>0x0/MMX_MASK</code>) after mwan3 restart. New connections arriving during the brief startup window before iface_in chains exist get ct mark=0; this clears them so they re-establish correctly. Called from <code>start_service</code>.</td></tr>
<tr><td><code>mwan3_flush_unreplied_conntrack</code></td><td>Flushes all mwan3-marked conntrack entries that have never received a reply (<code>IPS_SEEN_REPLY</code> not set). Called after <code>mwan3_set_policies_nft</code> on the <code>connected</code> action and on <code>ifup</code> when <code>status=online</code>. Breaks the cycle where a stale mark from a previous policy state pins traffic to the wrong WAN: because no reply arrives the application keeps retrying, refreshing the conntrack timeout indefinitely. Flushing only UNREPLIED entries leaves established connections untouched.</td></tr>
</tbody></table>
<hr>
<h2 id="8-load-balancing-with-numgen">8. Load Balancing with numgen</h2>
<p>The iptables version used <code>-m statistic --mode random --probability P</code> to distribute traffic. This required inserting rules in specific order and computing running probabilities. The nftables version uses <code>numgen inc mod N map { ... }</code>, which is simpler and more deterministic.</p>
<h3 id="how-numgen-works">How numgen Works</h3>
<p><code>numgen inc mod N</code> generates a counter that increments on each packet and wraps at N. The <code>map { range : value }</code> maps counter values to marks.</p>
<pre><code># Example: wan (weight 3) + wanb (weight 2) = mod 5
# wan gets range 0-2 (3 values), wanb gets range 3-4 (2 values)
nft add rule inet mwan3 mwan3_policy_balanced \
meta mark &amp; 0x3f00 == 0 \
meta mark set numgen inc mod 5 map { 0-2 : 0x0100, 3-4 : 0x0200 }</code></pre>
<div class="warn">
<p><strong>Kernel limitation on compound set expressions:</strong> An early implementation tried <code>meta mark set meta mark &amp; COMP | numgen inc mod ...</code> to preserve non-mwan3 bits while applying the numgen result. This fails with "Operation not supported" because the kernel cannot mix two register sources (meta mark and numgen) in one set expression. The solution is to use <code>meta mark set numgen ...</code> directly; the <code>meta mark &amp; MMX_MASK == 0</code> guard condition ensures the mwan3 bits are already zero before the numgen result is applied.</p>
</div>
<h3 id="build-algorithm-in-mwan3createpoliciesnft">Build Algorithm (in <code>mwan3_create_policies_nft</code>)</h3>
<ol>
<li>Iterate policy members via <code>config_list_foreach</code></li>
<li>Track lowest metric per family (v4/v6 separately); only members at the lowest metric are included</li>
<li>Accumulate online members as <code>iface:id:weight</code> tuples</li>
<li>Calculate total weight = sum of all member weights</li>
<li>If single member: direct <code>meta mark set</code> (no numgen overhead)</li>
<li>If multiple members: build numgen map entries with ranges proportional to weight</li>
<li>Append offline device fallback rules (only if no online members)</li>
<li>Append last-resort rule (unreachable/blackhole/default)</li>
</ol>
<div class="note">
<p><strong>Difference from iptables version:</strong> The iptables version used probabilistic matching (<code>--probability</code>) which is statistically correct over many packets but can have short-term imbalance. The nftables <code>numgen inc</code> counter gives perfectly deterministic round-robin distribution at the configured weights.</p>
</div>
<hr>
<h2 id="9-sticky-routing-detail">9. Sticky Routing Detail</h2>
<p>Sticky routing ensures that repeat connections from the same source IP use the same WAN interface (important for HTTPS sessions, banking sites, etc.). The iptables version used <code>ipset hash:ip,mark</code> sets. The nftables version uses per-member <strong>address sets</strong> with OR-immediate vmap dispatch for non-destructive mark restore.</p>
<h3 id="why-per-member-sets-instead-of-a-single-map">Why Per-Member Sets Instead of a Single Map</h3>
<p>With the vmap-dispatch infrastructure, the correct approach is a plain address set per policy member, with the mark encoded in the chain name rather than the map value. The corresponding <code>mwan3_or_meta_&lt;mark&gt;</code> setter chain ORs only the mwan3 bits into meta mark, preserving everything else.</p>
<h3 id="data-structure">Data Structure</h3>
<pre><code># Created per policy member for rule "https"
# (policy "balanced" has members: wan id=1 mark=0x100, wanb id=2 mark=0x200)
nft add set inet mwan3 mwan3_sticky_v4_https_1 { type ipv4_addr; flags timeout; timeout 600s; }
nft add set inet mwan3 mwan3_sticky_v4_https_2 { type ipv4_addr; flags timeout; timeout 600s; }</code></pre>
<p>One set per policy member, per rule, per address family. Sets hold source addresses only (no value side). The mark to apply is encoded in which set the saddr is found in.</p>
<h3 id="rule-chain-structure-for-sticky-rule-https-policy-balanced-wanid1-wanbid2">Rule Chain Structure (for sticky rule "https", policy "balanced", wan=id1 wanb=id2)</h3>
<pre><code>chain mwan3_rule_https {
# Restore: if saddr is in this member's set, OR its mark into meta mark
# (non-destructive - pbr bits in meta mark are preserved)
ip saddr @mwan3_sticky_v4_https_1 jump mwan3_or_meta_0x100
ip saddr @mwan3_sticky_v4_https_2 jump mwan3_or_meta_0x200
# New flows (no set match, mark still 0) fall through to policy
meta mark &amp; 0x3f00 == 0 jump mwan3_policy_balanced
# Save: after policy assigns a mark, record saddr in the matching member's set
meta mark &amp; 0x3f00 == 0x100 update @mwan3_sticky_v4_https_1 { ip saddr timeout 600s }
meta mark &amp; 0x3f00 == 0x200 update @mwan3_sticky_v4_https_2 { ip saddr timeout 600s }
}</code></pre>
<h3 id="flow-for-a-sticky-rule">Flow for a Sticky Rule</h3>
<ol>
<li>Packet arrives at <code>mwan3_rules</code> chain</li>
<li>Matches the user rule → <code>jump mwan3_rule_https</code></li>
<li><strong>Returning source</strong>: saddr is found in one of the per-member sets → <code>jump mwan3_or_meta_&lt;mark&gt;</code> ORs only the mwan3 bits into meta mark (non-destructive). The policy jump guard is false (mark already set), so policy is skipped. The matching update rule refreshes the timeout.</li>
<li><strong>New source</strong>: no set match, mark stays 0. Falls through to the policy chain which assigns a mark. The matching update rule then adds saddr to the corresponding member's set with the configured timeout.</li>
<li>After timeout seconds of inactivity the set entry expires and the source is re-evaluated at next connection.</li>
</ol>
<hr>
<h2 id="10-service-lifecycle-and-conntrack-management">10. Service Lifecycle and Conntrack Management</h2>
<h3 id="101-start">10.1 Start</h3>
<pre><code>/etc/init.d/mwan3 start
+--&gt; nft -f /lib/mwan3/mwan3-skeleton.nft (abort if fails)
+--&gt; mwan3_init() compute masks
+--&gt; mwan3_ensure_nft_framework() recreate 6 internal sets, ensure chains
+--&gt; mwan3_render_config_ipsets() create user-defined sets from config ipset
+--&gt; mwan3_write_dnsmasq_fragments() write nftset confdir fragments; restart dnsmasq if changed
+--&gt; start_tracker per interface launch mwan3track
+--&gt; mwan3_update_iface_to_table() build iface-&gt;table mapping
+--&gt; mwan3_set_dynamic/connected/custom sets
+--&gt; mwan3_set_general_rules() ip rule add blackhole/unreachable
+--&gt; mwan3_ifup per interface trigger hotplug (creates iface chains + ip rules)
+--&gt; wait for hotplug completion
+--&gt; mwan3_set_general_nft() populate hook chain rules
+--&gt; mwan3_set_policies_nft() create policy chains
+--&gt; mwan3_set_user_rules() populate user rules chain
+--&gt; mwan3_flush_stale_conntrack() flush zero-mark conntrack entries
+--&gt; mwan3_dnsmasq_hup() SIGHUP dnsmasq to populate nftset domain sets
+--&gt; [if flow_offloading=1] flush conntrack (force flow re-establishment)
+--&gt; start mwan3rtmon (ipv4 + ipv6) route monitor daemons</code></pre>
<h3 id="102-reload">10.2 Reload</h3>
<pre><code>/etc/init.d/mwan3 reload
+--&gt; mwan3_nft_reload_start() open batch + write preamble (flush/delete dynamic state)
+--&gt; mwan3_ensure_nft_framework() recreate 6 internal sets
+--&gt; mwan3_render_config_ipsets() add user-defined sets (skips delete inside batch)
+--&gt; mwan3_cleanup_orphaned_ipsets() delete user sets not in current config
+--&gt; mwan3_set_dynamic/connected/custom sets
+--&gt; config_foreach mwan3_rebuild_iface_nft (checks ubus for up status)
+--&gt; mwan3_set_general_nft()
+--&gt; mwan3_set_policies_nft()
+--&gt; mwan3_set_user_rules()
+--&gt; mwan3_nft_reload_commit() commit all as single atomic kernel transaction
+--&gt; mwan3_update_iface_to_table() (outside nft)
+--&gt; mwan3_set_general_rules() (outside nft)
+--&gt; mwan3_write_dnsmasq_fragments() restart dnsmasq only if fragment content changed
+--&gt; [if MWAN3_NEED_DNSMASQ_HUP=1] mwan3_dnsmasq_hup() repopulate domain sets lost on flag change
+--&gt; [tracker count check] if mismatch: stop + start (procd cannot add/remove
service instances in reload_service)</code></pre>
<p>User-defined sets (config ipset) and sticky sets (<code>mwan3_sticky_*</code>) survive the reload intact - they are never in the preamble delete path.</p>
<h3 id="103-interface-up-hotplug">10.3 Interface Up (hotplug)</h3>
<pre><code>netifd signals ifup for $INTERFACE
+--&gt; 25-mwan3 hotplug script
+--&gt; mwan3_update_peer_track_ip() write gateway IP (if track_gateway)
+--&gt; mwan3_create_iface_nft() create/flush chain, add rules
+--&gt; mwan3_create_iface_rules() ip rule add (iif, fwmark, unreachable)
+--&gt; mwan3_create_iface_route() copy routes to per-iface table
+--&gt; mwan3_set_iface_hotplug_state "online/offline"
+--&gt; [if not init startup:]
| +--&gt; mwan3_set_general_rules() ensure ip rules exist
| +--&gt; [if online:] mwan3_set_policies_nft() rebuild policy chains
| mwan3_flush_unreplied_conntrack() flush UNREPLIED entries
+--&gt; procd_send_signal track_$INTERFACE USR2</code></pre>
<pre><code>netifd signals connected for $INTERFACE
+--&gt; 25-mwan3 hotplug script
+--&gt; mwan3_set_iface_hotplug_state "online"
+--&gt; mwan3_create_iface_nft() flush chain, update rules
+--&gt; mwan3_set_policies_nft() rebuild policy chains
+--&gt; mwan3_flush_unreplied_conntrack() flush UNREPLIED entries</code></pre>
<p>The <code>connected</code> event fires when an interface completes its link-layer negotiation (e.g., PPPoE authentication) after the <code>ifup</code> routing state is already in place. It skips rule and route creation (those are done on <code>ifup</code>) and goes straight to policy rebuild and the unreplied flush.</p>
<h4 id="automatic-gateway-tracking-trackgateway">Automatic Gateway Tracking (<code>track_gateway</code>)</h4>
<p>When <code>option track_gateway '1'</code> is set on a <code>config interface</code> section, mwan3 automatically discovers the point-to-point peer/gateway IP and adds it to the tracking list at runtime. This is useful for PPPoE and other point-to-point links where the next-hop gateway IP changes on each connection and is not known in advance. Without this option, users must manually configure static <code>track_ip</code> addresses (typically public DNS servers) that may not test the actual link peer.</p>
<p><code>mwan3_update_peer_track_ip()</code> queries <code>ifstatus</code> for the interface's <code>ptpaddress</code> field (the point-to-point peer IP). If found, the gateway IP is written to <code>$MWAN3TRACK_STATUS_DIR/&lt;iface&gt;/GATEWAY</code>. <code>mwan3track</code> reads this file and prepends the gateway IP to the front of the probe list, ensuring it is always probed first on every round regardless of the <code>reliability</code> threshold; static <code>track_ip</code> entries follow after the gateway. On interface bounce, the hotplug <code>ifup</code> action calls <code>mwan3_update_peer_track_ip()</code> again, overwriting the state file with the new peer IP.</p>
<p>The gateway IP is stored as ephemeral state rather than committed to UCI, preventing stale IP accumulation across reboots or gateway changes. An interface definition may specify only <code>track_gateway</code> and omit static tracking IPs entirely. The option is silently ignored if no next-hop peer address is found (e.g., on Ethernet WAN interfaces).</p>
<div class="warn">
<p><strong>IPv4 only in practice.</strong> For IPv6 point-to-point links, the peer is typically a link-local address (e.g., <code>fe80::1</code>). Pinging link-local addresses requires interface scope specification (<code>ping6 fe80::1%pppoe-wan</code>), which mwan3track's probe mechanism does not handle. The option will be silently ignored if no peer address is found.</p>
</div>
<p><strong>UCI Configuration:</strong></p>
<pre><code>config interface 'wan'
option enabled '1'
option track_gateway '1'
# track_ip entries are optional when track_gateway is used
# list track_ip '8.8.8.8'</code></pre>
<p>The corresponding LuCI control is the "Track gateway" checkbox in the Interface tab (see <a href="#1512-interface">Section 15.1.2</a>).</p>
<h3 id="104-interface-down-hotplug">10.4 Interface Down (hotplug)</h3>
<pre><code>netifd signals ifdown for $INTERFACE
+--&gt; 25-mwan3 hotplug script
+--&gt; mwan3_set_iface_hotplug_state "offline"
+--&gt; mwan3_delete_iface_map_entries() flush sticky sets for this iface
+--&gt; mwan3_delete_iface_rules() ip rule del
+--&gt; mwan3_delete_iface_route() ip route flush table
+--&gt; mwan3_delete_iface_nft() remove chain + jump rule
+--&gt; procd_send_signal track_$INTERFACE USR1
+--&gt; mwan3_set_policies_nft() rebuild policies (failover)
+--&gt; mwan3_flush_conntrack() flush conntrack entries for this iface's fwmark</code></pre>
<h3 id="105-stop">10.5 Stop</h3>
<pre><code>/etc/init.d/mwan3 stop
+--&gt; service_running || exit 0
+--&gt; mwan3_interface_shutdown per interface trigger ifdown hotplug
+--&gt; flush ip routing tables (1..MWAN3_INTERFACE_MAX)
+--&gt; delete ip rules in configured ranges
+--&gt; flush ALL mwan3_* chains (rules removed, skeleton chains kept)
+--&gt; delete dynamic chains (iface_in_*, policy_*, etc.)
+--&gt; final safety flush of skeleton chains
+--&gt; flush ALL mwan3_* sets; flush and delete sticky maps
+--&gt; $NFT delete table inet mwan3 (complete removal)
+--&gt; rm -rf status dirs
Result: table inet mwan3 no longer exists. Clean slate.</code></pre>
<h3 id="106-conntrack-management">10.6 Conntrack Management</h3>
<p>mwan3 uses the Linux connection tracking table (conntrack) to persist routing decisions across the packets of a single connection. On the first packet of a new flow, mwan3 stamps the packet with an interface fwmark and saves that mark into the conntrack entry's ct mark field. All subsequent packets in the same connection restore the ct mark back to the packet mark at the start of the prerouting/output chain, bypassing policy re-evaluation entirely. This means that routing policy changes do not take effect for established connections until their conntrack entries are removed.</p>
<p><strong>What mwan3 does not do</strong> is often as important as what it does:</p>
<ul>
<li><strong>Stop</strong> performs no conntrack flush. All conntrack entries survive <code>mwan3 stop</code> intact. The marks are inert while the service is down since the nft table no longer exists to restore them. On restart with an unchanged config, interface IDs are assigned in the same order, so the saved marks remain valid and established connections survive the stop+start cycle without disruption. Flushing on stop would destroy that benefit for no practical gain.</li>
</ul>
<ul>
<li><strong>Reload</strong> also performs no conntrack flush. Existing connections keep their saved ct marks and continue routing via whatever interface they were assigned at classification time. New connections are classified immediately under the new rules. This is intentional: a conntrack flush on every UCI commit would disrupt all active connections. The tradeoff is that existing flows are not rerouted when policy rules change. If that behaviour is needed, a stop followed by start (which does flush; see below) is required.</li>
</ul>
<p><strong>Conntrack actions mwan3 does take:</strong></p>
<p><strong>1. Start - zero-mark flush (<code>mwan3_flush_stale_conntrack</code>)</strong></p>
<p>At the end of <code>start_service</code>, after all nft rules are in place, mwan3 runs:</p>
<pre><code>mwan3ct flush --mark 0x0/$MMX_MASK</code></pre>
<p>This removes every conntrack entry whose mwan3 mark bits are all zero. The target is entries created during the mwan3 startup window: the skeleton loads the hook chains immediately, making them live, but those chains are empty until <code>mwan3_set_general_nft</code> populates them at the end of the start sequence. Connections established during that window pass through prerouting with no classification rules present and are recorded in conntrack with ct mark=0. WireGuard's PersistentKeepalive refreshes the entry every 25 seconds, so without this flush the entry can survive indefinitely, causing the traffic to miss the mwan3 mark restore and fall through to the main routing table.</p>
<p>Only zero-mark entries are removed. Correctly-marked active connections are untouched.</p>
<p><strong>2. Start - flow offloading flush</strong></p>
<p>If software flow offloading is enabled (<code>uci get firewall.@defaults[0].flow_offloading</code> is <code>1</code>), mwan3 flushes the entire conntrack table immediately after the zero-mark flush:</p>
<pre><code>echo f &gt; /proc/net/nf_conntrack</code></pre>
<p>When flow offloading is active, the kernel fast-paths established flows directly through the flowtable, bypassing nftables entirely. Any flow that was offloaded before mwan3 started will never see the mwan3 prerouting chain and will never get a mark. Flushing everything forces all connections to re-enter the nftables pipeline and be classified correctly.</p>
<p>This does not apply to hardware flow offloading, which uses different kernel mechanisms and is not affected by writing to <code>/proc/net/nf_conntrack</code>.</p>
<p><strong>3. Interface down - selective flush (<code>mwan3_flush_conntrack</code>)</strong></p>
<p>Immediately after <code>mwan3_set_policies_nft</code> rebuilds the policies for a failing interface, the hotplug script calls <code>mwan3_flush_conntrack</code>. This function has two independent paths that both run on <code>ifdown</code>:</p>
<p>First, the UCI <code>flush_conntrack</code> path: if the per-interface <code>flush_conntrack</code> UCI option contains <code>ifdown</code>, mwan3 writes <code>f</code> to <code>/proc/net/nf_conntrack</code>, flushing the entire global conntrack table. This is the legacy behaviour, retained for users who need it. Because it disrupts all connections on all interfaces simultaneously, it is not set by default.</p>
<p>Second, the selective flush path, which always runs on <code>ifdown</code> regardless of the UCI option:</p>
<pre><code>mwan3ct flush --mark &lt;iface_mark&gt;/$MMX_MASK</code></pre>
<p>This removes only the conntrack entries that belong to the failing interface (those whose masked mark bits equal that interface's assigned fwmark). The effect is that TCP and UDP flows that were using the failed WAN immediately re-establish under the new failover policy rather than waiting out a TCP retransmit timeout (typically 15-30 seconds). Connections on healthy interfaces are completely undisturbed.</p>
<p><strong>4. Interface up - UCI flush only</strong></p>
<p>On <code>ifup</code> the <code>mwan3_flush_conntrack</code> function runs the same UCI <code>flush_conntrack</code> check: if the option contains <code>ifup</code>, the entire conntrack table is flushed. New connections will be classified correctly by the rebuilt chains; existing connections that survived will re-use their stored ct marks as before.</p>
<p><strong>5. Interface online - unreplied flush (<code>mwan3_flush_unreplied_conntrack</code>)</strong></p>
<p>When an interface transitions to online - either via the <code>connected</code> action (mwan3track declares the interface online) or via <code>ifup</code> when <code>status=online</code> (interface comes up with <code>initial_state</code> set to online) - mwan3 runs:</p>
<pre><code>mwan3ct flush --mark-any $MMX_MASK --status 0/0x2</code></pre>
<p>This removes all mwan3-marked conntrack entries that have never received a reply (<code>IPS_SEEN_REPLY</code> not set). These UNREPLIED entries can result from a race during startup or policy rebuild where a flow gets classified before the nft chains reflect the current state. The stale mark pins traffic to the wrong WAN; because no reply arrives the application keeps retrying, refreshing the conntrack timeout indefinitely. Flushing only UNREPLIED entries is non-disruptive to established connections. The next packet from the affected application creates a fresh conntrack entry that is classified under the current policy. This flush is not performed during init - the zero-mark flush in <code>start_service</code> handles the startup window.</p>
<p><strong>Summary table</strong></p>
<table>
<thead><tr><th>Event</th><th>Selective (per-iface) flush</th><th>Unreplied flush</th><th>Global flush</th><th>Zero-mark flush</th></tr></thead><tbody>
<tr><td><code>start</code></td><td>-</td><td>-</td><td>if flow_offloading</td><td>yes (always)</td></tr>
<tr><td><code>reload</code></td><td>-</td><td>-</td><td>-</td><td>-</td></tr>
<tr><td><code>ifdown</code></td><td>yes (always)</td><td>-</td><td>if <code>flush_conntrack</code> includes <code>ifdown</code></td><td>-</td></tr>
<tr><td><code>ifup</code></td><td>-</td><td>if <code>status=online</code></td><td>if <code>flush_conntrack</code> includes <code>ifup</code></td><td>-</td></tr>
<tr><td><code>connected</code></td><td>-</td><td>yes (always)</td><td>-</td><td>-</td></tr>
<tr><td><code>stop</code></td><td>-</td><td>-</td><td>-</td><td>-</td></tr>
</tbody></table>
<p><strong>Implications for users</strong></p>
<p>Changing mwan3 policy (via <code>reload</code>) does not reroute existing TCP connections. If a load-balance rule changes from 50/50 to 80/20, existing flows continue on whatever interface they were originally assigned. This is usually the right behaviour - rerouting mid-session would break most protocols - but means the new distribution only takes effect as connections naturally expire and re-establish.</p>
<p>The selective ifdown flush is the most operationally important flush. Its effect depends on the protocol.</p>
<p>For UDP-based protocols that identify connections by something other than the IP 4-tuple - WireGuard being the primary example - the flush allows transparent failover. WireGuard authenticates peers by cryptographic identity rather than source address. When the conntrack entry is removed and the next keepalive is reclassified via the failover WAN, the packet arrives at the remote peer from a new source IP, the peer authenticates it by key, updates its roaming endpoint, and the tunnel continues without interruption.</p>
<p>For TCP connections - file downloads, SSH sessions, browser connections - the picture is different. TCP identifies connections by the 4-tuple including source IP. When packets exit via the failover WAN, masquerade applies the new WAN's IP, and the remote end has no connection matching that source. It sends RST and the connection terminates. The connection was already dead the moment the WAN failed; what the flush controls is how quickly the application finds out. Without the flush, packets are silently blackholed (the old interface's ip rule no longer exists) and the application hangs in retransmit limbo until the TCP timeout expires - potentially minutes. With the flush, the RST arrives immediately and the application can reconnect at once.</p>
<p>The zero-mark flush on start is defensive infrastructure against a corner case that most users never notice explicitly. Its absence manifests as a WireGuard tunnel that routes through the wrong interface after a reboot, or a DNAT connection that falls to the main routing table immediately after mwan3 restarts following an fw4 reload.</p>
<p><strong>Custom behaviour via mwan3.user</strong></p>
<p>The default conntrack management fits most deployments. Three cases where additional action on interface recovery may be useful or desirable for some are:</p>
<ul>
<li><strong>Load-balanced configurations:</strong> when a failed WAN recovers, the automatic ifdown flush has already cleared entries for the failed interface, but all surviving flows remain pinned to the WANs that carried the load during the outage. They will not redistribute until they expire naturally.</li>
</ul>
<ul>
<li><strong>Failover with a preferred primary:</strong> when the primary WAN recovers, existing connections stay on the failover WAN until natural expiry rather than snapping back. In cost-sensitive setups where the primary is cheaper or unmetered, lingering on the failover WAN has a direct cost.</li>
</ul>
<ul>
<li><strong>Application of new policies / rules to <em>all</em> flows immediately:</strong> a config save will cause all <em>new</em> connections be be immediately routed according to the new policy / ruleset, but will leave existing connections intact rather than interrupting them. Users can choose to override that behaviour and make the new policy take effect immediately even for existing connections, but at the cost of interrupting connections such as SSH, file downloads, etc.</li>
</ul>
<p>In both cases, flushing all mwan3-marked conntrack entries on <code>ifup</code> causes connections to be reclassified immediately and routed according to the current policy:</p>
<pre><code>#!/bin/sh
[ "$ACTION" = "ifup" ] || exit 0
. /lib/functions.sh
. /lib/mwan3/common.sh
. /lib/mwan3/mwan3.sh
config_load mwan3
mwan3_init
mwan3_flush_marked_conntrack</code></pre>
<p>Note that the flush is broader than strictly necessary, although it's still a lot less of a hammer than <code>conntrack -F</code>. Since the recovering interface has no conntrack entries (they were removed on ifdown), the flush affects connections on all other interfaces, including any that were healthy throughout the outage. There is no way to target only those connections that moved as a result of the failure. The trade-off is that every connection is briefly interrupted on any interface recovery event. For most TCP connections the interruption is imperceptible; long-lived sessions such as SSH or active file transfers will be reset.</p>
<p>For this reason, default behaviour is not to flush all marked conntrack entries but to allow this action to fall-back on a per-installation level to <code>mwan3.user</code> through use of the <code>mwan3_flush_marked_conntrack</code> supplied expressly for this purpose.</p>
<hr>
<h2 id="11-atomic-non-destructive-reload">11. Atomic Non-destructive Reload</h2>
<p><code>reload_service</code> is a single atomic <code>nft -f</code> batch that rebuilds the entire ruleset while the old one is still serving traffic, committing in one kernel transaction with zero traffic disruption window.</p>
<h3 id="mwan3batchdepth-counter">MWAN3_BATCH_DEPTH Counter</h3>
<p><code>MWAN3_BATCH_DEPTH</code> is an integer counter (default 0) in <code>common.sh</code>. It enables nested batch accumulation:</p>
<ul>
<li><code>mwan3_nft_batch_start</code>: increments depth; truncates batch file only at depth 0-&gt;1.</li>
<li><code>mwan3_nft_batch_commit</code>: decrements depth; commits to kernel only at depth 1-&gt;0.</li>
<li><code>mwan3_nft_exec</code>: routes to <code>mwan3_nft_push</code> when depth &gt; 0, accumulating all operations.</li>
</ul>
<p>This means individual build functions (which each call <code>mwan3_nft_batch_start</code> / <code>mwan3_nft_batch_commit</code> internally) work correctly both standalone (immediate kernel commit) and when called from inside a reload batch (operations accumulate, committed as one transaction at the end).</p>
<h3 id="preamble-mwan3nftreloadstart">Preamble (mwan3_nft_reload_start)</h3>
<p>Opens the outermost batch (depth 0-&gt;1), then writes:</p>
<ol>
<li>Flush all 8 skeleton chains (<code>mwan3_prerouting</code>, <code>mwan3_output</code>, <code>mwan3_postrouting</code>, <code>mwan3_ifaces_in</code>, <code>mwan3_rules</code>, <code>mwan3_connected</code>, <code>mwan3_custom</code>, <code>mwan3_dynamic</code>).</li>
<li>Two-pass flush then delete all dynamic chains (<code>mwan3_iface_in_*</code>, <code>mwan3_policy_*</code>, <code>mwan3_rule_*</code>, <code>mwan3_or_meta_*</code>, <code>mwan3_or_ct_*</code>). Two passes are required because <code>mwan3_or_meta_*</code> chains are referenced by <code>mwan3_rule_*</code> sticky chains; a single alphabetical pass would attempt to delete <code>or_meta</code> before flushing <code>rule</code>, producing "Device or resource busy".</li>
<li>Delete the 6 internal <code>mwan3_*</code> sets so <code>mwan3_ensure_nft_framework</code> recreates them with correct flags.</li>
</ol>
<p><strong>What is NOT in the preamble:</strong> user-defined sets (from <code>config ipset</code>), sticky sets (<code>mwan3_sticky_*</code>). These survive the reload intact, preserving dnsmasq-populated addresses and sticky routing state.</p>
<h3 id="batch-guards-in-build-functions">Batch Guards in Build Functions</h3>
<p>Six locations in build functions query kernel state (chain/set existence) that are bypassed when inside a batch (<code>MWAN3_BATCH_DEPTH &gt; 0</code>), because the kernel still shows the pre-preamble state until the batch commits:</p>
<ul>
<li><code>mwan3_set_general_nft</code> idempotency guard</li>
<li><code>mwan3_create_iface_nft</code> SNAT loop / jump rule check</li>
<li><code>mwan3_create_policies_nft</code> (unconditional add+flush inside batch)</li>
<li><code>mwan3_set_policies_nft</code> orphan cleanup</li>
<li><code>mwan3_ensure_nft_framework</code> delete set loop</li>
<li><code>_mwan3_render_one_ipset</code> delete set (skipped inside batch when flags are unchanged, to preserve dnsmasq-populated elements; queued inside batch when flags differ)</li>
</ul>
<h3 id="commit">Commit</h3>
<p><code>mwan3_nft_reload_commit</code> calls <code>mwan3_nft_batch_commit</code>. At depth 1-&gt;0 the entire accumulated batch is submitted to the kernel as a single transaction. If <code>nft -f</code> returns non-zero, the kernel rolls back completely and the old ruleset continues serving traffic. No partial states are possible.</p>
<hr>
<h2 id="12-user-defined-nft-sets">12. User-defined nft Sets</h2>
<p><code>config ipset</code> sections in <code>/etc/config/mwan3</code> create named nft sets in <code>table inet mwan3</code>. These sets can be referenced in rules via the <code>ipset</code> (destination) and <code>ipset_src</code> (source) UCI options.</p>
<h3 id="uci-configuration">UCI Configuration</h3>
<pre><code>config ipset 'youtube_ipv4'
option name 'youtube_v4'
option family 'ipv4'
option enabled '1'
option maxelem '0' # 0 = unlimited (default)
option timeout '0' # 0 = no timeout (default), seconds
option counters '0' # 0 = no per-element counters (default)
list entry '8.8.8.8'
list entry '203.0.113.0/24'
list domain 'youtube.com'
list domain 'googlevideo.com'
option loadfile '/etc/mwan3/custom-ips.txt'</code></pre>
<h3 id="options">Options</h3>
<table>
<thead><tr><th>Option</th><th>Values</th><th>Purpose</th></tr></thead><tbody>
<tr><td><code>name</code></td><td>string</td><td>nft set name in <code>table inet mwan3</code>. Must be unique.</td></tr>
<tr><td><code>family</code></td><td><code>ipv4</code> or <code>ipv6</code></td><td>Address family. Determines set type (<code>ipv4_addr</code> or <code>ipv6_addr</code>).</td></tr>
<tr><td><code>enabled</code></td><td><code>0</code>/<code>1</code></td><td>Skip this set if <code>0</code>.</td></tr>
<tr><td><code>maxelem</code></td><td>integer</td><td>Maximum elements. <code>0</code> = unlimited (default).</td></tr>
<tr><td><code>timeout</code></td><td>integer</td><td>Per-element timeout in seconds. <code>0</code> = no timeout.</td></tr>
<tr><td><code>counters</code></td><td><code>0</code>/<code>1</code></td><td>Enable per-element packet/byte counter tracking.</td></tr>
<tr><td><code>list entry</code></td><td>CIDR or address</td><td>Inline address/CIDR added at startup.</td></tr>
<tr><td><code>list domain</code></td><td>domain name</td><td>Causes mwan3 to write a dnsmasq <code>nftset=</code> confdir fragment for this domain. dnsmasq populates the set via DNS resolution.</td></tr>
<tr><td><code>loadfile</code></td><td>path</td><td>File containing addresses/CIDRs, one per line. Loaded at startup.</td></tr>
</tbody></table>
<h3 id="counter-syntax">counter Syntax</h3>
<p><code>counter</code> is a <strong>standalone nft set statement</strong>, NOT a flag keyword. It appears after the type/flags declarations on its own line:</p>
<pre><code>set youtube_v4 {
type ipv4_addr
flags interval
auto-merge
counter
}</code></pre>
<p>Placing <code>counter</code> inside the <code>flags</code> list causes an nft parse error.</p>
<h3 id="dnsmasq-integration">dnsmasq Integration</h3>
<p>For each <code>list domain</code> entry, <code>mwan3_write_dnsmasq_fragments</code> writes a dnsmasq confdir fragment:</p>
<pre><code>nftset=/youtube.com,googlevideo.com/4#inet#mwan3#youtube_v4</code></pre>
<p>The fragment restarts dnsmasq only when its content has changed (comparing against the previously written fragment). On the next DNS query for the domain, dnsmasq adds the resolved address to the nft set automatically. A <code>mwan3_dnsmasq_hup</code> call from <code>start_service</code> forces initial resolution. <code>reload_service</code> also calls <code>mwan3_dnsmasq_hup</code> when <code>MWAN3_NEED_DNSMASQ_HUP</code> is set -- see Flag Changes below.</p>
<h3 id="flag-changes">Flag Changes</h3>
<p><code>nft add set</code> is idempotent and does NOT update flags (type, counter, timeout, size) on an existing set. When a set's flags are changed via LuCI or UCI command and the config is applied, <code>_mwan3_ipset_needs_delete</code> detects the mismatch by querying the live kernel set and comparing against the desired spec. If any flag differs, the set is queued for deletion inside the reload batch before being recreated with the new flags. The reload batch preamble has already flushed <code>mwan3_rules</code> (removing all references to user sets) before the delete fires at commit time, so the delete is safe.</p>
<p>If the deleted set had <code>list domain</code> entries, its dnsmasq-populated elements are lost when it is recreated empty. <code>_mwan3_render_one_ipset</code> sets <code>MWAN3_NEED_DNSMASQ_HUP=1</code> in this case. After <code>mwan3_write_dnsmasq_fragments</code> completes in <code>reload_service</code>, a <code>mwan3_dnsmasq_hup</code> call is issued to repopulate the set via DNS re-resolution.</p>
<h3 id="migration-from-fw4-ipsets">Migration from fw4 ipsets</h3>
<p><code>mwan3-migrate-ipset-v4.sh</code> (a one-shot script run from postinst and immediately deleted) copies <code>config ipset</code> sections from <code>/etc/config/firewall</code> to <code>/etc/config/mwan3</code> if any of the sets in <code>/etc/config/firewall</code> are referenced in an mwan3 user rule. The <code>match</code> UCI option required by fw4 is silently ignored by mwan3 (mwan3 does not call <code>config_get match</code>).</p>
<h3 id="orphan-cleanup">Orphan Cleanup</h3>
<p><code>mwan3_cleanup_orphaned_ipsets</code> is called in <code>reload_service</code> after <code>mwan3_render_config_ipsets</code>. It queries <code>nft list table inet mwan3</code> for user-defined sets (those not prefixed <code>mwan3_</code>), compares against currently configured set names, and emits <code>delete set</code> for any orphans. This prevents stale sets accumulating when a set is removed from UCI config via LuCI.</p>
<h3 id="rule-integration-ipset-and-ipsetsrc">Rule Integration (<code>ipset</code> and <code>ipset_src</code>)</h3>
<p>The <code>ipset</code> UCI option on a rule section matches the destination address against a named nft set. The <code>ipset_src</code> option provides the complementary source address match. Both can be set on the same rule and are ANDed together.</p>
<h4 id="uci-example">UCI Example</h4>
<pre><code>config rule 'corp_to_wan2'
option ipset_src corp_clients
option ipset blocked_dests
option use_policy wan2_policy</code></pre>
<p>This generates:</p>
<pre><code>ip saddr @corp_clients ip daddr @blocked_dests meta mark &amp; 0x3f00 == 0 jump mwan3_policy_wan2_policy</code></pre>
<p>If the named set does not yet exist in <code>table inet mwan3</code> at rule-install time, <code>mwan3_set_user_nft_rule()</code> pre-creates it with the appropriate type to prevent the nft batch from failing atomically. A present <code>ipset_src</code> is treated as an implicit family qualifier by the <code>meta nfproto</code> guard condition in <code>mwan3_set_user_nft_rule()</code>, consistent with <code>src_ip</code>, <code>dest_ip</code>, and <code>ipset</code>.</p>
<hr>
<h2 id="13-unchanged-files">13. Unchanged Files</h2>
<table>
<thead><tr><th>File</th><th>Reason</th></tr></thead><tbody>
<tr><td><code>etc/hotplug.d/iface/26-mwan3-user</code></td><td>Just calls <code>/etc/mwan3.user</code>, no firewall code</td></tr>
<tr><td><code>etc/mwan3.user</code></td><td>User script template, no firewall code</td></tr>
<tr><td><code>etc/uci-defaults/mwan3-migrate-flush_conntrack</code></td><td>UCI migration, no firewall code</td></tr>
</tbody></table>
<hr>
<h2 id="14-diagnostic-commands">14. Diagnostic Commands</h2>
<pre><code># Full mwan3 table dump
nft list table inet mwan3
# List specific chain
nft list chain inet mwan3 mwan3_prerouting
nft list chain inet mwan3 mwan3_output
nft list chain inet mwan3 mwan3_postrouting
nft list chain inet mwan3 mwan3_ifaces_in
nft list chain inet mwan3 mwan3_policy_balanced
# Check internal set contents
nft list set inet mwan3 mwan3_connected_v4
nft list set inet mwan3 mwan3_connected_v6
nft list set inet mwan3 mwan3_custom_v4
nft list set inet mwan3 mwan3_dynamic_v4
# Check user-defined sets (config ipset)
nft list set inet mwan3 youtube_v4
nft list table inet mwan3 | grep '^\tset ' # all sets in mwan3 table only
# Check sticky set entries (per-member sets, id suffix = interface id)
nft list set inet mwan3 mwan3_sticky_v4_https_1
nft list set inet mwan3 mwan3_sticky_v6_https_1
# List all mwan3 chains (names only)
# Note: nft list chains only accepts an optional family, not a table name
nft list chains inet | grep mwan3_
# List OR-immediate setter chains (vmap dispatch)
nft list chains inet | grep mwan3_or_
# Verify marks are being set (add temporary counter)
nft add rule inet mwan3 mwan3_prerouting meta mark and 0x3f00 != 0 counter
# Check connmarks
conntrack -L -o mark
# Check ip rules
ip rule list | grep -E '^[1-3][0-9]{3}:'
# Check per-interface routing table (table N for interface N)
ip route list table 1
# JSON output (for scripting/debugging)
nft -j list set inet mwan3 mwan3_connected_v4
nft -j list chain inet mwan3 mwan3_policy_balanced
# Check mwan3 status
mwan3 status
mwan3 internal
# RPC query
ubus call mwan3 status '{"section":"interfaces"}'
ubus call mwan3 status '{"section":"connected"}'
ubus call mwan3 status '{"section":"policies"}'
# Query user-defined set info and members
ubus call mwan3 nftset_info '{}'
ubus call mwan3 nftset_elements '{"set":"youtube_v4","max":200}'</code></pre>
<hr>
<h2 id="15-luci-app-mwan3">15. luci-app-mwan3</h2>
<p><code>luci-app-mwan3</code> is the LuCI web interface for mwan3, installed as a separate package from the core mwan3 daemon. It is organised into two menu groups: <strong>Network &gt; MultiWAN Manager</strong> for configuration and <strong>Status &gt; MultiWAN Manager</strong> for runtime monitoring and diagnostics.</p>
<h3 id="151-network-app">15.1 Network App</h3>
<p>The Network app provides tabs for configuring every aspect of mwan3: global parameters, WAN interfaces, policy members, policies, traffic rules, nftables IP sets, a traffic simulator, a static configuration analyser, and user notification scripts.</p>
<h4 id="1511-globals">15.1.1 Globals</h4>
<p>The Globals tab (<code>globals.js</code>) edits the <code>config globals</code> UCI section.</p>
<p><strong>Mark mask (<code>mmx_mask</code>):</strong> The hexadecimal bitmask used for policy mark operations. Determines the number of bits available for interface encoding, which sets <code>MWAN3_INTERFACE_MAX</code> (the maximum number of simultaneously active interfaces). A live computed display shows the derived interface limit as the value is changed. Default <code>0x3F00</code> (6 bits, 60 interfaces).</p>
<p><strong>Rule base priorities:</strong> Three numeric fields set <code>iif_rule_base</code>, <code>fwmark_rule_base</code>, and <code>unreachable_rule_base</code>. Live cross-field validation enforces both ordering constraints (see <a href="#55-configurable-rule-base-priorities">Section 5.5</a>) on blur; an error is shown if either constraint would be violated. When any rule base priority changes, saving and applying the form triggers <code>mwan3 restart</code> to rebuild all ip rules at the new priorities.</p>
<p><strong>Additional routing tables (<code>rt_table_lookup</code>):</strong> A dynamic list of extra routing table IDs whose routes are merged into each per-interface routing table alongside routes from the main table.</p>
<p><strong>Bypass networks (<code>bypass_network</code>):</strong> A dynamic list of CIDRs added to the connected-network bypass sets (<code>mwan3_connected_v4</code>/<code>mwan3_connected_v6</code>), causing traffic to those networks to skip mwan3 policy routing.</p>
<p><strong>Logging:</strong> A checkbox to enable mwan3 logging and a numeric loglevel field (0-7 following syslog severity levels).</p>
<p><strong>Verbose logging (<code>verbose_logging</code>):</strong> A checkbox that enables debug-level log output from mwan3 daemons. When enabled, the <code>LOG()</code> function emits debug messages to syslog from <code>mwan3.sh</code>, <code>common.sh</code>, and <code>mwan3track</code>. Takes effect on <code>HUP</code>-triggered config reloads without requiring a full restart.</p>
<h4 id="1512-interface">15.1.2 Interface</h4>
<p>The Interface tab (<code>interface.js</code>) presents a <code>GridSection</code> where each row represents a <code>config interface</code> section in the mwan3 UCI config. Clicking a row opens the per-interface modal editor.</p>
<p><strong>Tracking configuration:</strong> The tracking method field supports <code>ping</code> (always available), <code>httping</code>, <code>nping</code>, and <code>arping</code>. The httping, nping, and arping options are only shown if the corresponding binary is detected on the router via <code>fs.stat</code>. The <code>track_ip</code> field accepts a dynamic list of addresses or hostnames to probe. <code>track_method</code>, <code>reliability</code>, <code>count</code>, <code>size</code>, <code>max_ttl</code>, <code>timeout</code>, <code>interval</code>, <code>failure_interval</code>, <code>recovery_interval</code>, <code>keep_failure_interval</code>, <code>down</code>, and <code>up</code> thresholds are all configurable.</p>
<p><strong>Quality tracking:</strong> The <code>check_quality</code> checkbox enables latency and loss threshold monitoring. When enabled, four additional fields appear: <code>failure_latency</code>, <code>failure_loss</code>, <code>recovery_latency</code>, and <code>recovery_loss</code>.</p>
<p><strong>Track gateway (<code>track_gateway</code>):</strong> A checkbox shown only for IPv4 interfaces. When enabled, mwan3 automatically uses the interface's network gateway as the tracking target (see <a href="#automatic-gateway-tracking-track_gateway">Section 10.3</a>).</p>
<p><strong>IPv6 SNAT (<code>snat6</code>):</strong> A text field shown only for IPv6 interfaces. Accepts empty/<code>0</code> (disabled), <code>1</code> (SNAT to the interface's primary global address), or a literal IPv6 address for fixed-source pinning (see <a href="#router-originated-traffic-and-source-address-rewriting">Section 4</a>).</p>
<p><strong>Conntrack flushing (<code>flush_conntrack</code>):</strong> Controls whether the global conntrack table is flushed on interface state changes. Per-interface conntrack entries are flushed automatically on ifdown regardless of this setting.</p>
<p><strong>Validation:</strong> Interface section names must be 15 characters or fewer and unique across all section types in the mwan3 config. The interface's current network metric is shown as a read-only value derived from the netifd network configuration.</p>
<h4 id="1513-member">15.1.3 Member</h4>
<p>The Member tab (<code>member.js</code>) presents a <code>GridSection</code> for <code>config member</code> sections. Each member has three fields: <code>interface</code> (a dropdown of currently configured mwan3 interfaces), <code>metric</code> (1-256), and <code>weight</code> (1-1000). Members are the atomic routing units referenced by policies.</p>
<p>An informational banner at the top of the section directs users to the Policy Builder (see <a href="#1514-policy">Section 15.1.4</a>) for normal member management. The Member tab is preserved for direct inspection and manual adjustment of metric and weight values.</p>
<h4 id="1514-policy">15.1.4 Policy</h4>
<p>The Policy tab (<code>policy.js</code>) presents a <code>GridSection</code> where each row represents a <code>config policy</code> section. The grid shows two read-only derived columns - "IPv4 Priority order" and "IPv6 Priority order" - alongside an editable <code>last_resort</code> column. The derived columns render the policy behaviour in symbolic form: a single interface name for single-interface policies; <code>[iface N%, iface N%, ...]</code> for load-balanced tiers; <code>[primary] --&gt; [failover] --&gt; ...</code> for failover chains.</p>
<p><strong>Policy Builder modal:</strong> Clicking "Add..." or the edit button on an existing policy opens the policy builder. The builder presents the policy as an ordered list of tiers. Tier 1 is labelled "Primary"; subsequent tiers are labelled "Failover (N)". Each tier contains one or more interface entries, each with a family selector (IPv4/IPv6), an interface selector filtered to interfaces of the chosen family, and a percentage share field.</p>
<p>Shares are per-family within each tier: all IPv4 entries in a tier must sum to 100%, and similarly for IPv6. When an entry is added to a family group its share is computed by redistributing 100% equally across all entries in that family group (first entry absorbs any rounding remainder). Changing an entry's family redistributes both the old and new family groups. A single-entry family group has its share field locked at 100 and shown greyed out.</p>
<p><strong>Member management:</strong> On save the builder computes GCD-reduced integer weights from the percentage inputs, then for each (interface, metric, weight) tuple finds an existing member section with exactly those values and reuses it (regardless of name), or creates a new section named <code>&lt;iface&gt;_m&lt;metric&gt;_w&lt;weight&gt;</code>. After saving, any member sections that were in the old <code>use_member</code> list but are no longer referenced by any policy become orphans. A "Delete unused member definitions" checkbox in the page header (persisted in <code>localStorage</code>) controls automatic versus prompted orphan cleanup.</p>
<p><strong>Validation:</strong> Real-time validation reports missing or invalid policy name, name already in use, unselected interface on any entry, and family share totals that do not equal 100%. The Save button is disabled while any error is present.</p>
<h4 id="1515-rule">15.1.5 Rule</h4>
<p>The Rule tab (<code>rule.js</code>) presents a <code>GridSection</code> where each row represents a <code>config rule</code> section. Rules are evaluated in UCI declaration order; order matters.</p>
<p><strong>Fields:</strong> <code>family</code> (ipv4/ipv6/any), <code>proto</code> (all/tcp/udp/icmp/icmpv6/...), <code>src_ip</code>, <code>src_port</code>, <code>dest_ip</code>, <code>dest_port</code>, <code>fwmark</code>, <code>sticky</code>, <code>timeout</code>, <code>ipset_src</code> (source NFT set), <code>ipset</code> (destination NFT set), <code>logging</code>, <code>use_policy</code>, and <code>enabled</code>.</p>
<p><strong>NFT set dropdowns:</strong> The <code>ipset_src</code> and <code>ipset</code> fields are populated from the <code>mwan3.nftset_info</code> rpcd method. Disabled sets are excluded from the dropdowns. A destination set must match the rule's selected address family; combining a set with a same-dimension IP field (<code>ipset</code> with <code>dest_ip</code>, or <code>ipset_src</code> with <code>src_ip</code>) is flagged as an error since both constrain the same traffic dimension. A cross-set family mismatch between <code>ipset_src</code> and <code>ipset</code> on the same rule is also flagged.</p>
<p><strong>Fwmark field:</strong> The fwmark field renders as a combined <code>value/mask</code> input. On read, the stored <code>fwmark</code> and <code>fwmask</code> UCI options are combined; on write they are split back. Validation checks hex format and flags any overlap between the user mask and <code>MMX_MASK</code>.</p>
<p><strong>Column merging:</strong> The grid merges source port into the Source column and destination port into the Destination column. The merged cell renders as <code>address:port</code> or <code>set_name:port</code> when both are set, <code>address</code> or <code>set_name</code> alone when only an address or set is present, <code>*:port</code> when only a port is set, and <code>-</code> when neither is set. <code>src_port</code> and <code>dest_port</code> are <code>modalonly</code>, remaining fully editable in the modal.</p>
<p><strong>Enable column:</strong> An inline Enable checkbox in the grid uses <code>o.editable = true</code> for direct toggle without opening the modal. The checkbox is rendered non-interactive when the rule is disabled and references at least one currently disabled set, preventing re-enable into a broken state.</p>
<p><strong>Address family validation:</strong> <code>src_ip</code> and <code>dest_ip</code> accept a single address or CIDR, or a comma-separated list of addresses and CIDRs. Each element in a comma-separated list is individually validated for format and family consistency. A list containing a mix of IPv4 and IPv6 elements is rejected. Family consistency between the rule and any referenced NFT set is checked on save.</p>
<h4 id="1516-ip-sets">15.1.6 IP Sets</h4>
<p>The IP Sets configuration tab (<code>ipset.js</code>) presents a <code>GridSection</code> for <code>config ipset</code> sections.</p>
<table>
<thead><tr><th>Field</th><th>UCI option</th><th>Description</th></tr></thead><tbody>
<tr><td>Name</td><td><code>name</code></td><td>Identifier referenced in rule <code>ipset</code>/<code>ipset_src</code> fields. Must start with a letter or <code>_</code>; may contain letters, digits, <code>_</code>, <code>.</code>, <code>-</code>. Names beginning with <code>mwan3_</code> are reserved.</td></tr>
<tr><td>Family</td><td><code>family</code></td><td><code>ipv4</code> or <code>ipv6</code>. Sets the nftables element type (<code>ipv4_addr</code> or <code>ipv6_addr</code>).</td></tr>
<tr><td>IPs / Networks</td><td><code>entry</code></td><td>Static addresses or CIDR subnets. Validated against the selected family on blur.</td></tr>
<tr><td>Domains</td><td><code>domain</code></td><td>Domain names resolved by dnsmasq and added to the set at runtime. mwan3 generates corresponding <code>nftset=</code> fragments in the dnsmasq config.</td></tr>
<tr><td>Include File</td><td><code>loadfile</code></td><td>Path to a file of addresses or CIDRs (one per line, <code>#</code> comments ignored) loaded at service start.</td></tr>
<tr><td>Max Entries</td><td><code>maxelem</code></td><td>Maximum element count. Empty means no limit.</td></tr>
<tr><td>Timeout</td><td><code>timeout</code></td><td>Per-element lifetime in seconds. <code>0</code> means entries do not expire.</td></tr>
<tr><td>Counters</td><td><code>counters</code></td><td>Enables per-element packet and byte counters in the nftables set.</td></tr>
<tr><td>Enabled</td><td><code>enabled</code></td><td>Whether the set is created and populated at service start.</td></tr>
</tbody></table>
<p>Deletion of a set is blocked in the UI if any rule references the set name in its <code>ipset</code> or <code>ipset_src</code> field; the user is shown which rules reference the set. The <code>enabled</code> checkbox is rendered non-interactive when the set is referenced by at least one enabled rule, preventing the set from being disabled while a rule depends on it.</p>
<p>The <code>entry</code> and <code>loadfile</code> fields define the static content restored by the Reload button on the IP Sets status tab. The <code>domain</code> fields define the domains queried by the Resolve button; the Resolve button is only shown for sets that have at least one domain entry.</p>
<h4 id="1517-simulator">15.1.7 Simulator</h4>
<p>The Traffic Path Simulator (<code>simulator.js</code>) accepts source IP or hostname, destination IP or hostname, fwmark (hex), protocol, source port, destination port, and address family, then reports which mwan3 rule would match and which policy would handle that traffic.</p>
<p><strong>Hostname resolution:</strong> If a hostname is entered in either IP field, the simulator calls <code>mwan3.resolve_host</code> before running the simulation. This invokes <code>busybox nslookup</code> against 127.0.0.1, which has the side effect of populating any dnsmasq nftset entries configured for that domain. Resolved addresses are shown inline below the field. If resolution fails or returns no addresses for the selected family the simulation is aborted with an error message. When family is "IPv4 and IPv6", IPv4 addresses are preferred. Strings that are not valid IP addresses and do not resemble a valid hostname are rejected before simulation. IP address inputs are validated before the simulation runs; malformed addresses are rejected.</p>
<p><strong>Port fields:</strong> Source and destination port fields are shown only when the protocol is set to TCP or UDP. All fields except protocol are optional; a blank field means "any" and rules with a constraint on that field will not match. The fwmark field accepts a hex value and is matched against rules that have an <code>fwmark</code>/<code>fwmask</code> constraint; a blank fwmark is treated as <code>0x0</code> (unmarked packet).</p>
<p><strong>Simulation logic:</strong> On each simulate press, UCI configuration, live policy state, and the connected-network sets (<code>mwan3_connected_v4</code>/<code>mwan3_connected_v6</code>) are reloaded fresh. The simulator first checks whether the destination falls in a directly-connected network and, if so, reports that mwan3 rules are bypassed. Otherwise it evaluates each enabled UCI rule in declaration order, fetching live nftset membership via <code>mwan3.nftset_members</code> for any sets referenced by those rules. Elements in counter-enabled sets are correctly unwrapped from the <code>{"elem":{"val":"...","counter":{...}}}</code> JSON structure used by nft. The first matching rule is the result; any additional rules that also match are listed as shadowed rules.</p>
<h4 id="1518-configuration">15.1.8 Configuration</h4>
<p>The Configuration tab (<code>configuration.js</code>) performs a static analysis of the mwan3 UCI configuration with no live system state consulted. It loads the current UCI config, builds lookup tables for interfaces, members, policies, and rules, then reports issues at two severity levels: <strong>error</strong> (a broken reference or empty policy that will cause traffic to be silently misrouted or blackholed) and <strong>warning</strong> (an orphaned object that is defined but has no effect).</p>
<p><strong>Checks performed:</strong></p>
<ul>
<li>Member references an undefined interface</li>
<li>Member is defined but not used by any policy</li>
<li>Policy has no members</li>
<li>Policy references an undefined member</li>
<li>Policy has multiple members that all reference the same interface (no real redundancy; failover will not occur if that interface goes down)</li>
<li>Policy is defined but not used by any rule</li>
<li>Rule references an undefined policy</li>
<li>Rule is shadowed by an earlier rule (the earlier rule matches a superset of the later rule's traffic, so the later rule is unreachable)</li>
<li>Interface is defined but not referenced by any member</li>
</ul>
<p><strong>Rule shadowing check:</strong> <code>ruleAContainsB()</code> determines whether rule A (earlier) is a superset of rule B (later). The check is deliberately conservative: it only flags clear containment. CIDR containment is computed precisely for both IPv4 (32-bit unsigned arithmetic) and IPv6 (BigInt). Port specs use conservative matching: a rule with no port restriction contains any other; two rules with non-empty port specs are only flagged as contained when the specs are identical strings. When both rules reference an NFT set, the set names are compared: if both rules reference the same nftset the shadowing check proceeds normally; if the set names differ the check is skipped conservatively since set membership cannot be evaluated statically. If only one rule uses a set the check is also skipped. Address family is respected: an IPv4-only rule does not shadow an IPv6-only rule.</p>
<h4 id="1519-notify">15.1.9 Notify</h4>
<p>The Notify tab (<code>notify.js</code>) provides a raw textarea editor for <code>/etc/mwan3.user</code>. This file is sourced by the mwan3 hotplug handler at the end of interface up/down events, allowing site-specific shell commands to run in response to WAN state changes. The tab reads the file via <code>fs.read</code> on load and writes it back via <code>fs.write</code> on save; no UCI is involved.</p>
<h3 id="152-status-app">15.2 Status App</h3>
<p>The Status app provides live monitoring and diagnostic tools. All tabs that show runtime data poll automatically via the LuCI poll mechanism.</p>
<h4 id="1521-overview">15.2.1 Overview</h4>
<p>The Overview tab (<code>overview.js</code>) polls <code>mwan3.status</code> and displays a three-section summary of all mwan3 interface states, active policies, and configured rules.</p>
<p><strong>Interfaces section:</strong> A CSS grid (<code>repeat(auto-fill, 13em)</code>) of per-interface status cards. Each card has a coloured border (green/red/orange/grey by status) and shows the interface name, status label, and uptime or online/offline duration. Duration is rendered without seconds by <code>formatDuration()</code> (days, hours, minutes) to keep the display stable under polling.</p>
<p><strong>Policies section:</strong> A CSS grid of per-policy cards in the same bordered card style. Each card lists its member interfaces with live traffic share percentages. Member lines use <code>white-space: nowrap</code>. A member's line is coloured green when its share is nonzero, yellow when it is online but carrying zero percent (standby or failover member not currently active), and grey when offline.</p>
<p><strong>Rules section:</strong> A table of enabled rules. The Match column summarises the rule in compact <code>addr:port</code> format by <code>fmtAddr()</code>: <code>address:port</code> when both are set, <code>address</code> alone, <code>*:port</code> for port-only, or <code>(all traffic)</code> for unconstrained rules. Both source (<code>ipset_src</code>) and destination (<code>ipset</code>) NFT sets are shown where configured.</p>
<h4 id="1522-interface-status">15.2.2 Interface Status</h4>
<p>The Interface Status tab (<code>detail.js</code>) provides per-interface tracking detail, live-polled from <code>mwan3.status</code>. An interface selector at the top of the page drives the display.</p>
<p>For the selected interface, the tab shows a header card with status, tracking method, and composite reliability score, followed by a table of individual tracking targets. Each row shows the target address, probe status (up/down/skipped), current latency, and current loss percentage. When <code>check_quality</code> is disabled, the latency and loss columns show "Not enabled". Down targets show an infinity symbol for latency. Rows are sorted: up targets first, then down, then skipped.</p>
<h4 id="1523-routing-health">15.2.3 Routing Health</h4>
<p>The Routing Health tab (<code>routing.js</code>) polls <code>mwan3.routing_health</code> and performs a live sanity check of the ip rules and routing tables mwan3 maintains for each configured interface.</p>
<p><strong>Per-interface cards:</strong> Each card shows the interface name, sequential index, and mwan3track-reported status, followed by four status rows:</p>
<ul>
<li><strong>IP rule (iif)</strong> at priority <code>iif_base + index</code>: expected present when the interface is online.</li>
<li><strong>IP rule (fwmark)</strong> at priority <code>fwmark_base + index</code>: expected present when the interface is online.</li>
<li><strong>IP rule (unreachable)</strong> at priority <code>unreachable_base + index</code>: expected present when the interface is online.</li>
<li><strong>Routing table</strong> (table ID = index): whether the per-interface routing table has a default route.</li>
</ul>
<p>Each row renders a status badge: "Present" (green) or "Missing" (red) when the item is expected to be present (interface online); "Present (unexpected)" (yellow) or "Absent" (grey) when expected to be absent (interface offline); neutral grey "Present"/"Absent" for unknown or disabled interfaces. The card border is green when all four items are in the expected state, yellow for partial state, red when the interface is online and items are missing, and grey for offline or disabled interfaces in the expected state.</p>
<p><strong>Summary card:</strong> A single line at the top reports total healthy, degraded, and failing interface counts, plus stale rule count if any.</p>
<p><strong>Stale rules:</strong> ip rules whose priority falls within the mwan3 priority ranges but do not correspond to any currently configured UCI interface are listed in a separate table. Stale rules indicate that mwan3 was not cleanly shut down after an interface was removed from the configuration.</p>
<p><strong>Field guide:</strong> A reference panel at the bottom explains each field (Index, iif rule, fwmark rule, unreachable rule) in plain language, using the actual configured base priority values read from the <code>routing_health</code> response rather than hardcoded defaults.</p>
<h4 id="1524-ip-sets-status">15.2.4 IP Sets (Status)</h4>
<p>The IP Sets status tab (<code>ipsets.js</code>) displays runtime information about all user-defined sets currently present in <code>table inet mwan3</code>. At page load it calls <code>mwan3.nftset_info</code> to enumerate sets and their metadata, then renders each as a collapsible panel.</p>
<p><strong>Panel header:</strong> Shows the set name, family, element count (with <code>N+</code> suffix when the display is truncated), a Counters badge when per-element counters are enabled, and action buttons: Resolve (only for sets with at least one <code>domain</code> entry), Reload, Flush, and Expand/Collapse.</p>
<p><strong>Panel body:</strong> Shows UCI metadata (static entry count, domain count, loadfile name, maxelem, timeout), the list of configured domain names, and a table of current set members. Members are loaded lazily on first expand via <code>mwan3.nftset_elements</code> with a default limit of 200 elements. "Load more (1000)" and "Load all (5000)" buttons are shown when the set has more elements than the current display. Elements in counter-enabled sets show packet count and byte count (formatted as B/KiB/MiB/GiB) alongside the address.</p>
<p><strong>Actions:</strong></p>
<ul>
<li><strong>Flush</strong> (<code>mwan3.nftset_flush</code>): empties the set.</li>
<li><strong>Reload</strong> (<code>mwan3.nftset_reload</code>): flushes then re-adds static entries from UCI (<code>entry</code> values and <code>loadfile</code> contents).</li>
<li><strong>Resolve</strong> (<code>mwan3.nftset_resolve</code>): HUPs dnsmasq and re-queries each configured domain, triggering dnsmasq's nftset population as a side effect.</li>
</ul>
<p>After each action the header count and, if the panel is expanded, the member table refresh automatically.</p>
<h4 id="1525-diagnostics">15.2.5 Diagnostics</h4>
<p>The Diagnostics tab (<code>diagnostics.js</code>) provides an interactive runner for per-interface diagnostic commands. An interface selector and a task selector drive the operation; a Run button executes the selected task and displays output in a <code>&lt;pre&gt;</code> element. All buttons are disabled during execution.</p>
<table>
<thead><tr><th>Task</th><th>Implementation</th><th>Description</th></tr></thead><tbody>
<tr><td>Ping gateway</td><td><code>luci-mwan3 diag gateway &lt;iface&gt;</code></td><td>Pings the interface's current gateway 5 times via <code>mwan3 use</code></td></tr>
<tr><td>Ping tracking IPs</td><td><code>luci-mwan3 diag tracking &lt;iface&gt;</code></td><td>Pings each configured track_ip 5 times via <code>mwan3 use</code></td></tr>
<tr><td>Check IP rules</td><td><code>luci-mwan3 diag rules &lt;iface&gt;</code></td><td>Checks presence of all three ip rules for the interface</td></tr>
<tr><td>Check routing table</td><td><code>luci-mwan3 diag routes &lt;iface&gt;</code></td><td>Shows <code>ip route list table &lt;id&gt;</code> for the interface</td></tr>
<tr><td>Force interface up</td><td><code>mwan3 ifup &lt;iface&gt;</code></td><td>Triggers the ifup hotplug sequence</td></tr>
<tr><td>Force interface down</td><td><code>mwan3 ifdown &lt;iface&gt;</code></td><td>Triggers the ifdown hotplug sequence</td></tr>
</tbody></table>
<h4 id="1526-troubleshooting">15.2.6 Troubleshooting</h4>
<p>The Troubleshooting tab (<code>troubleshooting.js</code>) presents the full output of <code>mwan3 internal ipv4</code> and <code>mwan3 internal ipv6</code> as a set of collapsible <code>&lt;details&gt;</code> panels, one panel per output section. The output is parsed by splitting on underline-delimited section headings (lines of <code>=</code> characters).</p>
<p>The nftables dump section is filtered to remove the <code>mwan3_or_(meta|ct)_*</code> vmap setter chains from the display, replacing them with a comment indicating how many chains were removed. These chains are an implementation detail of the connmark operation (see <a href="#2-the-mark-bitmask-system">Section 2</a>) and would otherwise dominate the output. The IPv6 internal output deduplicates sections that are identical to the IPv4 output (Software Version and nft tables), showing only the IPv6-specific sections. Each section's content is shown in a <code>&lt;pre&gt;</code> element with <code>max-height: 250px</code> and vertical scroll.</p>
<h3 id="153-backend-helper-script-and-acl">15.3 Backend: Helper Script and ACL</h3>
<h4 id="helper-script-luci-mwan3">Helper Script (<code>luci-mwan3</code>)</h4>
<p>The <code>/usr/libexec/luci-mwan3</code> shell script provides backend commands invoked by the diagnostics tab.</p>
<p><strong><code>diag</code> subcommand:</strong> <code>diag gateway &lt;iface&gt;</code>, <code>diag tracking &lt;iface&gt;</code>, <code>diag rules &lt;iface&gt;</code>, and <code>diag routes &lt;iface&gt;</code>. The <code>diag rules</code> function reads <code>iif_rule_base</code>, <code>fwmark_rule_base</code>, and <code>unreachable_rule_base</code> from UCI config globals, computes the expected priority for each of the three ip rule tiers, queries <code>ip rule</code>, and reports found/missing counts. The <code>diag routes</code> function shows <code>ip route list table &lt;id&gt;</code> for the interface's sequential table ID.</p>
<p><strong><code>nftset</code> subcommand:</strong> <code>nftset dump</code> lists all user-defined set names in <code>table inet mwan3</code> by running <code>nft list table inet mwan3</code> and extracting set names via awk, then filtering out mwan3's own internal sets (prefixed <code>mwan3_</code>). This subcommand is no longer called by the rule editor (which uses the <code>mwan3.nftset_info</code> rpcd method directly) but is retained for command-line use.</p>
<div class="note">
<p><code>nft list sets inet mwan3</code> is not used because it lists sets from all inet tables, not just mwan3's table. <code>nft list table inet mwan3</code> is the correct form.</p>
</div>
<h4 id="acl-permissions-luci-app-mwan3json">ACL Permissions (<code>luci-app-mwan3.json</code>)</h4>
<p>The rpcd ACL file grants the LuCI frontend permission to call rpcd methods and execute files. It defines two ACL groups:</p>
<p><strong><code>luci-app-mwan3</code></strong> (configuration access): Read permission for <code>mwan3.status</code>, <code>mwan3.nftset_info</code>, <code>mwan3.nftset_members</code>, <code>mwan3.nftset_elements</code>, <code>mwan3.resolve_host</code>, and the mwan3 and network UCI packages. Write permission for <code>mwan3.nftset_flush</code>, <code>mwan3.nftset_reload</code>, <code>mwan3.nftset_resolve</code>, and the mwan3 UCI package. Also grants <code>fs.stat</code> read access for detecting optional binaries (httping, nping, arping) and exec permission for <code>luci-mwan3 nftset dump</code>.</p>
<p><strong><code>luci-app-mwan3-status</code></strong> (status and diagnostic access): Read permission for <code>mwan3.status</code>, <code>mwan3.nftset_info</code>, <code>mwan3.nftset_members</code>, <code>mwan3.nftset_elements</code>, <code>mwan3.resolve_host</code>, and <code>mwan3.routing_health</code>. Write permission for <code>mwan3.nftset_flush</code>, <code>mwan3.nftset_reload</code>, <code>mwan3.nftset_resolve</code>, and exec permission for the <code>luci-mwan3 diag</code> subcommands, <code>mwan3 internal</code>, <code>mwan3 ifup</code>, and <code>mwan3 ifdown</code>.</p>
<hr>
<h3 id="154-rpcd-methods">15.4 rpcd Methods</h3>
<p>The following methods are implemented in <code>usr/share/rpcd/ucode/mwan3</code> and declared in <code>root/usr/share/rpcd/acl.d/luci-app-mwan3.json</code>.</p>
<p><strong><code>mwan3.status {}</code></strong></p>
<p>Returns JSON data for interfaces, connected networks, and policies. Used by the Status tab.</p>
<ul>
<li><strong>Connected IPs</strong>: Parses <code>nft -j list set inet mwan3 mwan3_connected_v4/v6</code>.</li>
<li><strong>Policies</strong>: Reads membership from UCI config and cross-references mwan3track <code>STATUS</code> files. Every member is always reported with traffic share percentage.</li>
<li><strong>Interfaces</strong>: Reads status from <code>/var/run/mwan3track/</code> files and queries procd/netifd via ubus. Tracking IPs discovered by globbing <code>TRACK_*</code> files. Per-IP <code>latency</code> and <code>packetloss</code> populated when <code>check_quality=1</code>.</li>
</ul>
<p><strong><code>mwan3.nftset_info {}</code></strong></p>
<p>Returns the name, address-family type, counters flag, and runtime element count of all non-mwan3 sets in <code>table inet mwan3</code>. Used by the rule editor nftset dropdown and the IP Sets status tab.</p>
<p><strong><code>mwan3.nftset_elements { set: "&lt;name&gt;", max: N }</code></strong></p>
<p>Returns paginated elements of a named user-defined set with per-element packet/byte counters (when the set has <code>counter</code> enabled). The set name is validated against <code>^[a-zA-Z0-9_-]+$</code>. Default maximum 200 elements; supports up to 5000. Returns <code>{ elements: [...], truncated: true/false }</code>. Each element has <code>value</code>, and optionally <code>packets</code> and <code>bytes</code> for counter-enabled sets. Used by the IP Sets status tab for runtime member display.</p>
<p><strong><code>mwan3.nftset_members { set: "&lt;name&gt;" }</code></strong></p>
<p>Returns the current members of a named nft set in <code>table inet mwan3</code>. The set name is validated against <code>^[a-zA-Z0-9_-]+$</code>. Returns <code>{ members: [ ... ] }</code>. Used by the Simulator for ipset rule matching and connected-network bypass detection.</p>
<p>When a set has the <code>counters</code> flag enabled, nft wraps each element in the JSON output as <code>{"elem": {"val": "&lt;addr&gt;", "counter": {...}}}</code> rather than returning the address directly. <code>nftset_members</code> unwraps this structure so that counter-enabled sets return the same flat address list as non-counter sets.</p>
<p><strong><code>mwan3.resolve_host { host: "&lt;name&gt;", family: "&lt;ipv4|ipv6|&gt;" }</code></strong></p>
<p>Resolves a hostname by querying the local dnsmasq instance via <code>/bin/busybox nslookup</code>. Querying 127.0.0.1 populates any dnsmasq nftset entries configured for the domain as a side effect. Returns <code>{ v4: [ ... ], v6: [ ... ] }</code>. When <code>family</code> is <code>"ipv4"</code> only A records are queried; when <code>"ipv6"</code> only AAAA records; when empty (or omitted), both are resolved. The hostname is validated against <code>^[a-zA-Z0-9._-]+$</code> before any shell invocation; invalid input returns <code>{ error: "invalid hostname", v4: [], v6: [] }</code>. The DNS server echo line in nslookup output (which also contains an <code>Address:</code> prefix) is filtered by checking for the loopback address.</p>
<p><strong><code>mwan3.nftset_flush { set: "&lt;name&gt;" }</code></strong></p>
<p>Flushes all elements from the named set via <code>nft flush set inet mwan3 &lt;name&gt;</code>. The set name is validated against <code>^[a-zA-Z0-9_-]+$</code>. Returns <code>{}</code> on success or <code>{ error: "..." }</code> on failure. Used by the Flush button on the IP Sets status tab.</p>
<p><strong><code>mwan3.nftset_reload { set: "&lt;name&gt;" }</code></strong></p>
<p>Flushes the set then re-adds all static entries from the corresponding UCI <code>config ipset</code> section. Collects <code>list entry</code> values and, if <code>loadfile</code> is set, reads addresses line by line from the file (stripping comments and whitespace). Passes all entries to <code>nft add element inet mwan3 &lt;name&gt; { ... }</code> in a single invocation. Returns <code>{}</code> on success or <code>{ error: "..." }</code> on failure. If no UCI section is found for the set name, or the section is disabled (<code>enabled=0</code>), returns <code>{}</code> without error after flushing. Used by the Reload button on the IP Sets status tab.</p>
<p><strong><code>mwan3.nftset_resolve { set: "&lt;name&gt;" }</code></strong></p>
<p>Looks up the named set in UCI, collects <code>list domain</code> entries, sends SIGHUP to dnsmasq via <code>ubus.call('service', 'signal', {name:'dnsmasq', signal:1})</code> to clear its DNS cache, then queries each domain through the local dnsmasq instance (127.0.0.1) using <code>nslookup_resolve()</code>. The queries trigger dnsmasq's <code>nftset=</code> population mechanism as a side effect; no explicit <code>nft add element</code> call is needed. The address family (<code>A</code> or <code>AAAA</code>) is derived from the set's <code>family</code> UCI option. Returns <code>{ resolved: N }</code> where N is the count of domains that returned at least one address. Returns <code>{ error: "..." }</code> if the set is not found in UCI or has no <code>list domain</code> entries. Used by the Resolve button on the IP Sets status tab (only shown for sets with domain entries).</p>
<p><strong><code>mwan3.routing_health {}</code></strong></p>
<p>Compares the UCI configuration against live kernel state. For each mwan3 interface (by 1-based UCI order index N):</p>
<ul>
<li>Checks for ip rule at priority IIF_BASE+N (iif), FWMARK_BASE+N (fwmark), and UNREACHABLE_BASE+N (unreachable) via rtnl RTM_GETRULE. Base values read from UCI globals, defaulting to 1000/2000/3000.</li>
<li>Checks routing table N for a default route via rtnl RTM_GETROUTE</li>
<li>Reads <code>/var/run/mwan3track/&lt;ifname&gt;/STATUS</code> for current online/offline state</li>
<li>Reports stale ip rules (priorities in mwan3's range with no matching UCI interface)</li>
<li>Reports whether mwan3 is actively running (presence of any <code>STATUS</code> file under <code>/var/run/mwan3track/</code>)</li>
</ul>
<hr>
<h2 id="16-iptables-to-nftables-porting-notes">16. Iptables-to-nftables Porting Notes</h2>
<p>Key translation patterns used in this port, useful for anyone maintaining or extending the code:</p>
<table>
<thead><tr><th>iptables Concept</th><th>nftables Equivalent</th><th>Notes</th></tr></thead><tbody>
<tr><td><code>iptables -t mangle</code></td><td>Chains in <code>table inet mwan3</code></td><td>mwan3's own standalone table at mangle priority</td></tr>
<tr><td><code>-A PREROUTING -j chain</code></td><td>Own hook chain at <code>priority mangle + 1</code></td><td>No need to jump from fw4's chain; own table is independent</td></tr>
<tr><td><code>-A OUTPUT -j chain</code></td><td>Own <code>type route</code> hook chain</td><td>Must be <code>type route</code> for mark-based rerouting</td></tr>
<tr><td><code>iptables-restore -T mangle -n</code></td><td><code>nft -f batchfile</code></td><td>Batch file for atomic multi-command operations</td></tr>
<tr><td><code>-N chain</code></td><td><code>nft add chain inet mwan3 name</code></td><td></td></tr>
<tr><td><code>-F chain</code></td><td><code>nft flush chain inet mwan3 name</code></td><td></td></tr>
<tr><td><code>-X chain</code></td><td><code>nft delete chain inet mwan3 name</code></td><td>Must be empty first</td></tr>
<tr><td><code>-D chain match...</code></td><td><code>nft delete rule ... handle N</code></td><td>Must look up handle with <code>nft -a</code></td></tr>
<tr><td><code>-j MARK --set-xmark V/M</code></td><td><code>meta mark set meta mark &amp; ~M | V</code></td><td>See <code>mwan3_nft_mark_expr()</code>; use <code>&amp;</code>/<code>|</code> symbols not keywords</td></tr>
<tr><td><code>-j CONNMARK --restore-mark --nfmask M</code></td><td>vmap-dispatch into <code>mwan3_or_meta_&lt;imm&gt;</code> setter chains</td><td>Non-destructive masked restore. Kernel rejects the compound <code>(meta mark &amp; ~M) | (ct mark &amp; M)</code>; vmap-dispatch synthesises the same effect via per-mark OR-immediate setter chains. See <a href="#connmark-operations">§2 Connmark Operations</a>.</td></tr>
<tr><td><code>-j CONNMARK --save-mark --nfmask M</code></td><td><code>ct mark set ct mark &amp; ~M</code>, then vmap-dispatch into <code>mwan3_or_ct_&lt;imm&gt;</code> setter chains</td><td>Non-destructive masked save. Same kernel limitation, same vmap-dispatch workaround.</td></tr>
<tr><td><code>-m mark --mark V/M</code></td><td><code>meta mark &amp; M == V</code></td><td></td></tr>
<tr><td><code>-m set --match-set S dst</code></td><td><code>ip daddr @S</code></td><td>Set lives in <code>table inet mwan3</code></td></tr>
<tr><td><code>-m statistic --probability P</code></td><td><code>numgen inc mod N map { ... }</code></td><td>Deterministic round-robin instead of probabilistic</td></tr>
<tr><td><code>-m multiport --dports P</code></td><td><code>th dport { P1, P2 }</code></td><td><code>th</code> = transport header (works for tcp/udp)</td></tr>
<tr><td><code>-m icmp6 --icmpv6-type T</code></td><td><code>icmpv6 type { T1, T2, ... }</code></td><td></td></tr>
<tr><td><code>-p ipv6-icmp</code></td><td><code>icmpv6 type { ... }</code></td><td>Protocol match is implicit</td></tr>
<tr><td><code>ipset create S hash:net</code></td><td><code>set S { type ipv4_addr; flags interval; auto-merge; }</code></td><td>Defined via <code>config ipset</code> in mwan3 UCI; <code>auto-merge</code> handles overlapping elements</td></tr>
<tr><td><code>ipset add S element</code></td><td><code>nft add element inet mwan3 S { element }</code></td><td></td></tr>
<tr><td><code>ipset flush S</code></td><td><code>nft flush set inet mwan3 S</code></td><td></td></tr>
<tr><td><code>ipset create S hash:ip,mark</code></td><td><code>map S { type addr : mark; flags dynamic,timeout; }</code></td><td>Maps store key-&gt;value pairs</td></tr>
<tr><td><code>-j SET --add-set S src,src</code></td><td><code>update @S { ip saddr : meta mark &amp; M }</code></td><td></td></tr>
<tr><td><code>-m set --match-set S src,src</code></td><td><code>meta mark set ip saddr map @S</code></td><td>Regular map lookup (not <code>vmap</code> which requires verdicts)</td></tr>
<tr><td>Separate ipv4/ipv6 chains</td><td>Single <code>inet</code> chain + <code>meta nfproto</code></td><td>Or just <code>ip</code>/<code>ip6</code> selectors in rules</td></tr>
</tbody></table>
<div class="warn">
<p><strong>Key kernel limitations to be aware of:</strong></p>
<ul>
<li><strong>No compound two-source bitwise:</strong> Expressions like <code>meta mark set meta mark | ct mark &amp; X</code> or <code>ct mark set ct mark &amp; ~M | meta mark &amp; M</code> fail with "Operation not supported". Each set expression can only draw from one register source. <strong>Workaround:</strong> synthesise the masked operation via <code>vmap</code>-dispatch into per-mark setter chains whose body is a single-source <code>meta/ct mark | &lt;constant immediate&gt;</code>. mwan3 uses this for masked connmark save and restore - see <a href="#connmark-operations">§2 Connmark Operations</a>.</li>
<li><strong>No numgen in compound expressions:</strong> <code>meta mark set meta mark &amp; COMP | numgen inc mod N map { ... }</code> fails for the same reason. Use <code>meta mark set numgen ...</code> with a guard condition ensuring the target bits are already zero.</li>
<li><strong>vmap vs map:</strong> <code>vmap</code> expects verdict values (accept/drop/jump), not data values like marks. For IP→mark lookups, use regular <code>map</code>.</li>
<li><strong><code>nft add set</code> flag immutability:</strong> Creating a set is idempotent, but flags (like <code>auto-merge</code>) cannot be updated on existing sets. Must delete and recreate to change flags.</li>
</ul>
</div>
<hr>
<h2 id="17-command-line-tools">17. Command Line Tools</h2>
<h3 id="171-mwan3-lb-test-load-balancing-distribution-verifier">17.1 mwan3-lb-test: Load Balancing Distribution Verifier</h3>
<p>A diagnostic tool <code>/usr/sbin/mwan3-lb-test</code> verifies that load balancing is distributing traffic across policy members in the expected proportions.</p>
<h4 id="usage">Usage</h4>
<pre><code>mwan3-lb-test [-6] -c &lt;client_ip&gt; &lt;policy_name&gt; [ip1 ip2 ...]
mwan3-lb-test cleanup</code></pre>
<p><code>-6</code> selects IPv6 mode. <code>-c &lt;client_ip&gt;</code> is mandatory and specifies the LAN client that will run the test pings. Optional IP arguments override the default destination pool. The <code>cleanup</code> subcommand removes stale sets and rules from an aborted run.</p>
<h4 id="design">Design</h4>
<ul>
<li><strong>NITER computation:</strong> The number of test iterations is computed from member weights using GCD: <code>base_N = total_weight / GCD(weights)</code>, <code>NITER = base_N * ceil(30 / base_N)</code>. This ensures per-member expected hit counts are whole numbers and that NITER is always at least 30.</li>
<li><strong>ICMP-only test rule:</strong> A temporary <code>meta l4proto icmp ip daddr @mwan3_lb_test_&lt;PID&gt;</code> counter rule is inserted into <code>mwan3_rules</code> ahead of user rules. The ICMP restriction prevents DNS queries, TCP connections, and other traffic from contaminating the count. <code>-6</code> mode uses <code>meta l4proto ipv6-icmp ip6 daddr @set</code>.</li>
<li><strong>Client isolation:</strong> A <code>forward</code> chain drop rule blocks pings to the test destination set from all LAN clients except the nominated test client (<code>-c</code>). An <code>mwan3_output</code> return rule bypasses mwan3 marking for any router process pinging the same IPs. Both rules are scoped to the test set and removed on exit.</li>
<li><strong>Tracking IP exclusion:</strong> The default destination pool excludes IPs already configured as mwan3 <code>track_ip</code> values. mwan3track pings those IPs via <code>mwan3_output -&gt; mwan3_rules</code>, which would match the test rule and inflate the count.</li>
<li><strong>Windows command:</strong> A <code>cmd.exe</code> <code>for</code> loop is output alongside the Linux shell loop. Windows <code>ping</code> uses a fixed ICMP identifier (id=1), causing conntrack entry reuse on repeated pings to the same destination. The Windows command uses an inter-ping delay of <code>30/TRACK_COUNT + 3</code> seconds so the full cycle through all test IPs exceeds the 30s ICMP conntrack timeout, ensuring each revisit generates a fresh conntrack entry. The IP list is formatted with <code>^</code> line continuation at 4 IPs per line.</li>
<li><strong>Cleanup:</strong> Removes the temporary set and rules on normal exit, SIGINT, SIGTERM, and SIGPIPE. A startup sweep removes stale <code>mwan3_lb_test_*</code> sets and rules from any aborted previous run.</li>
</ul>
<p><strong>Files changed:</strong> <code>usr/sbin/mwan3-lb-test</code> (new), <code>Makefile</code></p>
<hr>
<h3 id="172-mwan3-diag-network-diagnostic-report">17.2 mwan3-diag: Network Diagnostic Report</h3>
<p><code>mwan3-diag</code> is a ucode diagnostic script installed to <code>/usr/sbin/mwan3-diag</code> that collects a comprehensive snapshot of the network state relevant to mwan3 operation. It is intended to produce a report that can be posted in a forum thread or bug report without manual redaction.</p>
<h4 id="usage">Usage</h4>
<pre><code>mwan3-diag</code></pre>
<p>The script collects interface addresses, routing tables (including all per-WAN tables), policy rules, neighbour cache, mwan3 interface status, the mwan3 UCI configuration, the complete mwan3 nftables ruleset, the fw4 mangle chains that interact with mwan3 packet marking, and the last 200 lines of the mwan3 log.</p>
<p>Before printing any output the script builds a map of every public routable IPv4 and IPv6 address present in the collected data and replaces each one with a stable placeholder -- <code>PUB4_1</code>, <code>PUB4_2</code>, <code>PUB6_1</code> and so on -- throughout the entire report, including free-form text such as nftables rules and log lines. The same address always receives the same placeholder, so cross-references between sections remain consistent. Private addresses (RFC1918, link-local <code>fe80::</code>, ULA <code>fc00::/7</code>, loopback) are left unchanged as they are diagnostically important. The elements of user-defined nftables sets are replaced with <code>{ ... }</code> rather than disclosed.</p>
<p><strong>Files changed:</strong> <code>usr/sbin/mwan3-diag</code> (new), <code>Makefile</code></p>
<hr>
<h2 id="18-changelog">18. Changelog</h2>
<h3 id="181-version-369">18.1 Version 3.6.9</h3>
<p><strong>Summary:</strong> Stop router-originated IPv6 link-local traffic, most visibly odhcpd's DHCPv6 replies to LAN clients, from being policy-routed into a WAN routing table and failing, by accepting single-link scopes before classification. Clear an inherited skb mark on ingress so a tunnelled-WAN reply is no longer pinned to the wrong routing table. Emit dnsmasq <code>nftset=</code> directives correctly when a domain is shared across multiple ipset sections, and stage the fragment atomically under the runtime directory.</p>
<hr>
<h3 id="mwan3-clear-inherited-mark-on-ingress-before-conntrack-restore">mwan3: clear inherited mark on ingress before conntrack restore</h3>
<p>A packet decapsulated from a tunnel interface can inherit the outer packet's skb mark. On a tunnelled WAN (a tunnel broker or an L2TP link, for example) an inbound packet can therefore arrive already carrying mwan3's mark bits; the reply of a LAN-initiated flow is the common case. The inherited mark is independent of the inner address family, so the same failure occurs for IPv4 as for IPv6.</p>
<p>That stale mark defeats the conntrack mark restore in the prerouting chain, which is guarded on the mwan3 mark bits being clear, so the restore is skipped. The unguarded save that follows writes the inherited mark to the connection's conntrack mark, overwriting the classification the forward direction stored. Every subsequent packet of the flow then restores the wrong mark and is dispatched to the wrong routing table, pinning established flows to the wrong WAN.</p>
<p>Add a rule at the start of the prerouting chain, before the restore, that clears mwan3's mask bits. The restore and the interface and policy classification then re-derive the mark from a clean slate. The rule is a no-op for traffic that arrives with the bits already clear, the normal case on non-tunnel WANs, and it preserves any bits outside mwan3's mask. Mwan3 owns its mask exclusively and the coexisting policy-routing and firewall layers use disjoint bits, so no upstream hook legitimately sets these bits on ingress; the clear is therefore unconditional and covers both address families.</p>
<hr>
<h3 id="mwan3-combine-dnsmasq-nftset-directives-for-multi-family-domains">mwan3: combine dnsmasq nftset directives for multi-family domains</h3>
<p>dnsmasq silently ignores duplicate <code>nftset=</code> directives for the same domain: when a second directive names a domain already seen, it is discarded. This means that if an IPv4 and IPv6 ipset section both list the same domain, emitting separate per-section <code>nftset=</code> lines causes only the first set to populate.</p>
<p>Replace the per-section emission with a two-pass approach: pass 1 collects all (domain, set-specifier) mappings into a temp file, pass 2 sorts by domain and emits one <code>nftset=</code> line per unique domain with all its set targets comma-separated. The comma-separated multi-set format is documented dnsmasq grammar for the <code>nftset</code> directive.</p>
<p>This also fixes the same-family variant of the bug: a domain appearing in two sets of the same address family previously populated only the first set, and now correctly populates both.</p>
<hr>
<h3 id="mwan3-stage-dnsmasq-nftset-fragment-atomically-under-the-runtime-dir">mwan3: stage dnsmasq nftset fragment atomically under the runtime dir</h3>
<p>The per-instance dnsmasq nftset fragment was staged at a fixed <code>&lt;fragment&gt;.new</code> path inside the dnsmasq confdir. This has two defects:</p>
<ul>
<li>Two concurrent mwan3 invocations both write the same staging path and then rename it, so one can clobber or read the other's half-written file.</li>
</ul>
<ul>
<li>The transient <code>.new</code> file sits inside the directory dnsmasq scans, meaning dnsmasq can attempt to parse a partially-written or stale staging file depending on <code>conf-dir</code> filter configuration.</li>
</ul>
<p>Stage under the mwan3 runtime directory using a per-process, per-instance unique name. The existing <code>mv -f</code> swap remains and is now a guaranteed atomic same-filesystem rename (both paths are on tmpfs), so dnsmasq only ever observes the complete final fragment. Concurrent runs never share a staging path by construction.</p>
<hr>
<h3 id="mwan3-never-policy-route-ipv6-link-local-traffic">mwan3: never policy-route IPv6 link-local traffic</h3>
<p>A catch-all <code>dest_ip ::/0</code> rule also matches link-local destinations. Router-originated link-local packets, most visibly odhcpd's DHCPv6 replies to LAN clients, then get a WAN fwmark and are re-routed into that WAN's routing table. That table has no link-local route for LAN devices, so sending fails with <code>ENETUNREACH</code>: <code>odhcpd[1234]: Failed to send to fe80::...%lan@br-lan.10 (Network unreachable)</code>. This breaks DHCPv6 on all LANs whenever an IPv6 WAN member is online.</p>
<p>Link-local traffic can never leave its link, so policy-routing it is never correct. Accept it before classification in <code>mwan3_output</code> and <code>mwan3_prerouting</code>. Covered are exactly the scopes that are single-link by definition (RFC 4291): link-local unicast (<code>fe80::/10</code>) and interface-/link-scope multicast (<code>ff01::/16</code>, <code>ff02::/16</code>). Wider multicast scopes remain subject to policy routing.</p>
<p>The existing ICMPv6 ND bypass stays: NS/NA can also target global unicast addresses, which a link-local destination match does not cover. Its stale comment is rewritten.</p>
<hr>
<h3 id="182-version-368">18.2 Version 3.6.8</h3>
<p><strong>Summary:</strong> Fix an operator precedence regression in 3.6.7 in the address family guard.</p>
<hr>
<h3 id="mwan3-fix-operator-precedence-in-the-address-family-guard">mwan3: fix operator precedence in the address-family guard</h3>
<p>The address-family guard in <code>mwan3_set_user_nft_rule()</code>combined its tests as <code>[ ] &amp;&amp; [ ] || [ ] &amp;&amp; [ ]</code>, which POSIX shell evaluates as <code>((A &amp;&amp; B) || C) &amp;&amp; D</code> rather than the intended <code>(A &amp;&amp; B) || (C &amp;&amp; D)</code>, so the guard no longer skipped a mismatched-family pass. Group each <code>&amp;&amp;</code> pair to restore the intended behaviour.</p>
<hr>
<h3 id="183-version-367">18.3 Version 3.6.7</h3>
<p><strong>Summary:</strong></p>
<p>Comma-separated address lists are now accepted in <code>src_ip</code> and <code>dest_ip</code> rule options, with each element individually validated and the list converted to nft anonymous set syntax at rule-build time.</p>
<p>IP address validation has been overhauled. A new <code>mwan3ipcheck</code> binary replaces the fragile regex-based validation, using the <code>inet_pton()</code> function for address parsing that correctly rejects invalid input the old regex accepted silently. Of a list of 168 test cases, including both valid and invalid addresses, the regex expressions incorrectly passed 53 that were syntactically incorrect, meaning that mwan3ipcheck is a substantial reliability enhancement.</p>
<p>The <code>mwan3 status</code> command now parses <code>nft list set</code> output via <code>nft -j</code> and <code>jshn.sh</code> rather than a hand-built regex, fixing a bug where IPv6 prefix lengths were silently dropped and range elements were split into unrelated bare addresses.</p>
<p>Interface tracking log output has been substantially cleaned up. A new <code>verbose_logging</code> globals option activates the existing debug-level <code>LOG()</code> calls, which were previously permanently suppressed; the option takes effect on <code>HUP</code>-triggered reloads without a full restart. Several messages that fired at notice level during normal operation have been demoted to debug level, including per-host probe results during recovery, the "lost host(s)" recovery message, the "skip disconnected event" notice, and the check failure message produced when an interface is brought down manually before the hotplug <code>USR1</code> signal arrives.</p>
<p>The lost host count reported in tracking log messages has been corrected; it previously multiplied the host failure count by pings-per-host and reported the result as "lost ping(s)" rather than "lost host(s)".</p>
<p>Two tracking robustness fixes have been applied. When a signal (<code>HUP</code>, <code>USR1</code>, or <code>USR2</code>) arrives mid-round, the partially-completed ping loop now aborts immediately rather than continuing to remaining targets and scoring the killed pings as genuine failures. During service stop, <code>mwan3track</code> instances are now terminated via <code>procd_kill</code> before the routing infrastructure is torn down, eliminating spurious 100% loss messages that appeared on every restart. A syntax error in the <code>nping</code> tracking method has also been fixed: when <code>nping</code> produces no output the <code>result</code> variable was empty, causing a bare <code>[ -eq 0 ]</code> test error; it now defaults to <code>1</code>.</p>
<p>In <code>luci-app-mwan3</code>, the <code>verbose_logging</code> option is exposed as a checkbox on the Globals page. The rule editor's source and destination IP fields now accept comma-separated address lists with per-element validation and family consistency checking, matching the support added in mwan3. IPv6 validation across all files has been switched to <code>validation.parseIPv6()</code>, replacing heuristic detection. The configuration checker's rule shadowing detection now recognises when two rules reference the same nftset. A LuCI framework bug where the datatype validator corrupts CIDR values passed to custom validate functions has been worked around in the ipset entry validator.</p>
<hr>
<h3 id="mwan3-suppress-spurious-check-failure-log-on-manual-ifdown">mwan3: suppress spurious check failure log on manual ifdown</h3>
<p>When an interface is manually brought down, the tracking pings fail immediately but the <code>USR1</code> signal from hotplug has not yet arrived. This produces a misleading check failure log message moments before the interface is marked offline anyway.</p>
<p>The hotplug script writes the interface state to the hotplug state file before sending <code>USR1</code>, so <code>mwan3track</code> can read it to detect that an ifdown is already in progress. Skip the failure log when the hotplug state is already offline, since the message is redundant and misleading in that context.</p>
<hr>
<h3 id="mwan3-demote-lost-hosts-recovery-message-to-debug">mwan3: demote "lost host(s)" recovery message to debug</h3>
<p>This message fires every tracking round where score has not fully recovered and at least one host failed, even though the reliability threshold was met. During recovery this adds per-round noise alongside the connecting and connected notices that already communicate the state transition.</p>
<p>Demote to debug since it is only useful for detailed recovery diagnostics.</p>
<hr>
<h3 id="mwan3-fix-misleading-lost-pings-count-in-tracking-log-message">mwan3: fix misleading "lost ping(s)" count in tracking log message</h3>
<p>The <code>lost</code> variable counts how many track hosts failed to respond, not how many individual pings were lost. Multiplying by <code>count</code> (pings per host) and reporting the result as "lost ping(s)" overstated the actual loss. For example, with <code>count=5</code> and 1 of 3 hosts unreachable, it reported "Lost 5 ping(s)" when only 1 host failed.</p>
<p>Report the host count directly as "Lost N host(s)" instead.</p>
<hr>
<h3 id="mwan3-demote-skip-disconnected-event-message-to-debug">mwan3: demote "skip disconnected event" message to debug</h3>
<p>This notice-level message logged when <code>disconnected()</code> was called on an interface that was already offline or connecting. It sat at the same syslog priority as genuine state transitions like "is offline" and "is online", adding noise that obscured real status changes.</p>
<p>Demote to debug since it is only useful for diagnosing redundant state transitions.</p>
<hr>
<h3 id="mwan3-demote-per-host-recovery-success-messages-to-debug">mwan3: demote per-host recovery success messages to debug</h3>
<p>During recovery (<code>score &lt;= up</code>), <code>mwan3track</code> logged an info message for every track IP that responded successfully, every tracking round. With multiple track IPs and a default <code>up</code> threshold of 5, this produced up to 15 info messages per recovery episode.</p>
<p>The connecting and connected notices already communicate recovery progress at the operational level. Per-host probe results during recovery are diagnostic detail, now available via the <code>verbose_logging</code> option.</p>
<hr>
<h3 id="mwan3-add-verboselogging-option-to-enable-debug-log-output">mwan3: add verbose_logging option to enable debug log output</h3>
<p>The <code>LOG()</code> function has historically suppressed all debug-level messages unconditionally, making the existing debug log calls across <code>common.sh</code>, <code>mwan3.sh</code>, and <code>mwan3track</code> permanently dead code.</p>
<p>Add a globals UCI option <code>verbose_logging</code> (default <code>0</code>) that controls whether debug-level messages are emitted to syslog. When enabled, all existing <code>LOG</code> debug calls become active, providing detailed diagnostic output for troubleshooting interface tracking and nft rule setup.</p>
<p>The option is loaded in <code>mwan3_init()</code> for all scripts, and also in <code>mwan3track</code>'s <code>load_tracking_config()</code> so that changes take effect on <code>HUP</code>-triggered config reloads without requiring a full restart.</p>
<hr>
<h3 id="mwan3-fix-test-syntax-error-in-nping-tracking-method">mwan3: fix test syntax error in nping tracking method</h3>
<p>The <code>nping</code> tracking method extracts the lost-packet count from <code>nping</code> output via <code>grep</code> and <code>awk</code>. If <code>nping</code> produces no output or the output does not contain a "Lost" line, the <code>result</code> variable is empty. The subsequent arithmetic comparison expands to <code>[ -eq 0 ]</code>, which is a test syntax error in POSIX sh.</p>
<p>Default <code>result</code> to <code>1</code> (failure) when the extraction produces an empty string, so a missing "Lost" line is correctly treated as packet loss.</p>
<hr>
<h3 id="mwan3-terminate-trackers-before-infrastructure-teardown-on-stop">mwan3: terminate trackers before infrastructure teardown on stop</h3>
<p>During service stop, <code>stop_service()</code> tears down ip rules, routing tables, and nft chains before procd sends <code>TERM</code> to the <code>mwan3track</code> instances. The trackers continue pinging during this window but their packets can no longer be routed (the fwmark-based ip rules are gone), causing every tracking target to report 100% loss. This produces spurious failure log messages on every service restart.</p>
<p>Stop all <code>mwan3track</code> instances at the start of <code>stop_service</code> via <code>procd_kill</code>, which tells procd to terminate them and mark them as intentionally stopped.</p>
<hr>
<h3 id="mwan3-skip-tracking-round-when-signal-interrupts-ping-loop">mwan3: skip tracking round when signal interrupts ping loop</h3>
<p>When a signal (<code>HUP</code>, <code>USR1</code>, or <code>USR2</code>) arrives while <code>mwan3track</code> is mid-way through a ping round, the signal handler kills the current ping subprocess and sets an event flag. The killed ping returns non-zero, and the for loop continues to remaining tracking targets, all of which also fail. The event flags are not checked until after the for loop completes, so the entire round of killed-ping failures is scored and logged as genuine tracking failures.</p>
<p>Add an event flag check inside the per-IP for loop that breaks immediately when any signal flag is set. After the for loop, if any flag is set, reset the round counters and continue to the top of the while loop where event handling already runs, skipping the scoring, logging, and sleep for the aborted round.</p>
<hr>
<h3 id="mwan3-support-comma-separated-addresses-in-srcip-and-destip-rule-options">mwan3: support comma-separated addresses in src_ip and dest_ip rule options</h3>
<p>The <code>mwan3ipcheck</code> binary returns <code>"mixed"</code> for comma-separated address lists containing both IPv4 and IPv6 elements. Add <code>"mixed"</code> to the validation rejection check alongside <code>"invalid"</code>, since a single nft anonymous set cannot contain both address families.</p>
<p>When a comma is present in <code>src_ip</code> or <code>dest_ip</code>, convert the value to nft anonymous set syntax by wrapping in <code>{ }</code> with normalized spacing. Single addresses (no comma) pass through unchanged, preserving identical nft output to prior behaviour.</p>
<p>CIDR elements within comma lists are supported natively by nft anonymous sets and require no special handling.</p>
<hr>
<h3 id="mwan3-replace-ipv4ipv6-regex-parsing-with-nft-json-in-mwan3-status">mwan3: replace IPv4/IPv6 regex parsing with nft JSON in mwan3 status</h3>
<p>The IPv6 regex had a bug where the CIDR suffix <code>(/[0-9]+)?</code> only bound to the final alternation branch because the 12-branch regex was joined with bare <code>|</code> and no outer grouping. This caused all IPv6 prefix lengths to be silently dropped from <code>mwan3 status</code> output. Range elements (produced by nft auto-merge on interval sets) were also mishandled, split into two unrelated bare addresses with no indication they form a range.</p>
<p>Replace the regex-based text parsing of <code>nft list set</code> output with structured JSON parsing using <code>nft -j</code> and <code>jshn.sh</code>. A new helper <code>_mwan3_report_connected_set()</code> handles both v4 and v6 sets, extracting prefix elements as <code>addr/len</code> and range elements as <code>start-end</code>.</p>
<p>Delete the <code>IPv4_REGEX</code> and <code>IPv6_REGEX</code> variables which are now unused.</p>
<hr>
<h3 id="mwan3-replace-regex-ip-validation-with-mwan3ipcheck">mwan3: replace regex IP validation with mwan3ipcheck</h3>
<p>Replace the <code>grep</code>-based IPv4/IPv6 regex validation in the rule-building path with calls to the <code>mwan3ipcheck</code> binary, which uses the <code>inet_pton()</code> function.</p>
<p>The previous validation used a 12-line IPv6 extended regular expression and a separate IPv4 regex via <code>grep</code>, which matched substrings rather than whole strings. This caused it to accept invalid input such as addresses with trailing garbage, leading zeros in octets, extra octets, and out-of-range values.</p>
<p>Additionally, the validation only detected family mismatches (an IPv6 address in an IPv4 rule pass, or vice versa). An invalid IP address that matched neither regex would silently pass through and produce a nft syntax error at batch commit time.</p>
<p><code>inet_pton()</code> is the authoritative address parser used by nftables. It handles all valid representations by definition and rejects everything else, eliminating both the substring-matching problem and the missing validation for invalid addresses.</p>
<hr>
<h3 id="mwan3-add-mwan3ipcheck-address-validation-binary">mwan3: add mwan3ipcheck address validation binary</h3>
<p>Add a small C binary that validates and classifies IP addresses and CIDR notation using the POSIX <code>inet_pton()</code> function. Given a string argument, <code>mwan3ipcheck</code> prints <code>"ipv4"</code>, <code>"ipv6"</code>, or <code>"invalid"</code> to stdout and exits 0 on valid input or 1 on invalid input.</p>
<p>When a CIDR prefix is present (e.g. <code>192.168.1.0/24</code> or <code>2001:db8::/32</code>), the prefix length is validated against the appropriate maximum for the detected family (32 for IPv4, 128 for IPv6). Leading zeros in the prefix length are rejected.</p>
<p>This binary replaces the fragile regex-based IP address validation currently used in the rule-building path. <code>inet_pton()</code> is the authoritative POSIX address parser and handles all valid IPv4 and IPv6 representations by definition, eliminating the need for a hand-built 12-branch IPv6 extended regular expression.</p>
<p>The binary has no dependencies beyond libc (uses only <code>inet_pton</code>, <code>strtol</code>, and standard string functions).</p>
<hr>
<h3 id="luci-app-mwan3-fix-column-alignment-across-ip-set-member-tables">luci-app-mwan3: fix column alignment across IP set member tables</h3>
<p>Set fixed column widths (50%/25%/25%) on both header and data cells so all expanded set tables align consistently.</p>
<hr>
<h3 id="luci-app-mwan3-add-verbose-logging-option-to-globals-page">luci-app-mwan3: add verbose logging option to globals page</h3>
<p>Add a checkbox for the <code>verbose_logging</code> UCI option to the Network &gt; MultiWAN Manager &gt; Globals page, positioned below the existing Logging checkbox. This exposes the mwan3 debug-level logging toggle.</p>
<hr>
<h3 id="luci-app-mwan3-add-comma-separated-address-support-and-enhanced-validation">luci-app-mwan3: add comma-separated address support and enhanced validation</h3>
<p>Add comma-separated address support to the rule editor's source and destination IP fields, matching the support already present in mwan3. Each address in a comma-separated list is individually validated for format and family consistency.</p>
<p>Replace IP family detection heuristics with <code>validation.parseIPv6()</code> across all files for robust IPv6 validation.</p>
<p>Improve validation in the simulator by tightening <code>looksLikeFqdn()</code> to reject malformed IPv4s instead of sending them to DNS resolution and validate addresses so that malformed ones are rejected before simulation.</p>
<p>Work around a LuCI framework bug where the datatype validator corrupts the value passed to custom validate functions for CIDR inputs. The ipset entry validator now performs its own format checking via a stub validator instead of relying on <code>o.datatype</code>.</p>
<p>Improve the configuration checker's rule shadowing detection to recognise when two rules reference the same nftset, rather than conservatively skipping all nftset comparisons.</p>
<hr>
<h3 id="184-version-366">18.4 Version 3.6.6</h3>
<p><strong>Summary:</strong> A race condition in the ct mark save logic has been fixed. The previous two-step clear and set sequence first cleared the MMX bits to zero and then set the new value in a consecutive rule, but between those two rules, under the right conditions, a packet on another CPU core could read the intermediate zero state, causing connections to be re-evaluated by the rules chain instead of being pinned to their assigned WAN. The clear is now folded into each setter chain as a single atomic expression, eliminating the race window and ensuring clear-set atomicity.</p>
<p>Adds a startup guard: mwan3 exits cleanly during startup if all interfaces are disabled, avoiding unnecessary nft chain loading and daemon startup. The default configuration now disables all interfaces and rules. Previously, a fresh install with no explicit user configuration would break the IPv6 internet because the balanced policy had no enabled IPv6 members, causing all IPv6 traffic to hit fall through to the unreachable last resort. A fresh install is now inert until explicitly configured by the user.</p>
<hr>
<h3 id="mwan3-skip-startup-when-no-interfaces-are-enabled">mwan3: skip startup when no interfaces are enabled</h3>
<p>Add a guard in <code>start_service</code> after <code>mwan3_init</code> loads the configuration: if no interface section has <code>option enabled '1'</code>, return without loading the nft framework or starting any daemons.</p>
<hr>
<h3 id="mwan3-disable-all-interfaces-and-rules-in-default-config">mwan3: disable all interfaces and rules in default config</h3>
<p>The default configuration ships with the wan interface enabled and three rules (<code>https</code>, <code>default_rule_v4</code>, <code>default_rule_v6</code>) active. On a fresh install with no user customisation, this breaks IPv6 internet: the balanced policy has no enabled IPv6 members, so all IPv6 traffic that reaches the policy chain hits the unreachable last resort.</p>
<p>IPv4 works by coincidence (wan is the only enabled member and routes traffic normally), but the tracking pings and nft chain overhead serve no purpose on a single-WAN setup.</p>
<p>Set all interfaces and all rules to enabled=0 so a fresh install is completely inert until the user explicitly configures mwan3.</p>
<hr>
<h3 id="mwan3-fix-ct-mark-save-race-by-folding-clear-into-setter-chains">mwan3: fix ct mark save race by folding clear into setter chains</h3>
<p>The nftables <code>ct mark save</code> operation used a two-step sequence: clear the MMX bits in ct mark to zero, then <code>vmap-dispatch</code> into a setter chain that ORed the new value in. Between those two rules, a packet on another CPU could observe ct mark with zeroed MMX bits, causing the restore step to miss the vmap lookup and allowing the connection to be reclassified to a random WAN.</p>
<p>Fold the clear into each save setter chain so that ct mark goes directly from old-value to new-value in a single nft expression (<code>ct mark set ct mark &amp; COMPLEMENT | VALUE</code>). This eliminates the intermediate zero state entirely, providing the same single-write atomicity that <code>iptables CONNMARK --save-mark</code> had.</p>
<p>The standalone clear rules in the prerouting and output chains are removed since the setter chains now handle the masking internally.</p>
<hr>
<h3 id="185-version-365">18.5 Version 3.6.5</h3>
<p><strong>Summary:</strong> Replaces all <code>ip</code> command output parsing with direct <code>ucode-mod-rtnl</code> netlink calls via four new helper scripts in <code>/lib/mwan3</code>: <code>mwan3-create-iface-route.uc</code>, <code>mwan3-get-addr.uc</code>, <code>mwan3-list-routes.uc</code>, <code>mwan3-manage-rules.uc</code>. These scripts replace the shell invocations of the ip binary and consequent (fragile) parsing of the ip output using <code>sed</code>, <code>awk</code> and <code>grep</code>, helping to make mwan3 more robust and future proofing it against potential changes in the output format of ip commands, since the helper scripts return precisely the information needed by mwan3 and do not require any parsing of the output. The scripts also contribute to a substantially enhanced efficiency profile, since there is now only one call per operation instead of multiple <code>ip</code> subprocesses each invoking <code>ip</code>, <code>sed</code>, <code>grep</code> and <code>awk</code>.</p>
<p>Adds <code>mwan3ct</code>, a small C-language replacement for the conntrack user-space CLI tool that takes over all conntrack flush operations previously handled by the <code>conntrack</code> CLI tool. <code>mwan3ct</code> makes possible several highly targeted flush operations currently not possible with the <code>conntrack</code> CLI tool without extensive (fragile) parsing of the output and multiple invocations of <code>conntrack -D</code>.</p>
<p><code>mwan3ct</code> flushes <code>UNREPLIED</code> conntrack entries when an interface comes online to prevent stale marks from pinning traffic to the wrong WAN. This was previously not possible and is now enabled via <code>mwan3ct</code>. It's particularly helpful in the scenario of a load-balanced connection that has keepalives that could pin a connection to a failed WAN because the keepalives maintain the conntrack entry current.</p>
<p>Applies minor hardening by moving nft batch files from <code>/tmp</code> to the root-owned <code>/var/run/mwan3</code> directory and applies more restrictive umasks to the directory and file creation.</p>
<p>Fixes a scoping issue / buggy behaviour in nft that returns ipsets from other tables even though the query is scoped to a specific table, leading to a slow load time on <code>luci-app-mwan3</code> <code>Ipset</code> and <code>Rule</code> tabs if the other tables contain large set-based blocklists (eg., adblock packages, banIP).</p>
<hr>
<h3 id="mwan3-fix-nftsetinfo-fetching-all-inet-family-sets">mwan3: fix nftset_info fetching all inet family sets</h3>
<p><code>nft 1.1.6</code> does not scope <code>nft list sets &lt;family&gt; &lt;table&gt;</code> to the named table - it returns every set in the family, including sets from unrelated tables. On systems with firewall packages that maintain large element sets (e.g. IP blocklists), <code>nft -j list sets inet mwan3</code> would return several megabytes of JSON containing elements from those foreign sets, making the rpcd <code>nftset_info</code> call take much longer than it should.</p>
<p>Replace <code>nft -j list sets inet mwan3</code> with <code>nft -j list table inet mwan3</code>, which is correctly scoped to the mwan3 table and returns only its content.</p>
<p>Since <code>nft list table</code> includes set element data inline, the separate <code>count_nftset_elements()</code> subprocess call per set is also eliminated: replace it with a direct <code>length()</code> of the elem array already present in the parsed JSON.</p>
<hr>
<h3 id="mwan3-remove-conntrack-runtime-dependency">mwan3: remove conntrack runtime dependency</h3>
<p><code>mwan3ct</code> replaced all <code>conntrack</code> CLI invocations in <code>mwan3.sh</code>. The <code>+conntrack</code> package dependency is no longer needed.</p>
<hr>
<h3 id="mwan3-improve-readability-of-shell-source-files">mwan3: improve readability of shell source files</h3>
<p>Added blank lines around comment blocks in <code>mwan3.sh</code> and <code>common.sh</code> to improve readability. Whitespace-only change, no functional modifications.</p>
<hr>
<h3 id="mwan3-replace-ip-output-parsing-with-ucode-mod-rtnl-helpers">mwan3: replace ip output parsing with ucode-mod-rtnl helpers</h3>
<p>Replace all <code>ip</code> command invocations that parse output with calls to the ucode helper scripts added in the previous commit. This eliminates fragile <code>sed</code>/<code>awk</code> text parsing and reduces subprocess spawning.</p>
<p>Shell changes:</p>
<ul>
<li><code>mwan3_get_src_ip</code>: replace <code>$IP address ls | sed</code> with <code>mwan3-get-addr.uc</code></li>
<li><code>mwan3_set_general_rules</code>: replace <code>$IP rule list | awk</code> with <code>mwan3-manage-rules.uc add-general</code></li>
<li><code>mwan3_delete_iface_rules</code>: replace <code>$IP rule list | awk</code> with <code>mwan3-manage-rules.uc delete-iface</code></li>
<li><code>mwan3_report_iface_status</code>: replace <code>$IP rule/route</code> checks with <code>mwan3-manage-rules.uc check</code> and <code>check-route</code></li>
<li><code>mwan3_set_custom_set</code>, <code>mwan3_set_connected_ipv4/v6</code>: replace <code>$IP route | awk | grep</code> pipelines with <code>mwan3-list-routes.uc</code></li>
<li><code>mwan3_create_iface_route</code>: replace <code>$IP route list | sed | while read</code> with <code>mwan3-create-iface-route.uc</code></li>
</ul>
<p>Define <code>MWAN3_GET_ADDR</code>, <code>MWAN3_LIST_ROUTES</code>, <code>MWAN3_MANAGE_RULES</code>, and <code>MWAN3_CREATE_IFACE_ROUTE</code> variables in <code>common.sh</code> for consistent path references.</p>
<p>Remove dead code no longer reachable after these changes: <code>mwan3_update_dev_to_table</code>, <code>mwan3_route_line_dev</code>, <code>mwan3_get_routes</code>, <code>mwan3_extra_tables_routes</code>, and the <code>MWAN3_ROUTE_LINE_EXP</code> construction.</p>
<p>rpcd plugin: replace <code>popen ip -j rule/route</code> calls with direct <code>rtnl.request()</code> netlink queries, and fix field name mappings for rtnl (<code>oif</code> not <code>dev</code>, <code>null</code> not <code>"default"</code>, family as integer).</p>
<p>Makefile: add <code>INSTALL_BIN</code> lines for the four <code>.uc</code> scripts.</p>
<hr>
<h3 id="mwan3-add-ucode-helper-scripts-for-rtnl-netlink-access">mwan3: add ucode helper scripts for rtnl netlink access</h3>
<p>Add four ucode scripts that use <code>ucode-mod-rtnl</code> for direct netlink communication instead of spawning <code>ip</code> subprocesses and parsing their text output through <code>sed</code>, <code>awk</code>, and <code>grep</code> pipelines. The existing approach is both fragile (sensitive to output format changes) and inefficient (each <code>ip</code> invocation forks a subprocess, and the subsequent text processing requires further subprocesses for each filter stage).</p>
<ul>
<li><code>mwan3-get-addr.uc</code>: query interface addresses via <code>RTM_GETADDR</code></li>
<li><code>mwan3-manage-rules.uc</code>: check, add, and delete policy rules via <code>RTM_GETRULE</code>/<code>RTM_NEWRULE</code>/<code>RTM_DELRULE</code></li>
<li><code>mwan3-list-routes.uc</code>: list CIDR routes from a routing table via <code>RTM_GETROUTE</code>, used for nft set population</li>
<li><code>mwan3-create-iface-route.uc</code>: replicate routes from the main table into per-interface tables via <code>RTM_GETROUTE</code>/<code>RTM_NEWROUTE</code></li>
</ul>
<p>Each script produces exactly the output needed with no text parsing required. In several cases a single ucode invocation replaces what previously required multiple <code>ip</code> subprocess calls: <code>delete-iface</code> handles both address families in one call, <code>create-iface-route</code> performs route dump, filtering, deduplication, and table writes in a single process, and the rule <code>check</code> mode tests all three rule priorities from a single netlink dump.</p>
<hr>
<h3 id="mwan3-move-nft-batch-files-from-tmp-to-varrunmwan3">mwan3: move nft batch files from /tmp to /var/run/mwan3</h3>
<p>Move nft batch file paths out of the world-writable <code>/tmp</code> directory into the existing <code>/var/run/mwan3</code> runtime directory, which is root-owned and created by <code>mwan3_init</code> at service start.</p>
<p>Restrict the runtime directory to mode <code>0700</code> and create batch files with mode <code>0600</code> via <code>umask</code> in <code>common.sh</code> and an explicit mode argument in <code>mwan3rtmon</code>. Nothing outside of mwan3's own root-running code accesses this directory.</p>
<p>This is a minor hardening change that eliminates a theoretical symlink/race window in <code>/tmp</code> for the nft batch files used by <code>common.sh</code> and <code>mwan3rtmon</code>.</p>
<hr>
<h3 id="mwan3-flush-unreplied-conntrack-entries-on-interface-online">mwan3: flush unreplied conntrack entries on interface online</h3>
<p>When an interface transitions to online, flush conntrack entries that carry an mwan3 mark but have never received a reply. These entries can result from a race during startup or policy rebuild where a flow gets classified before the nft chains reflect the new state. The stale mark pins traffic to the wrong WAN, and because no reply arrives (the traffic is misrouted or blackholed), the application keeps retrying, refreshing the conntrack timeout indefinitely.</p>
<p>Flushing only UNREPLIED entries is non-disruptive to established connections. The next packet from the application creates a fresh conntrack entry that gets classified under the current policy.</p>
<p>The flush is called after <code>mwan3_set_policies_nft</code> in both the <code>connected</code> action (mwan3track declares interface online) and the <code>ifup</code> action with <code>status=online</code> (interface comes up with <code>initial_state</code> online). It is not called during init - the existing <code>mwan3_flush_stale_conntrack</code> handles the startup window.</p>
<hr>
<h3 id="mwan3-use-mwan3ct-for-all-conntrack-flush-operations">mwan3: use mwan3ct for all conntrack flush operations</h3>
<p>Replace all <code>conntrack</code> CLI invocations in <code>mwan3.sh</code> with <code>mwan3ct</code> calls:</p>
<ul>
<li><code>mwan3_flush_stale_conntrack()</code>: <code>conntrack -D --mark 0x0/MASK</code> becomes <code>mwan3ct flush --mark 0x0/MASK</code></li>
<li><code>mwan3_flush_marked_conntrack()</code>: shell loop spawning up to 63 <code>conntrack -D</code> processes becomes a single <code>mwan3ct flush --mark-any</code></li>
<li><code>mwan3_flush_conntrack()</code> ifdown branch: <code>conntrack -D --mark VAL/MASK</code> becomes <code>mwan3ct flush --mark VAL/MASK</code></li>
</ul>
<p>Since <code>mwan3ct</code> is part of the mwan3 package, the <code>command -v conntrack</code> guards and missing-tool warnings are no longer needed.</p>
<hr>
<h3 id="mwan3-add-mwan3ct-conntrack-flush-helper">mwan3: add mwan3ct conntrack flush helper</h3>
<p>Add a small C helper that uses libnetfilter_conntrack's <code>NFCT_Q_FLUSH_FILTER</code> to perform kernel-side filtered conntrack entry deletion by mark and/or status bits. This replaces all <code>conntrack</code> CLI usage in mwan3 with a purpose-built tool that is part of the package.</p>
<p>Motivations:</p>
<ul>
<li>Fragility: the previous approach for targeted unreplied-entry deletion required parsing <code>conntrack -L</code> output in shell, extracting fields with parameter expansion, and issuing per-entry <code>conntrack -D</code> calls. <code>mwan3ct</code> uses the library API directly with no text parsing.</li>
<li>Efficiency: the "flush all marked entries" operation previously spawned up to 63 separate <code>conntrack</code> processes in a shell loop. <code>mwan3ct</code> handles this in a single process with one netlink socket.</li>
<li>Correctness: the <code>conntrack</code> CLI ignores the <code>-u</code> status filter on delete operations, making it impossible to selectively delete only UNREPLIED entries. <code>mwan3ct</code> passes status filters to the kernel correctly via <code>NFCT_Q_FLUSH_FILTER</code>.</li>
</ul>
<p>The tool supports <code>--mark &lt;val&gt;/&lt;mask&gt;</code> for exact match, <code>--mark-any &lt;mask&gt;</code> for any non-zero mark within the mask, <code>--status &lt;val&gt;/&lt;mask&gt;</code> for status bit filtering, and combinations of these.</p>
<p>The Makefile gains build dependencies on <code>libnetfilter_conntrack</code> and <code>libmnl</code>, a runtime dependency on <code>libnetfilter-conntrack</code>, and removes the incorrect <code>PKGARCH:=all</code> (the package already compiled architecture-specific code via <code>sockopt_wrap.c</code>).</p>
<hr>
<h3 id="mwan3-fix-diagnostic-sanitizer-treating-0000-as-a-public-address">mwan3: fix diagnostic sanitizer treating 0.0.0.0 as a public address</h3>
<p>The <code>is_public_v4()</code> function in <code>mwan3-diag</code> only excluded loopback, RFC1918, and link-local addresses. The "this network" block (<code>0.0.0.0/8</code>, RFC 1122) and multicast/reserved ranges (<code>224.0.0.0/4</code> and above) were not excluded, causing them to be replaced with placeholder addresses in the diagnostic output rather than shown verbatim.</p>
<p>Add exclusions for <code>0.0.0.0/8</code> and <code>224.0.0.0/3</code> so that these non-routable addresses are preserved in diagnostic output alongside the other non-routable ranges already excluded.</p>
<hr>
<h3 id="186-version-364">18.6 Version 3.6.4</h3>
<p><strong>Summary:</strong> Fixes a regression in 3.6.3 that prevents mwan3's hotplug handler from running when the service is started but not explicitly enabled</p>
<hr>
<h3 id="mwan3-remove-enabled-check-from-hotplug-handler">mwan3: remove enabled check from hotplug handler</h3>
<p>The enabled guard added in b363e9b50 causes the hotplug handler to exit silently when the rc.d symlink is absent. This breaks the startup path: start_service spawns per-interface hotplug handlers via mwan3_ifup using env -i, and those handlers exit at the enabled check without creating per-interface nft chains, ip rules, or routing tables. All WAN-bound traffic then hits the unreachable last-resort rule and connectivity is lost.</p>
<p>The guard was intended to prevent hotplug side effects when mwan3 is disabled, but it is redundant. The nft table existence check that follows already covers every practical scenario: the table is created by start_service and deleted by stop_service, so it is present only while the service is running. A service that was started manually without being enabled should still process hotplug events.</p>
<hr>
<h3 id="187-version-363">18.7 Version 3.6.3</h3>
<p><strong>Summary:</strong> Correctness fixes for edge cases and incomplete feature implementations: fixes ipset and ICMP protocol translation issues when rules use family=any; rebuilds ip rules and routing tables on reload to handle interface reordering and family changes; completes track_gateway support that was missing from two tracking status checks; adds runtime configuration reload to mwan3track via a SIGHUP handler and fixes hotplug handler initialization issues and boot race conditions.</p>
<hr>
<h3 id="mwan3-fix-ipsetfamilyany-batch-failure-and-sticky-chain-orphan">mwan3: fix ipset+family=any batch failure and sticky chain orphan</h3>
<p>When a rule has <code>family=any</code> and references a UCI ipset whose declared family is ipv6, both the ipv4 and ipv6 passes run. The ipv4 pass finds the set absent from the kernel and pre-creates it as <code>ipv4_addr</code>. The ipv6 pass then generates an "ip6 daddr @setname" expression against an <code>ipv4_addr</code> set, causing the atomic nft batch to fail and killing all user rules.</p>
<p>Fix by adding a UCI lookup before the absent-set guard. The new helper <code>_mwan3_uci_ipset_addrtype()</code> maps a set name to its UCI-declared address type. When a referenced set is absent from the kernel but present in UCI, the incompatible pass is silently skipped and the compatible pass proceeds without pre-creation, since <code>mwan3_render_config_ipsets</code> will add the set to the same batch. The existing external-set guard (skip ipv6 pass, let ipv4 pre-create as <code>ipv4_addr</code>) is retained for sets absent from both kernel and UCI.</p>
<p>Separately, the preamble in <code>mwan3_set_user_rules()</code> that pre-creates and flushes per-rule sticky chains sourced chain names by listing <code>mwan3_rule_*</code> chains already present in the kernel table. After a UCI rule section is deleted, <code>mwan3_nft_reload_start</code> removes the chain in the same batch; however, because the batch has not yet been committed when the kernel is enumerated, the chain is still visible. The old code therefore flushes and reinserts the deleted chain into the batch, causing it to survive the reload as an orphan. Change the preamble to iterate over enabled UCI rule sections instead, so only chains for active rules are created and chains for deleted rules are not recreated.</p>
<hr>
<h3 id="mwan3-fix-icmp-protocol-translation-for-familyany-rules">mwan3: fix ICMP protocol translation for family=any rules</h3>
<p>UCI <code>proto=icmp</code> is family-agnostic, meaning ICMP appropriate to the address family of the rule. For <code>family=any</code> rules, mwan3 generates nft rules on two passes (ipv4 and ipv6). The ipv4 pass should emit "meta l4proto icmp" (protocol 1) and the ipv6 pass should emit "meta l4proto ipv6-icmp" (protocol 58).</p>
<p>Two interacting defects prevented the ipv6 translation from occurring:</p>
<p>The dedup optimisation that skips the ipv6 pass for <code>family=any</code> rules with no IP-version-specific elements incorrectly treated <code>proto=icmp</code> as family-agnostic. While TCP, UDP, and other L4 protocols share the same protocol number across families, ICMP (1) and ICMPv6 (58) are distinct protocols with distinct nft keywords. The dedup now excludes <code>proto=icmp</code> so both passes run.</p>
<p>The ICMP translation guard checked the UCI family config value instead of the current address-family pass variable. For <code>family=any</code>, the config value is "any", never "ipv6", so the translation never fired. The guard now checks the pass variable.</p>
<p>Without this fix, <code>family=any</code> rules with <code>proto=icmp</code> silently fail to match IPv6 ICMPv6 traffic.</p>
<hr>
<h3 id="mwan3-rebuild-ip-rules-and-routing-tables-on-reload">mwan3: rebuild ip rules and routing tables on reload</h3>
<p>The trigger conditions for this bug are narrow - all three must hold: mwan3 must be running with two or more interfaces, the user must reorder config interface sections in <code>/etc/config/mwan3</code> without adding or removing interfaces, and the user must trigger a reload rather than a full restart.</p>
<p>When <code>reload_service</code> is called, it atomically rebuilds the entire nft ruleset using the current UCI configuration, but never rebuilds the kernel ip rules or per-interface routing tables. These kernel-side structures encode the interface table ID (tid), which is derived from the ordinal position of config interface sections in <code>/etc/config/mwan3</code>.</p>
<p>If the UCI section ordering changes between start and reload, the nft rules use the new tid-to-mark mapping while the ip rules and routing tables still reflect the old mapping. Traffic is silently routed to the wrong WAN interface.</p>
<p>A secondary manifestation of the same root cause: if an interface's family option changes (e.g. ipv4 to ipv6), the old ip rules remain in the wrong address family and new rules are never created.</p>
<p>Add <code>mwan3_rebuild_iface_rules</code>, called from <code>reload_service</code> via <code>config_foreach</code>, which deletes and recreates ip rules and routing tables for each enabled, up interface. Modify <code>mwan3_delete_iface_rules</code> and <code>mwan3_delete_iface_route</code> to search both address families unconditionally, so that a family change cleans up rules left behind in the old family.</p>
<hr>
<h3 id="mwan3-fix-mwan3rtmon-hastracking-ignoring-trackgateway">mwan3: fix mwan3rtmon has_tracking ignoring track_gateway</h3>
<p>The <code>load_config</code> function in mwan3rtmon determines whether an interface has tracking enabled by checking only <code>track_ip</code>. Interfaces configured with <code>track_gateway</code> but no <code>track_ip</code> are incorrectly treated as having tracking disabled. This causes <code>get_track_status</code> to return "disabled" without consulting the tracker's PID and STARTED state files, bypassing the secondary health check guard in <code>handle_route_event</code>.</p>
<p>The shell equivalent in <code>common.sh</code> (<code>mwan3_get_mwan3track_status</code>) and the init script (<code>start_tracker</code>) both correctly account for <code>track_gateway</code>. Align mwan3rtmon's <code>has_tracking</code> logic to match by including <code>track_gateway</code> in the check.</p>
<hr>
<h3 id="mwan3-reload-tracker-configuration-on-service-reload">mwan3: reload tracker configuration on service reload</h3>
<p><code>mwan3track</code> reads UCI tracking parameters (interval, reliability, count, timeout, quality thresholds, etc.) once at startup and has no mechanism to update them at runtime. When <code>reload_service</code> rebuilds the nft ruleset, it does not restart tracker instances unless the tracker count changes, so any parameter changes made via UCI are silently ignored until a full service restart.</p>
<p>Add a <code>load_tracking_config()</code> function that extracts the parameter reads from <code>main()</code> into a reusable function called both at startup and on reload. Add a HUP signal handler to <code>mwan3track</code> that sets a reload flag and interrupts the current sleep/probe cycle. The main loop checks the flag at the same two points where IFDOWN and IFUP events are processed, re-reads UCI config, refreshes tracking parameters and track IPs, and clamps the score to the new down+up ceiling if it exceeds it.</p>
<p>In <code>reload_service</code>, send HUP to each running tracker instance after the existing rtmon HUP signals.</p>
<hr>
<h3 id="mwan3-fix-mwan3getmwan3trackstatus-ignoring-trackgateway">mwan3: fix mwan3_get_mwan3track_status ignoring track_gateway</h3>
<p>The <code>mwan3_get_mwan3track_status</code> function in <code>common.sh</code> returns "disabled" when no <code>track_ip</code> list entries are configured, without checking the <code>track_gateway</code> option. This causes the "mwan3 interfaces" CLI command to report "tracking is disabled" for interfaces that use gateway-only tracking, even though a tracker instance is actively running and monitoring the interface.</p>
<p>The rpcd ucode equivalent (<code>get_mwan3track_status</code>) already checks both <code>track_ip</code> and <code>track_gateway</code> before returning "disabled". Align the shell function with the same logic by also reading <code>track_gateway</code> before the early return.</p>
<hr>
<h3 id="mwan3-fix-hotplug-handler-to-exit-silently-when-mwan3-is-not-running">mwan3: fix hotplug handler to exit silently when mwan3 is not running</h3>
<p>Add <code>/etc/init.d/mwan3</code> enabled guard before <code>procd_lock</code> and <code>mwan3_init</code> so that hotplug events are silently ignored when mwan3 has no rc.d symlink.</p>
<p>Move the nft table existence check to before <code>mwan3_init</code>. Previously <code>mwan3_init</code> was called unconditionally, which created <code>/var/run/mwan3</code> as a side effect and corrupted the stopped-service indicator in the case where mwan3 is enabled but has been manually stopped. With the check before <code>mwan3_init</code>, the handler exits without side effects whenever the nft table is absent, covering both the boot-race window and the stopped-but-enabled case.</p>
<p>Remove the running check that followed <code>mwan3_init</code>. It was dead code: <code>mwan3_init</code> created <code>/var/run/mwan3</code> before the check ran, so <code>service_running()</code> always returned true and the exit path was unreachable.</p>
<hr>
<h3 id="188-version-362">18.8 Version 3.6.2</h3>
<p><strong>Summary:</strong> A set of defensive edge-case fixes and correctness improvements. Binds the mwan3rtmon route listener before the initial netlink dumps to close a narrow startup race window, and replaces <code>main_route_cache</code> with an on-demand kernel query to eliminate a class of cache-drift failures that could only manifest if route events arrived during the dump phase. Aligns shell and mwan3rtmon custom-set filtering so both paths apply identical exclusions. Adds a dormant <code>is_default_route</code> guard to <code>populate_connected_set</code> as a forward-compatibility precaution. Clamps <code>check_quality</code> to 0 when the configured track method cannot produce quality samples, preventing an arithmetic error in the unusual case where <code>check_quality</code> is paired with a non-ping method. Fixes <code>mwan3_track_clean</code> which targeted incorrect paths and was a no-op, and tightens ip rule deletion at <code>stop_service</code> to use content-based matching rather than a priority-range regex, which matters only when rule bases are configured outside the default 1000-3999 band.</p>
<hr>
<h4 id="mwan3-exclude-default-routes-from-mwan3rtmon-connected-set-population">mwan3: exclude default routes from mwan3rtmon connected set population</h4>
<p><code>populate_connected_set</code> filters routes with <code>is_cidr_route</code> and <code>is_linklocal_route</code> but does not check <code>is_default_route</code>. Currently dormant because rtnl returns <code>dst=null</code> for default routes, which <code>is_cidr_route</code> already rejects. Add an explicit <code>is_default_route</code> guard so the filter remains correct if the rtnl module ever emits "0.0.0.0/0" or "::/0" as a string instead of null.</p>
<hr>
<h4 id="mwan3-replace-mainroutecache-with-on-demand-kernel-query-in-mwan3rtmon">mwan3: replace main_route_cache with on-demand kernel query in mwan3rtmon</h4>
<p><code>main_route_cache</code> replaced an on-demand kernel query (<code>route_still_exists</code>, originally from e119a57e9) with a <code>route_key</code> map built from the initial dump and maintained incrementally from listener events.</p>
<p>The cache made the per-event ECMP-suppress check O(1) but introduced a class of dump-vs-listener race failure modes: any event arriving between the snapshot used to build the cache and the listener becoming active can double-count or undercount the cache, leading to stuck suppressions or premature cleanups of per-interface routing table entries with no self-healing path. The listener-first swap in an earlier commit shifts but does not eliminate this race.</p>
<p><code>route_still_exists</code> queries the kernel directly on every delete and is unaffected by cache drift. Cost is one <code>RTM_GETROUTE</code> dump per <code>DELROUTE</code> event whose per-iface table entry exists, single-digit milliseconds on typical routing-table sizes and infrequent in practice.</p>
<hr>
<h4 id="mwan3-align-shell-custom-set-population-with-mwan3rtmon-filtering">mwan3: align shell custom-set population with mwan3rtmon filtering</h4>
<p>The shell <code>mwan3_set_custom_set</code> function seeds <code>mwan3_custom_v4/v6</code> at start and reload so the sets are populated before mwan3rtmon starts. mwan3rtmon takes over incremental maintenance once procd schedules it.</p>
<p>Previously the shell and mwan3rtmon disagreed on filtering: the shell matched via IPv4_REGEX / IPv6_REGEX without excluding defaults or link-locals, while mwan3rtmon's <code>repopulate_custom_sets</code> excludes both via is_default_route and <code>is_linklocal_route</code>. Rewrite the shell function to apply the same exclusions (default, 0.0.0.0/0, ::/0, 169.254.<em>, fe80::</em>) and require a CIDR slash in IPv4 matches, so both paths produce identical set contents.</p>
<p>Also extend mwan3rtmon's <code>repopulate_custom_sets</code> to flush the set when no <code>rt_table_lookup</code> entries are configured, so removal of all <code>rt_table_lookup</code> entries followed by a reload correctly empties the set.</p>
<hr>
<h4 id="mwan3-bind-mwan3rtmon-route-listener-before-initial-dumps">mwan3: bind mwan3rtmon route listener before initial dumps</h4>
<p>mwan3rtmon's <code>main()</code> previously ran the initial netlink dumps before creating the route listener. Any <code>RTM_NEWROUTE</code> or <code>RTM_DELROUTE</code> emitted between mwan3rtmon process start and the rtnl.listener() call was discarded by the kernel (no multicast subscriber existed), leaving per-iface tables and the mwan3_custom_v4/v6 sets diverged from kernel state with no self-healing path.</p>
<p>Bind the listener first so its multicast socket starts buffering route events immediately, then issue the dumps. <code>ucode-mod-rtnl</code> creates the netlink socket and joins the multicast group synchronously inside the <code>rtnl.listener()</code> call, so the socket buffer captures concurrent events during the dump phase even before uloop.run() begins draining them.</p>
<p>Also add an explicit <code>repopulate_custom_sets()</code> call at startup so mwan3rtmon reconciles the <code>mwan3_custom_v4/v6</code> sets with kernel state once it begins running. The shell seeds these sets during <code>start_service</code>, but custom-table route changes between the shell pass and mwan3rtmon startup are unobserved by either side; the startup dump closes that window.</p>
<p>The connected set is unchanged; it already self-heals through a debounced rebuild triggered by every CIDR-route event.</p>
<hr>
<h4 id="mwan3-clamp-checkquality-for-track-methods-that-lack-losslatency">mwan3: clamp check_quality for track methods that lack loss/latency</h4>
<p>mwan3track's per-iteration case statement only populates <code>$loss</code> and <code>$latency</code> for the ping and httping methods. arping, nslookup, and nping-* set only <code>$result</code>. With <code>check_quality=1</code> configured against one of those methods the post-case decision block reaches</p>
<p><code>[ "$loss" -ge "$failure_loss" ] || [ "$latency" -ge ... ]</code></p>
<p>with empty (or stale from a previous iteration) operands, which raises a busybox ash arithmetic error and leaves the per-host up/down classification undefined.</p>
<p>Clamp <code>check_quality</code> to 0 at startup when the configured <code>track_method</code> cannot produce quality samples, and emit a notice so the user can see that the option was overridden. Apply the same clamp in <code>mwan3_load_track_ips</code> so that <code>LATENCY_/LOSS_</code> state files are still cleaned up after a config edit that transitions the interface from a quality-capable method to one that is not.</p>
<hr>
<h4 id="mwan3-clean-correct-per-interface-state-paths-in-mwan3trackclean">mwan3: clean correct per-interface state paths in <code>mwan3_track_clean</code></h4>
<p><code>mwan3_track_clean</code> is called from <code>stop_service</code> via <code>mwan3_interface_shutdown</code> to tear down per-interface runtime state. The original implementation removed <code>$MWAN3_STATUS_DIR/&lt;iface&gt;</code> and attempted to rmdir <code>$MWAN3_STATUS_DIR</code>. Neither matched where the per-interface state actually lives:</p>
<ul>
<li>tracker runtime state at <code>$MWAN3TRACK_STATUS_DIR/&lt;iface&gt;/(PID, STATUS, LATENCY_*, LOSS_*, TRACK_*, GATEWAY, ...)</code>, written by mwan3track;</li>
<li>hotplug state file at <code>$MWAN3_STATUS_DIR/iface_state/&lt;iface&gt;</code>, written by <code>mwan3_set_iface_hotplug_state</code>.</li>
</ul>
<p>The <code>$MWAN3_STATUS_DIR/&lt;iface&gt;</code> path the old code targeted does not exist, and the bare <code>$MWAN3_STATUS_DIR</code> rmdir always failed because that directory holds session-wide pinned state (mmx_mask and the iif_rule_base/fwmark_rule_base/unreachable_rule_base records used by mwan3_init across stop/start cycles), so the function was a no-op despite its name.</p>
<p>Point the rm at the correct per-interface paths and replace the bogus rmdir with a best-effort rmdir of <code>$MWAN3TRACK_STATUS_DIR</code> itself, which is the dir that should be empty once every iface has been cleaned. <code>$MWAN3_STATUS_DIR</code> is left intact because it holds state that must survive a stop/start cycle.</p>
<p>Errors are redirected to <code>/dev/null</code> so transient mismatches do not hit the console or logs. A comment in the function documents a small residual race with mwan3track that can briefly recreate the tracker dir between <code>rm</code> and <code>procd_kill</code>; the leak is bounded to one iteration and harmless on tmpfs.</p>
<hr>
<h4 id="mwan3-identify-ip-rules-to-delete-precisely-at-stopservice">mwan3: identify ip rules to delete precisely at stop_service</h4>
<p><code>stop_service</code> identified mwan3-owned ip rules with a fixed regex match on <code>^[1-3][0-9]{3}:</code>, limiting cleanup to priorities 1000-3999 and deleting every rule in that band regardless of origin. Two problems:</p>
<ol>
<li>When iif_rule_base, fwmark_rule_base or unreachable_rule_base are configured outside 1000-3999 (all three are exposed as globals and validated in mwan3_init), the affected tier of rules is leaked across service mwan3 stop and accumulates across stop/start cycles.</li>
</ol>
<ol>
<li>Even within the band, a rule placed by another package or by a local admin override sitting at one of those priorities is indistinguishable from a mwan3 rule and gets deleted.</li>
</ol>
<p>Replace the regex with a two-gate filter parsed in pure shell from ip rule list output. A rule is deleted only when it passes both:</p>
<ul>
<li>Priority gate: priority falls in one of the three configured rule-base ranges. The fwmark range extends to <code>fwmark_base + MM_UNREACHABLE</code> so the global blackhole and unreachable rules at <code>fwmark_base + MM_BLACKHOLE / MM_UNREACHABLE</code> are included.</li>
</ul>
<ul>
<li>Content gate: the line references a mwan3-owned routing table (<code>lookup &lt;N&gt;</code> with N in <code>1..MWAN3_INTERFACE_MAX</code>), or carries an fwmark masked by mwan3's <code>MMX_MASK</code>.</li>
</ul>
<p>To make the gates trustworthy at stop time, <code>mwan3_init</code> now persists <code>iif_rule_base</code>, <code>fwmark_rule_base</code> and <code>unreachable_rule_base</code> to state via <code>uci_toggle_state</code> at <code>start_service</code>, mirroring the existing <code>iface_max</code> persistence, and reads them back via <code>uci_get_state</code> on subsequent invocations. The state values reflect the rule bases the running instance actually created its rules with, even if <code>/etc/config/mwan3</code> has been edited since start. The <code>mmx_mask</code> state file doubles as the "instance started" indicator. The validation step that reverts to default bases on ordering-constraint violation runs only in the fresh-start branch, so persisted values are always post-validation.</p>
<hr>
<h3 id="189-version-361">18.9 Version 3.6.1</h3>
<p><strong>Summary:</strong> Extends the legacy mwan3 custom sets that were previously only loaded statically during <code>start_service()</code> and <code>reload_service()</code> from the tables defined in the UCI global config list option <code>rt_table_lookup</code> to be fully dynamic, using mwan3rtmon to listen for and to add and remove routes from the custom sets in response to <code>RTM_NEWROUTE</code> and <code>RTM_DELROUTE</code> events on the tables defined with <code>list rt_table_lookup &lt;tableid&gt;</code>. Adds a <code>SIGHUP</code> handler to mwan3rtmon to cause it to flush and repopulate these custom sets, ensuring that their contents remain in sync with any newly added or removed <code>list rt_table_lookup &lt;tableid&gt;</code> options in the mwan3 config.</p>
<hr>
<h4 id="mwan3-reload-mwan3rtmon-config-on-mwan3-reload-via-sighup">mwan3: reload mwan3rtmon config on mwan3 reload via SIGHUP</h4>
<p>mwan3rtmon loads its UCI configuration once at startup and holds it in memory for the lifetime of the process. This includes <code>extra_table_set</code>, the in-memory set of routing table IDs derived from the <code>rt_table_lookup</code> UCI option. When mwan3 reloads without a full restart (the common path when interface count does not change), mwan3rtmon stays running with its original <code>extra_table_set</code> intact. If the operator changes <code>rt_table_lookup</code> during that reload, mwan3rtmon will continue routing events against the old table list until it is manually restarted.</p>
<p>Fix this by adding a <code>SIGHUP</code> handler to mwan3rtmon. On receipt of <code>SIGHUP</code> it calls <code>load_config()</code> to refresh all in-memory UCI state and then <code>repopulate_custom_sets()</code> which performs a live netlink route dump of every table now in <code>extra_table_set</code> and rebuilds <code>mwan3_custom_v4/v6</code> from scratch. The flush-then-add pattern ensures that tables removed from rt_table_lookup have their routes evicted from the custom sets as well as tables newly added having their current routes immediately mirrored in.</p>
<p><code>reload_service()</code> in the init script sends <code>SIGHUP</code> to the rtmon_ipv4 (and rtmon_ipv6 if IPv6 is enabled) procd instances via <code>procd_send_signal</code> after <code>mwan3_nft_reload_commit</code> and the ip rule updates, so the nft sets and ip rules are fully consistent before mwan3rtmon re-dumps.</p>
<p>The shell-level <code>mwan3_set_custom_sets()</code> call already present in <code>reload_service()</code> is retained. It runs inside the atomic nft batch and provides a synchronous static snapshot that ensures the sets are never empty during the reload window. The SIGHUP-triggered re-dump that follows is the authoritative update because it runs after the batch commits and picks up any route changes that occurred in the interval between the shell dump and the signal delivery.</p>
<hr>
<h4 id="mwan3-handle-rttablelookup-route-events-dynamically">mwan3: handle <code>rt_table_lookup</code> route events dynamically</h4>
<p>The <code>rt_table_lookup</code> feature populated the <code>mwan3_custom_v4</code> and <code>mwan3_custom_v6</code> nftables sets at startup by reading existing routes from the configured tables, but did not update those sets when routes changed at runtime. Route additions and deletions in <code>rt_table_lookup</code> tables were therefore not reflected until mwan3 was restarted.</p>
<p>Add <code>handle_custom_set_event()</code>, called from <code>handle_route_event()</code> whenever a route event arrives for a table listed in <code>extra_table_set</code>. <code>RTM_NEWROUTE</code> events add the destination to the appropriate custom set; <code>RTM_DELROUTE</code> events remove it. Default routes and link-local routes are excluded, matching the exclusions already applied at startup.</p>
<hr>
<h3 id="1810-version-36">18.10 Version 3.6</h3>
<p><strong>Summary:</strong> Version 3.6 adds three user-visible features to mwan3 rules. Rules now support an <code>fwmark</code>/<code>fwmask</code> option to match packets by meta mark using a masked comparison, working alongside or instead of address and ipset matching; mwan3 logs a warning if the fwmask overlaps its internal <code>MMX_MASK</code> since such a mask would match packets already carrying an mwan3 classification mark. The ip rule priority tiers for per-interface rules are now configurable via three new globals UCI options (<code>iif_rule_base</code>, <code>fwmark_rule_base</code>, <code>unreachable_rule_base</code>), shifting from the fixed 1000/2000/3000 defaults; two ordering constraints are enforced at startup and rule deletion is rewritten to use content-based matching so it remains correct across base or <code>mmx_mask</code> changes. Rules gain <code>option enabled 0/1</code>, consistent with interfaces, ipsets, and members.</p>
<p>The LuCI interface is updated throughout to reflect all three additions. The rule modal gains an Fwmark field with <code>MMX_MASK</code> overlap validation; the Globals tab gains the three base priority fields with live cross-field ordering validation and an automatic mwan3 restart when any base changes. The Routing Health tab gains the unreachable rule row per interface and marks an interface as degraded if the unreachable rule is absent; base priorities are now displayed dynamically from the rpcd endpoint rather than assumed from fixed offsets. The Traffic Simulator, rule shadowing analysis, Status Overview, and diagnostics helper are all updated to handle fwmark matching, disabled rules, and configurable bases. Mutual enable protection is added to prevent the silent misconfiguration of an enabled rule referencing a disabled IP set. mwan3track now validates that <code>libwrap_mwan3_sockopt.so</code> is present at startup, exiting with a clear error rather than silently producing incorrect tracking results if the library is missing.</p>
<hr>
<h3 id="mwan3-add-per-interface-unreachable-rule-to-rpcd-routinghealth">mwan3: add per-interface unreachable rule to rpcd routing_health</h3>
<p>The <code>routing_health()</code> rpcd endpoint reported iif and fwmark rule presence per interface but omitted the unreachable rule. This meant the Routing Health tab could not detect a missing unreachable rule, and would show a fully healthy interface even if its unreachable safety net was absent.</p>
<p>Add <code>unreach_rule</code> (present + priority) to each interface in the response, and include the unreachable base priority in the rule_bases object returned to the frontend.</p>
<hr>
<h3 id="mwan3-update-rpcd-routinghealth-for-configurable-bases-and-dynamic-mmxmask">mwan3: update rpcd <code>routing_health</code> for configurable bases and dynamic mmx_mask</h3>
<p>The <code>routing_health()</code> function in the rpcd ucode module used hardcoded constants for the ip rule base priorities (1000/2000) and the maximum interface count (63). This meant the Routing Health tab reported incorrect data when configurable rule base priorities were in use, and would also be wrong for any installation with a non-default <code>mmx_mask</code>.</p>
<p>Read <code>iif_rule_base</code>, <code>fwmark_rule_base</code>, and <code>unreachable_rule_base</code> from UCI globals with the same defaults and ordering constraint validation as <code>mwan3_init</code> in <code>common.sh</code>. Derive <code>mmdefault</code> and <code>iface_max</code> dynamically from the configured <code>mmx_mask</code> by counting set bits, replacing the hardcoded <code>MAX_IFACES</code> constant. This ensures the blackhole and unreachable global policy rule priorities, stale rule range detection, and ordering constraint validation all reflect the actual runtime configuration.</p>
<p>Add the unreachable rule tier to valid priority tracking and stale rule detection. Previously only the iif and fwmark tiers were checked, so per-interface unreachable rules were invisible to the health report and could be falsely flagged as stale.</p>
<p>Return the active rule base priorities in the rpcd response so the frontend can display them dynamically rather than assuming fixed offsets.</p>
<hr>
<h3 id="mwan3-add-configurable-ip-rule-base-priorities-and-fwmark-rule-matching">mwan3: add configurable ip rule base priorities and fwmark rule matching</h3>
<p>fwmark/fwmask rule matching is added to <code>mwan3_set_user_nft_rule()</code>. A policy rule may now specify an fwmark and fwmask to match against the packet's meta mark using a masked comparison. The match is address-family agnostic, operating on meta mark rather than IP-layer fields. fwmark and fwmask must be specified together; a rule with only one set is skipped with a warning. A warning is also logged if the fwmask overlaps mwan3's internal <code>MMX_MASK</code>, since such a mask would match packets already carrying an mwan3 classification mark.</p>
<p>Three new globals UCI options: <code>iif_rule_base</code>, <code>fwmark_rule_base</code>, and <code>unreachable_rule_base</code> allow the ip rule priority tiers at which mwan3 installs its per-interface rules to be shifted from the original fixed offsets of 1000/2000/3000. Defaults are unchanged, so existing installations are unaffected and other packages that rely on mwan3's rules being inserted at those priorities remain unaffected insofar as the defaults are not overridden.</p>
<p>Two ordering constraints are enforced at startup:</p>
<p><code>iif_rule_base + MWAN3_INTERFACE_MAX &lt; fwmark_rule_base</code></p>
<p><code>fwmark_rule_base + MWAN3_INTERFACE_MAX + 1 &lt; unreachable_rule_base</code></p>
<p>If either constraint is violated, all three values are reverted to defaults with a logged warning.</p>
<p>The ip rule deletion logic in <code>mwan3_delete_iface_rules()</code> is rewritten to use content-based matching rather than a numeric range filter. Previously rules were deleted by checking whether their priority modulo 1000 equalled the interface id and fell in the range 1001-3999. This breaks when configurable bases place rules outside that range or at priorities that collide modulo 1000. The new logic identifies the iif rule by the iif keyword and table id, then discovers the fwmark/mask value from the lookup rule and deletes both the fwmark lookup and unreachable rules by content. This also handles upgrades from pre-configurable-base versions and <code>MMX_MASK</code> changes transparently.</p>
<p>For rules with <code>family=any</code> that have no IP-layer match criteria (no <code>src_ip</code>, <code>dest_ip</code>, or <code>pset</code>), the nft expression is address-family agnostic and would be emitted identically on both the ipv4 and ipv6 generation passes. The ipv6 pass is now skipped in that case to avoid installing a duplicate chain rule.</p>
<hr>
<h3 id="mwan3-add-missing-ucode-mod-socket-dependency">mwan3: add missing <code>ucode-mod-socket</code> dependency</h3>
<p>mwan3-diag imports the ucode socket module to normalise IPv4 and IPv6 address strings via <code>sock.sockaddr()</code>. The <code>ucode-mod-socket</code> package was not listed in DEPENDS, so on a fresh install the module would be absent and mwan3-diag would abort at startup with an import error.</p>
<p>Add <code>+ucode-mod-socket</code> to the package DEPENDS.</p>
<hr>
<h3 id="mwan3-wire-validatewrap-startup-check-for-libwrapmwan3sockopt">mwan3: wire <code>validate_wrap()</code> startup check for <code>libwrap_mwan3_sockopt</code></h3>
<p>If <code>libwrap_mwan3_sockopt.so</code> is absent, mwan3track now exits with a clear error at startup rather than silently producing incorrect tracking results. Also define the library path as a single <code>WRAP_LIB</code> constant, and fix the file test from <code>-x</code> to <code>-f</code>.</p>
<hr>
<h3 id="mwan3-add-option-enabled-support-for-rules">mwan3: add option enabled support for rules</h3>
<p>UCI rules lacked the <code>option enabled 0/1</code> guard that interfaces, ipsets, and members already support. Add <code>config_get_bool</code> checks in two places:</p>
<ul>
<li><code>mwan3_set_user_nft_rule(</code>: skip disabled rules at install time, consistent with the default of 1 (enabled) so existing configs without the option are unaffected. </li>
</ul>
<ul>
<li><code>iface_rule()</code> inside <code>mwan3_set_user_iface_rules()</code>: skip disabled rules when scanning for <code>src_iface</code> matches so a disabled rule with a matching src_iface does not trigger a needless mwan3_set_user_rules rebuild on ifup hotplug events.</li>
</ul>
<hr>
<h3 id="luci-app-mwan3-fix-hardcoded-rule-base-priorities-in-diagnostics-helper">luci-app-mwan3: fix hardcoded rule base priorities in diagnostics helper</h3>
<p>The luci-mwan3 helper script used hardcoded iif (1000) and fwmark (2000) base priorities when checking ip rules for an interface. With configurable rule base priorities, the diagnostic "Check IP rules" would grep for the wrong priorities and report rules as missing when they are present at different priorities.</p>
<p>Read <code>iif_rule_base</code>, <code>fwmark_rule_base</code>, and <code>unreachable_rule_base</code> from UCI globals with the same defaults as the backend. Add the unreachable rule check that was missing entirely. Improve the output format to show each rule type individually with its expected priority and a summary count.</p>
<hr>
<h3 id="luci-app-mwan3-filter-disabled-rules-from-status-overview">luci-app-mwan3: filter disabled rules from Status Overview</h3>
<p>The Overview tab displayed all configured rules regardless of enabled state. Disabled rules have no runtime effect and their presence in the overview misrepresents the active ruleset.</p>
<p>Skip rules with enabled=0 so the overview reflects what mwan3 actually evaluates at runtime.</p>
<hr>
<h3 id="luci-app-mwan3-display-per-interface-unreachable-rule-in-routing-health">luci-app-mwan3: display per-interface unreachable rule in Routing Health</h3>
<p>The Routing Health tab showed only the iif and fwmark ip rules per interface. The unreachable rule, which prevents packets marked for a down interface from being silently misrouted via the main routing table, was not visible.</p>
<p>Add an unreachable rule row to the per-interface health card using the <code>unreach_rule</code> data now provided by the rpcd endpoint. Include the unreachable rule in the health calculation so a missing unreachable rule is flagged as degraded. Update the field guide to describe all three rule tiers and display the unreachable base priority dynamically.</p>
<hr>
<h3 id="luci-app-mwan3-add-fwmark-rule-support-and-configurable-rule-base-priorities">luci-app-mwan3: add fwmark rule support and configurable rule base priorities</h3>
<p>Adds an Fwmark field to the rule edit modal. The field takes a combined combined value/mask hex expression (e.g. <code>0x80000/0xff0000</code>) and splits it across the fwmark and fwmask UCI options on write. Validation checks hex format and rejects masks that overlap mwan3's internal <code>MMX_MASK</code> bits. The Destination column in the rule list is extended to append a <code>mark:value/mask</code> token when fwmark/fwmask are set, so fwmark-only rules do not appear as wildcard entries.</p>
<p>Traffic Simulator tab is extended to match fwmark rules. An Fwmark input field is added to the simulator form; its value is compared against each rule's fwmark/fwmask pair using a masked comparison, with an empty or absent mark treated as zero. <code>matchSummary</code> is updated to include the fwmark expression in the match description, and a pre-existing bug is fixed where <code>family=any</code> was displayed as IPv6 rather than being omitted.</p>
<p>Status Overview tab is extended to display the fwmark/fwmask expression in the Match column of the status page rules grid. Previously, rules that match solely on packet mark displayed as '(all traffic)'.</p>
<p>Globals tab gains three new form fields: <code>iif_rule_base</code>, <code>fwmark_rule_base</code>, and <code>unreachable_rule_base</code>, exposing the ip rule base priority options added to mwan3. Each field carries live cross-field validation enforcing the two ordering constraints required by the backend:</p>
<p><code>iif_rule_base + MWAN3_INTERFACE_MAX &lt; fwmark_rule_base</code></p>
<p><code>fwmark_rule_base + MWAN3_INTERFACE_MAX + 1 &lt; unreachable_rule_base</code></p>
<p>The tooltip for the <code>fwmark</code> and <code>unreachable</code> fields shows the minimum required offset above the preceding base, computed at render time from the current firewall mask setting so the displayed value reflects the actual interface capacity.</p>
<p>Changing any of the three base values requires a full mwan3 restart to delete and recreate all ip rules at the new priorities. <code>handleSaveApply</code> is overridden to detect base changes by comparing pre- and post-save UCI values and, if a change is found, registers a one-shot uci-applied event listener that calls mwan3 restart after the config is committed to disk. Pages that change only unrelated globals settings (logging, loglevel) do not trigger a restart.</p>
<p>Routing Health field guide displays the actual configured rule base priorities dynamically rather than hardcoded 1000/2000 labels, using the rule_bases object returned by the rpcd routing_health endpoint.</p>
<p>Configuration tab rule shadowing analysis is extended to account for fwmark/fwmask matching, and disabled rules (enabled=0) are filtered out before the analysis since mwan3 skips them at runtime.</p>
<hr>
<h3 id="luci-app-mwan3-simulator-skip-disabled-rules-during-matching">luci-app-mwan3: simulator: skip disabled rules during matching</h3>
<p>The introduction of the enabled flag for rules created a gap in the traffic path simulator: disabled rules were still evaluated and could appear as matching results, giving incorrect output.</p>
<p>Filter disabled rules from both the nftset collection pass and the rule matching loop so that the simulator reflects the active ruleset as mwan3 sees it at runtime.</p>
<hr>
<h3 id="luci-app-mwan3-ipsetrule-add-mutual-enable-protection">luci-app-mwan3: ipset/rule: add mutual enable protection</h3>
<p>An enabled rule referencing a disabled IP set silently misfires at runtime: mwan3 auto-creates the set as empty and the rule never matches any traffic. Three guards are added to prevent this misconfiguration.</p>
<p>On the IP sets tab, the Enable checkbox is rendered disabled (greyed out) when the set is currently enabled and referenced by at least one enabled rule. This prevents the user from disabling a set that is in active use.</p>
<p>On the rules tab, the Enable checkbox is rendered disabled when the rule is currently disabled and references at least one disabled IP set. This prevents the user from re-enabling a rule whose set dependency is not yet satisfied. The guard applies only when the rule is currently disabled; if the rule is already enabled the checkbox remains free so the user can uncheck it to correct the state.</p>
<p>On the rules tab, the Source NFT set and Destination NFT set dropdowns exclude any set that is explicitly disabled in the mwan3 UCI config. This prevents a disabled set from being assigned to an already-enabled rule via the modal form.</p>
<p>Both tabs also show an explanatory note in the section header describing when the checkbox will be greyed.</p>
<p>The implementation uses this.readonly set temporarily before calling the parent <code>form.Flag renderWidget</code>, which passes disabled: true to ui.Checkbox and sets the HTML disabled attribute on the input element. The enabled state is read via <code>uci.get()</code> rather than the <code>cfgvalue</code> argument, which is unreliable for options stored at their default value.</p>
<hr>
<h3 id="luci-app-mwan3-use-addrport-format-in-overview-rule-listing">luci-app-mwan3: use <code>addr:port</code> format in Overview rule listing</h3>
<p>The Status--&gt;Overview rule grid was displaying source and destination match fields as separate tokens (src:, src ipset:, sport:, dst:, dst ipset:, dport:), inconsistent with the Network--&gt;Rules grid which combines address/ipset and port into a single addr:port expression.</p>
<p>Add a <code>fmtAddr()</code> helper mirroring the textvalue logic used in <code>rule.js</code>: address (or ipset if no address) is joined with port as <code>addr:port</code>, or <code>*:port</code> when only a port is set. The overview now shows <code>proto</code>, <code>src:</code> and <code>dst:</code> in the same compact format as the Rules grid.</p>
<hr>
<h3 id="luci-app-mwan3-add-enable-column-to-rules-grid">luci-app-mwan3: add Enable column to rules grid</h3>
<p>Add an inline Enable checkbox to the rules GridSection. The checkbox uses <code>o.editable = true</code> so it is interactive directly in the grid row without requiring the modal to be opened.</p>
<p>To accommodate the new column without widening the grid, the separate Source port and Destination port columns are merged into the Source and Destination columns respectively. The merged <code>textvalue</code> format is <code>ddress:port</code> or <code>ipset_name:port</code> when both are set, <code>address</code> or <code>ipset_name</code> when only an address is present, <code>*:port</code> when only a port is set, and <code>-</code> when neither is set. <code>src_port</code> and <code>dest_port</code> are marked modalonly so they remain editable via the modal.</p>
<p>The Policy assigned column label is shortened to Policy.</p>
<hr>
<h3 id="1811-version-353">18.11 Version 3.5.3</h3>
<p><strong>Summary:</strong> Version 3.5.3 adds two major LuCI features and a set of bug fixes and routing reliability improvements.</p>
<p>The Policy tab is rewritten with a tier-based policy builder that lets users specify failover tiers and per-interface load-balancing weights without manually creating member definitions. The builder handles member creation and garbage-collects orphaned members automatically, and updates the policy list to display IPv4 and IPv6 policies in symbolic form. The traffic simulator gains hostname resolution: Source and Destination fields now accept hostnames as well as IP addresses, resolved via the new <code>resolve_host</code> rpcd method against the local DNS server. The new Policy tab metaphor and layout should greatly improve how both new and existing users get to grips with mwan3, whose flexible but less than intuitive interface-member-policy definitions sometimes pose a substantial barrier to immediate use.</p>
<p>Three new rpcd methods - <code>nftset_flush</code>, <code>nftset_reload</code>, and <code>nftset_resolve</code> - allow the LuCI IP Sets tab to flush a set, repopulate it from UCI static entries and a loadfile, or trigger fresh DNS resolution by sending SIGHUP to dnsmasq before resolving configured domains. Matching Flush, Reload, and Resolve buttons appear on each set panel alongside the existing Expand button. <code>get_nftset_members</code> is fixed to unwrap the counter-decorated element wrapper that nft emits for counter-enabled sets, which previously caused all members of such sets to be silently dropped, breaking simulator rule matching for counter-enabled nftsets.</p>
<p><code>mwan3_create_iface_route</code> is changed to use <code>ip route replace</code> instead of <code>ip route add</code>, making route installation unconditionally idempotent and eliminating spurious <code>EEXIST</code> errors when mwan3rtmon or a racing event had already inserted the route. A missing default argument in the <code>config_get_bool</code> call for the <code>enabled</code> option is added, suppressing cosmetic <code>sh: out of range</code> noise on startup.</p>
<hr>
<h4 id="mwan3-add-nftsetflush-nftsetreload-and-nftsetresolve-rpc-methods">mwan3: add nftset_flush, nftset_reload, and nftset_resolve RPC methods</h4>
<p><code>nftset_flush</code> empties a named nft set via <code>nft flush set</code>.</p>
<p><code>nftset_reload</code> flushes the set then repopulates it from the UCI ipset config: static entries from the <code>entry</code> list and any entries read from the <code>loadfile</code> path.</p>
<p><code>nftset_resolve</code> sends <code>SIGHUP</code> to dnsmasq to clear its cache, then iterates the <code>domain</code> list for the ipset and calls <code>nslookup_resolve</code> for each, triggering fresh DNS lookups that cause dnsmasq to populate the set via its nftset integration.</p>
<hr>
<h4 id="mwan3-fix-missing-default-in-configgetbool-enabled-call">mwan3: fix missing default in config_get_bool enabled call</h4>
<p><code>mwan3_update_dev_to_table</code> iterates all mwan3 interface sections to build the device-to-routing-table map. The <code>config_get_bool</code> call for the enabled option had no default argument. When an interface section has no explicit option enabled set, get_bool receives an empty value and an empty default, falls through to the wildcard case, and returns an empty string. The subsequent <code>[ "$enabled" -eq 0 ]</code> comparison then receives an empty string as its left operand, causing busybox ash to emit sh: out of range (empty string has no digits consumed, triggering the error path in the test builtin's integer conversion).`</p>
<p>The error is cosmetic. The failed test exits non-zero, so the &amp;&amp; return does not fire and the interface is treated as enabled, which is correct, but the error messages are misleading and concern users.</p>
<p>Fix by adding the missing default of 1, consistent with all other <code>config_get_bool</code> enabled calls in the file.</p>
<hr>
<h4 id="mwan3-use-ip-route-replace-in-mwan3createifaceroute">mwan3: use ip route replace in mwan3_create_iface_route</h4>
<p>The previous ip route add would fail with <code>EEXIST</code> if the route was already present in the per-interface table -- for example when mwan3rtmon had already copied it from the main table, or when a connected event raced an ifup event. The string-match dedup check intended to prevent this was fragile: ip route list output for the same route can differ in text representation between table main and a per-interface table (field ordering, explicit metric 0, etc.), causing the check to miss a match and fall through to a duplicate add.</p>
<p>ip route replace uses <code>NLM_F_CREATE | NLM_F_REPLACE</code> at the kernel level, making the operation unconditionally idempotent. This is the same semantics mwan3rtmon already uses for its route copies. The duplicate-detection pre-check is retained as a cheap optimisation to avoid a redundant syscall when the route is already known to be present, but it is no longer safety-critical.</p>
<hr>
<h4 id="mwan3-add-resolvehost-rpc-method-for-hostname-resolution">mwan3: add resolve_host RPC method for hostname resolution</h4>
<p>Add <code>nslookup_resolve()</code> helper and <code>resolve_host</code> RPC method to allow the LuCI traffic simulator to accept hostnames in addition to IP addresses. The resolver invokes <code>/bin/busybox nslookup</code> against the local DNS server (127.0.0.1), populating any dnsmasq nftset entries configured for the queried domain as a side effect. Returns separate v4 and v6 address arrays; the family parameter restricts resolution to A or AAAA records. Input is validated against [a-zA-Z0-9._-]+ before use.</p>
<hr>
<h4 id="mwan3-fix-getnftsetmembers-to-unwrap-counter-decorated-elements">mwan3: fix get_nftset_members to unwrap counter-decorated elements</h4>
<p>nft wraps set elements in <code>{"elem":{"val":"...","counter":{...}}}</code> objects when the set has the counter flag. The previous code passed these wrapper objects directly to <code>parse_elem_val()</code>, which returned null, silently dropping all members of counter-enabled sets. This caused the traffic simulator to fail to match rules that reference counter-enabled nftsets.</p>
<p><code>get_nftset_elements()</code> already handled this case correctly; apply the same unwrap logic to <code>get_nftset_members()</code>.</p>
<hr>
<h4 id="luci-app-mwan3-add-flush-reload-and-resolve-buttons-to-ip-sets-tab">luci-app-mwan3: add Flush, Reload, and Resolve buttons to IP Sets tab</h4>
<p>Each set panel gains Flush, Reload, and Resolve action buttons alongside the existing Expand button. Flush empties the nft set. Reload flushes and repopulates from UCI static entries and the loadfile. Resolve is shown only for sets with configured domains; it sends SIGHUP to dnsmasq to clear its cache and triggers fresh DNS lookups to repopulate the set via dnsmasq's nftset integration.</p>
<p>All three buttons disable during the async operation and refresh the element count on completion. A page-level description explains each button's behaviour.</p>
<p>ACL grants for nftset_flush, nftset_reload, and nftset_resolve are added to both read-only and read-write rpcd permission groups.</p>
<hr>
<h4 id="luci-app-mwan3-reimplement-the-policy-tab">luci-app-mwan3: reimplement the Policy tab</h4>
<p>mwan3's use of members, metrics and weights is counter-intuitive for new users and even a barrier to effective use for some.</p>
<p>Create a new policy builder modal that allows policies to be specified by choosing interfaces and percentages to create a load balanced policy and policy tiers to allow for failover configurations.</p>
<p>The policy builder automatically handles member definitions, meaning that users never have to be concerned with creating members and with calculating appropriate metrics and weights. The policy builder does it all according to the configured tiers (metrics) and weights.</p>
<p>Update the policy list to remove the member column and instead show the IPv4 and IPv6 policies in a symbolic form that makes the configuration immediately obvious to the user.</p>
<p>Policy builder will automatically garbage collect orphaned members; a "Delete unused member definitions" checkbox in the policy tab header controls whether orphaned members are removed on save or not.</p>
<p>The Member tab is kept visible for manual inspection and editing of metric/weight values, although with the advent of policy builder, it is entirely redundant. Preserved more as a comfort to users who've been using mwan3 for a long time than for any really functional need.</p>
<hr>
<h4 id="luci-app-mwan3-add-hostname-resolution-to-traffic-simulator">luci-app-mwan3: add hostname resolution to traffic simulator</h4>
<p>Allow the Source IP/Name and Destination IP/Name fields in the traffic simulator to accept hostnames in addition to IP addresses.</p>
<p>When a hostname is entered, the simulator calls the resolve_host RPC method to resolve it via the local DNS server before running the simulation. The resolved address is displayed inline as <code>Resolved: x.x.x.x (+N more)</code>. Resolution errors abort the simulation with an error message.</p>
<p>The address family selector controls A vs AAAA record resolution; IPv4 is preferred when the selector is set to both families. Update the ACL to permit the resolve_host method, update field labels and placeholder text to indicate hostname support, and widen the input fields.</p>
<hr>
<h3 id="1812-version-352">18.12 Version 3.5.2</h3>
<p><strong>Summary:</strong> Version 3.5.2 is a bug-fix and maintenance release. It corrects a misrouting bug where kernel-generated NDP Neighbor Solicitation probes entered <code>mwan3_output</code> without a conntrack entry, fell through to <code>mwan3_rules</code>, and received a WAN policy mark that caused the kernel to probe the gateway via the wrong interface, cycling the NDP entry to FAILED state and breaking WRAP ping tracking for that interface. It updates the package dependency from <code>ip</code> to <code>ip-full</code> to ensure the full iproute2 implementation is always present, since the busybox <code>ip</code> is a minimal subset that does not support all options mwan3 requires. It adds <code>mwan3-diag</code>, a ucode diagnostic script installed to <code>/usr/sbin/mwan3-diag</code> that collects a comprehensive snapshot of mwan3 state -- interface status, policy routing rules, nftables ruleset, routing tables, conntrack summary and system log -- with all public IP addresses anonymised with stable placeholders so output can be shared safely.</p>
<hr>
<h4 id="mwan3-add-mwan3-diag-network-diagnostic-script">mwan3: add mwan3-diag network diagnostic script</h4>
<p>mwan3-diag is a ucode script that collects a comprehensive snapshot of mwan3 state. It gathers interface status, policy routing rules, nftables ruleset, routing tables, conntrack summary, system log and anonymises all public IP addresses with stable placeholders so output can be shared safely.</p>
<hr>
<h4 id="mwan3-depend-on-ip-full-instead-of-ip">mwan3: depend on ip-full instead of ip</h4>
<p>The busybox ip implementation is a minimal subset of iproute2 and does not support all options and subcommands that mwan3 requires for correct operation. Depend on ip-full to ensure the full iproute2 implementation is always present.</p>
<hr>
<h4 id="mwan3-bypass-ndp-in-mwan3output-to-prevent-re-routing-of-ndp-probes">mwan3: bypass NDP in mwan3_output to prevent re-routing of NDP probes</h4>
<p>Kernel-generated NDP Neighbor Solicitation probes start with mark=0 and enter mwan3_output. They have no conntrack entry, so the ct mark restore is a no-op. They are not matched by mwan3_connected, mwan3_custom, or mwan3_dynamic. They fall through to mwan3_rules, where the default IPv6 rule (ip6 daddr ::/0) applies a WAN policy mark -- the same mark that would be assigned to outbound user traffic. Because mwan3_output is type route, the mark change triggers a routing re-evaluation, which may route the probe to a different WAN interface than the one whose gateway the kernel is trying to resolve. The gateway NDP entry cycles to FAILED state, and subsequent WRAP ping probes are dropped because the kernel cannot resolve the gateway MAC address.</p>
<p>Add an icmpv6 NDP accept rule at the top of mwan3_output, mirroring the equivalent rule already present in mwan3_prerouting.</p>
<hr>
<h3 id="1813-version-351">18.13 Version 3.5.1</h3>
<p><strong>Summary:</strong> Version 3.5.1 is a bug-fix and maintenance release. It corrects a silent failure in <code>mwan3rtmon</code> where route replication to per-interface routing tables was completely non-functional, adds nft set flag-change detection on reload so that changing a set's timeout, counter, or size options takes effect immediately without requiring a full service restart, suppresses spurious stderr noise from ip rule and ip route operations during upgrades and teardown, and removes version number references from comments.</p>
<hr>
<h4 id="mwan3-fix-mwan3rtmon-route-replication-broken-by-stale-table-name">mwan3: fix mwan3rtmon route replication broken by stale table name</h4>
<p><code>refresh_active_chains()</code> filtered for chains in <code>table inet fw4</code> instead of <code>table inet mwan3</code>. Because mwan3's interface chains live in <code>table inet mwan3</code>, the <code>active_chains</code> cache was always empty. Every caller that depended on it -- <code>is_iface_nft_active</code>, <code>get_active_tids</code>, <code>populate_iface_routes</code>, and the route-replication path in <code>handle_route_event</code> -- silently did nothing. Route replication from the main routing table to per-interface routing tables was completely non-functional.</p>
<p>Most deployments did not notice because <code>mwan3_create_iface_route</code> in the hotplug script populates per-interface tables at ifup time, covering the static routing table case. The bug manifests when routes are added to or removed from the main table after mwan3 starts (VPN tunnels, PPPoE reconnection, etc.).</p>
<p>The connected set population path (<code>populate_connected_set</code>) was unaffected by the bug and continues to work correctly.</p>
<hr>
<h4 id="mwan3-detect-nft-set-flag-changes-on-reload-and-deleterecreate-as-needed">mwan3: detect nft set flag changes on reload and delete+recreate as needed</h4>
<p><code>nft add set</code> is idempotent on existence: if a set already exists it returns without error but does not update its flags (timeout, counters, size). A reload that changed any of these flags silently left the live set with the old configuration until the next full service restart.</p>
<p>Add <code>_mwan3_nft_time_to_sec</code> to parse nft time unit strings (<code>1h</code>, <code>5m</code>, <code>300s</code>) into a common integer-seconds representation for comparison. Add <code>_mwan3_ipset_needs_delete</code> which queries the live set via <code>nft list set</code> and returns true if the live flags differ from the desired spec.</p>
<p><code>_mwan3_render_one_ipset</code> now calls <code>_mwan3_ipset_needs_delete</code> before the <code>add set</code> statement. If flags differ the set is deleted first, clearing its elements but ensuring the recreated set has the correct type, timeout, counter, and size flags.</p>
<p>When a set is deleted and recreated its dnsmasq-populated domain entries are lost. Set <code>MWAN3_NEED_DNSMASQ_HUP</code> when this occurs and call <code>mwan3_dnsmasq_hup</code> after the reload batch in <code>reload_service</code> to repopulate those entries.</p>
<p>Move the ipset and dnsmasq fragment functions from <code>common.sh</code> to <code>mwan3.sh</code>. They are only called from <code>mwan3.sh</code> or <code>init.d/mwan3</code>, and the new helpers (<code>_mwan3_nft_time_to_sec</code>, <code>_mwan3_ipset_needs_delete</code>) naturally belong with them.</p>
<hr>
<h4 id="mwan3-remove-version-number-references-from-comments">mwan3: remove version number references from comments</h4>
<p>Comments referencing specific version numbers become misleading as the codebase evolves. Replace all such references with descriptions of the actual state or behaviour they document.</p>
<hr>
<h4 id="mwan3-suppress-stderr-on-unguarded-ip-ruleroute-operations">mwan3: suppress stderr on unguarded ip rule/route operations</h4>
<p>Four locations in <code>mwan3.sh</code> produced noise on stderr during package upgrades and edge-case teardown:</p>
<ul>
<li><code>mwan3_create_iface_rules</code>: ip rule add calls had no error suppression. <code>mwan3_delete_iface_rules</code> runs first, so a "File exists" error means the rule is already in the desired state - a valid outcome that should not be reported as an error.</li>
</ul>
<ul>
<li><code>mwan3_delete_iface_route</code>: ip route flush on a never-populated table produces "FIB table does not exist". This is a normal teardown scenario when an interface was never brought online.</li>
</ul>
<ul>
<li><code>mwan3_extra_tables_routes</code>: ip route list on a missing rt_table_lookup table produces the same error. Suppressed here without a warning since this is called per-interface per-connect; <code>mwan3_set_custom_set</code> provides the warning at a more appropriate point.</li>
</ul>
<ul>
<li><code>mwan3_set_custom_set</code>: ip route list calls restructured to capture output and check exit code separately, so a missing rt_table_lookup table is suppressed on stderr but logged via LOG warn. This preserves the error as a diagnosable signal without printing raw kernel errors to the console.</li>
</ul>
<hr>
<h3 id="1814-version-35">18.14 Version 3.5</h3>
<p><strong>Summary:</strong> Version 3.5 is a major architectural release that moves mwan3 out of <code>table inet fw4</code> and into its own <code>table inet mwan3</code>, eliminating the fw4 rebuild scaffold and the mwan3evtd debounce daemon entirely.</p>
<p>The reload path is replaced with a single atomic nft batch that commits the complete new ruleset while the old one is still serving traffic, with zero window of misrouted connections. User-declared nft sets are now configured directly in <code>/etc/config/mwan3</code> with inline, file, and dnsmasq-populated modes, and per-element packet and byte counters are optionally available. The APK install lifecycle is hardened to eliminate RTNETLINK errors on both fresh install and upgrade. LuCI gains a full IP Sets configuration tab and a new IP Sets status view with paginated element display, and the overview layout is redesigned with CSS grid cards. Additional bug fix to debounce <code>RTM_NEWROUTE</code> calls in <code>mwan3rtmon</code>.</p>
<hr>
<h4 id="mwan3-update-mwan3rtmon-to-debounce-rtmnewroute-calls">mwan3: update mwan3rtmon to debounce RTM_NEWROUTE calls</h4>
<p><code>RTM_NEWROUTE</code> events for connected routes were fast-pathed directly to <code>nft_exec</code>, bypassing the debounce timer that was only applied to deletes. On IPv6 systems with prefix delegation, the kernel appears to send repeated <code>RTM_NEWROUTE</code> updates for already-present connected routes, causing nft processes to be spawned multiple times per second and producing measurable CPU load.</p>
<p>Fix: route both <code>RTM_NEWROUTE</code> and <code>RTM_DELROUTE</code> for CIDR routes through the same 100ms debounce timer, calling <code>populate_connected_set()</code> once after the burst settles rather than once per event. Add a fingerprint (sorted, joined element list) to <code>opulate_connected_set()</code> so that calls where the connected set content has not changed skip the <code>nft_batch</code> call entirely. Also add ECMP deduplication (seen map) and link-local filtering to the element build loop in <code>populate_connected_set()</code>.</p>
<hr>
<h4 id="mwan3-add-conntrack-as-a-hard-dependency">mwan3: add conntrack as a hard dependency</h4>
<p>mwan3 relies on the conntrack userspace tool in several places: <code>flush_conntrack</code> is called when interfaces go down or policies change to force existing connections to be re-evaluated under the new routing state. Without conntrack installed these operations silently fail, leaving stale connections pinned to a dead or reconfigured WAN interface. Making the dependency explicit ensures conntrack is always present when mwan3 is installed.</p>
<hr>
<h4 id="mwan3-add-counters-support-nftsetelements-rpc-and-orphaned-set-cleanup">mwan3: add counters support, nftset_elements RPC, and orphaned set cleanup</h4>
<p>Add option counters (bool, default 0) to config ipset sections. When set, enables per-element packet and byte count tracking via nft set counter statement.</p>
<p>Change <code>maxelem</code> default from 65536 to 0 (unlimited), matching fw4 behaviour where sets have no size limit unless explicitly configured.</p>
<p>Add <code>mwan3_cleanup_orphaned_ipsets</code>, called during reload_service after <code>mwan3_render_config_ipsets</code>. Queries nft for user-defined sets not prefixed mwan3_, compares against configured set names, and deletes any orphans. Prevents stale sets accumulating when a set is removed via LuCI and the config is applied.</p>
<p>rpcd ucode: refactor <code>get_nftset_members</code> to use a shared <code>parse_elem_val</code> helper; add <code>get_nftset_elements</code> which returns elements with optional per-element counter data (packets/bytes) and supports pagination via a max parameter; add <code>count_nftset_elements</code> for lightweight element counting; enhance <code>nftset_info</code> to include flags, counters, and count fields; add <code>nftset_elements</code> RPC method with a 5000-element hard cap.</p>
<hr>
<h4 id="mwan3-atomic-non-destructive-reload-via-single-nft-batch">mwan3: atomic non-destructive reload via single nft batch</h4>
<p>Replaces the stop/start reload_service with an atomic single <code>nft -f</code> batch that rebuilds the entire ruleset while the old one serves traffic. The batch commits in one kernel transaction with zero window of misrouted traffic.</p>
<p>No conntrack flush in reload: existing connections keep their ct marks and current routing; new connections use new rules immediately.</p>
<p>Race condition immunity:</p>
<p>Reload: the entire rebuild is a single <code>nft -f</code> batch. The kernel commits the complete new ruleset atomically or rolls back to the old one. There is no intermediate state where prerouting exists but iface_in chains are absent. Adding or removing an interface requires a full restart because procd service instances for new trackers can only be registered during <code>start_service</code>. The tracker mismatch check at the end of reload_service detects the discrepancy and falls through to stop/start automatically.</p>
<p>Startup: the <code>wait $hotplug_pids</code> barrier in <code>start_service</code> ensures all background ifup jobs complete before <code>mwan3_set_general_nft</code> populates prerouting. Prerouting's vmap dispatch never references a chain that does not yet exist.</p>
<hr>
<h4 id="mwan3-update-mwan3-lb-test-for-standalone-table-inet-mwan3">mwan3: update mwan3-lb-test for standalone table inet mwan3</h4>
<p><code>mwan3-lb-test</code> creates a temporary test set and inserts rules into <code>mwan3_output</code>, <code>mwan3_rules</code>, and fw4's forward chain. Moving mwan3 to its own <code>table inet mwan3</code> requires two changes.</p>
<p>Rename TABLE from "inet fw4" to "inet mwan3" so that <code>mwan3_output</code>, <code>mwan3_rules</code>, and the test set creation all target the correct table.</p>
<p>Introduce <code>FORWARD_TABLE="inet fw4"</code> for the forward isolation rule. nft sets are table-scoped: the forward chain lives in <code>table inet fw4</code>, so the test set and the drop rule that references it must both exist in fw4. All forward chain operations (rule insertion, handle lookup, rule deletion, set creation/deletion, and stale-set cleanup) are updated to use <code>FORWARD_TABLE</code>.</p>
<hr>
<h4 id="mwan3-fix-apk-preinst-and-postinst-for-table-inet-mwan3">mwan3: fix APK preinst and postinst for table inet mwan3</h4>
<p>Fix the APK install lifecycle to ensure clean operation on both fresh install and upgrade from an installation that used table inet fw4.</p>
<p>Add a preinst script that stops mwan3 before APK replaces any files so that procd's inotify trigger does not auto-restart it during package installation. A safety-net stop at the start of postinst covers the case where preinst did not run or procd restarted the service between preinst and postinst.</p>
<p>Make the fw4 reload in postinst conditional on the <code>firewall.mwan3_reload</code> UCI section existing. On a fresh install that section is absent, so an unconditional reload triggers queued ifup hotplug events that cause a race with <code>mwan3_create_iface_rules</code>, producing <code>RTNETLINK "File exists"</code> errors. On upgrade the section exists, so the delete succeeds, and fw4 is reloaded exactly as before.</p>
<p>Add a second mwan3 stop immediately before the final mwan3 start in postinst. By this point the procd auto-start has completed and its ip rules are present. The second stop calls <code>stop_service</code> and removes them before <code>start_service</code> re-adds them, eliminating the remaining source of <code>RTNETLINK "File exists"</code> errors.</p>
<hr>
<h4 id="mwan3-remove-mwan3evtd-debounce-daemon">mwan3: remove mwan3evtd debounce daemon</h4>
<p><code>mwan3evtd</code> was introduced to debounce rapid-fire mwan3 rebuild events and deliver a single safe dnsmasq <code>SIGHUP</code> after each fw4 reload. Since mwan3 now lives in its own <code>table inet mwan3</code> which <code>fw4 reload</code> does not touch, there are no fw4-triggered rebuilds, no dnsmasq <code>SIGHUP</code>s from mwan3, and no event storms to debounce. <code>mwan3evtd</code> is redundant.</p>
<p>Remove the daemon, its init script, config file, helper binary, example files, and ACL. Remove the <code>ucode-mod-log</code> dependency (used only by <code>mwan3evtd</code>). Remove <code>/etc/config/mwan3evtd</code> from conffiles. Remove the <code>postinst enable/start</code> and <code>postrm stop/disable</code> calls.</p>
<p>The <code>postrm dnsmasq</code> cleanup (removing <code>mwan3-nftsets.conf</code> fragments and restarting dnsmasq) is retained since mwan3 still writes those fragments for domain-based config ipset population; removal of the package should clean them up so dnsmasq stops attempting to populate sets that no longer exist.</p>
<hr>
<h4 id="mwan3-fix-port-range-rendering-for-nftables">mwan3: fix port-range rendering for nftables</h4>
<p>UCI stores port ranges as <code>x:y</code> (e.g. <code>'47813:47814'</code>) but nftables requires <code>x-y</code> (e.g. <code>'47813-47814'</code>). The colon is map-element syntax in nft; a set element like <code>{ 47813:47814 }</code> is rejected with "mapping outside of map context", causing the entire mwan3_rules batch to fail atomically and leaving the chain empty.</p>
<p>Fix: add <code>s/:/-/g</code> to the sed transform in <code>mwan3_set_user_nft_rule</code> for both <code>src_port</code> and <code>dest_port</code>. Old configs with colon separators and new configs with dash separators both work correctly after this change.</p>
<hr>
<h4 id="mwan3-auto-flush-mwan3-marked-conntrack-on-service-reload">mwan3: auto-flush mwan3-marked conntrack on service reload</h4>
<p>Add <code>mwan3_flush_marked_conntrack()</code> in <code>mwan3.sh</code> and invoke it from a new <code>reload_service</code> override in <code>init.d/mwan3</code>. Flushes every conntrack entry whose mark has any <code>MMX_MASK</code> bit set, so UCI-driven reloads (including the <code>uci-commit-trigger</code> path via <code>procd_add_reload_trigger</code>) cause live flows to re-enter the classification chains and re-evaluate against the new rules instead of staying pinned to a previously saved ct mark.</p>
<p>Complements the existing <code>mwan3_flush_stale_conntrack</code>, which handles the distinct zero-mark case (flow slipped through unclassified during the fw4-rebuild window). The two cover orthogonal cleanup needs.</p>
<p>conntrack's <code>-D --mark VALUE/MASK</code> filter does exact-match on the masked bits; there is no "any bit set" predicate. The helper iterates the mwan3 id-space (default 6 bits =&gt; 63 ids) and issues one targeted -D per id. Bounded and fast.</p>
<p>Not called from <code>start_service</code>: <code>service mwan3 restart</code> should leave live TCP flows intact; only a config-driven reload reclassifies them. The <code>mwan3_init</code> at the top of <code>reload_service</code> ensures <code>MMX_MASK</code> is populated before the flush, since stop wipes the persisted status dir.</p>
<p>Fixes the stale-ct-mark class of issues where a rule change does not take effect on a live flow, leaving it pinned to the old interface even after the user adds a mwan3 rule that should reclassify it.</p>
<hr>
<h4 id="mwan3-add-table-inet-mwan3-nft-set-support">mwan3: add table inet mwan3 nft set support</h4>
<p>Add support for user-declared nft sets via a new config ipset section type in <code>/etc/config/mwan3</code> that matches fw4 syntax and replaces sets declared in /etc/config/firewall and which live within table inet fw4's namespace, which is not in scope in table inet mwan3.</p>
<p>Three population modes are supported: inline entries via list entry, file-based population via option loadfile, and dnsmasq-populated sets via list domain (mwan3 writes confdir fragments and signals dnsmasq only when fragment content changes).</p>
<p>Add <code>mwan3-migrate-ipset-v4.sh</code>, a one-shot idempotent migration helper that copies existing config ipset declarations from <code>/etc/config/firewall</code> to <code>/etc/config/mwan3</code> on upgrade from v3.x. Called from postinst. Makefile gains the corresponding install line and a postrm cleanup block that removes dnsmasq confdir fragments and reloads dnsmasq on package removal.</p>
<p>Update rpcd <code>nftset_info</code>: nftset membership lookup was filtering on <code>s.table == 'fw4'</code> but sets now live in <code>table inet mwan3</code>, so the filter is updated to <code>s.table == 'mwan3'</code>.</p>
<hr>
<h4 id="mwan3-move-to-standalone-table-inet-mwan3-remove-fw4-rebuild-scaffold">mwan3: move to standalone table inet mwan3, remove fw4 rebuild scaffold</h4>
<p>Move mwan3 from <code>table inet fw4</code> to its own <code>table inet mwan3</code>.</p>
<p>Rename every inet fw4 literal to <code>inet mwan3</code> across <code>common.sh</code>, <code>mwan3.sh</code>, <code>init.d/mwan3</code>, <code>hotplug.d/iface/25-mwan3</code>, <code>usr/sbin/mwan3</code>, <code>mwan3rtmon</code>, and <code>usr/share/rpcd/ucode/mwan3</code>. The Makefile postinst retains its <code>inet fw4</code> references: those are legacy cleanup of v3.x-era chains that remain correct.</p>
<p>Add <code>files/lib/mwan3/mwan3-skeleton.nft</code>: a standalone nftables ruleset that creates (or atomically re-creates) <code>table inet mwan3</code> with its base chains and sets using the delete+recreate idiom for idempotency. Wire the skeleton load into <code>init.d/mwan3 start_service</code> with an early return on failure so subsequent nft add calls cannot paper over a missing table.</p>
<p>Delete <code>mwan3-fw-include.sh</code> and <code>mwan3-fw-rebuild.sh</code>. <code>fw4 reload</code> no longer touches <code>table inet mwan3</code>, so there is nothing to detect or rebuild. The fw4-include UCI registration script <code>mwan3-firewall-include</code> is replaced with <code>mwan3-remove-firewall-include</code>, a one-shot <code>uci-defaults</code> script that removes the stale <code>firewall.mwan3_reload</code> section left by v3.x installs. Remove the <code>fw4 reload</code> detection block from <code>25-mwan3</code>.</p>
<p><code>stop_service</code> gains a final <code>nft delete table inet mwan3</code> so a clean stop leaves no nft residue.</p>
<p>Add postinst cleanup of legacy mwan3 chains, sets, and sticky maps from <code>table inet fw4</code> so upgrading from a v3.x install removes the old hooked chains that would otherwise remain after <code>fw4 reload</code> (fw4 uses <code>flush-table</code> not <code>delete-table</code>, so they survive indefinitely).</p>
<p>Simplify <code>mwan3_dnsmasq_hup</code> in <code>mwan3.sh</code> to a direct ubus call. The <code>mwan3evtd</code> dispatch path is no longer needed as mwan3 now lives in its own table and <code>fw4 reload</code> does not trigger rebuilds or <code>SIGHUP</code> storms. Call <code>mwan3_dnsmasq_hup</code> from <code>start_service</code> after all sets are created so dnsmasq clears its cache and re-populates the new nft sets.luci-app-mwan3: improve overview status layout</p>
<p>Switch the interface and policy card grids from flexbox to CSS grid with fixed 13em columns using repeat(auto-fill). Cards wrap gracefully as the browser is resized rather than forcing all items onto one row regardless of available width.</p>
<p>Replace the LuCI <code>%t</code> format with a custom <code>formatDuration()</code> that omits seconds, keeping the uptime display stable in width and removing the visual noise of a ticking seconds counter.</p>
<p>Add Interfaces and Policies section headings above each grid, matching the existing Rules heading, which also cleanly separates the two grids visually so their differing column counts do not appear misaligned.</p>
<p>Remove the indent from policy member entries and add white-space:nowrap to prevent wrapping within a member line on narrow columns.</p>
<hr>
<h4 id="luci-app-mwan3-add-ip-sets-status-tab">luci-app-mwan3: add IP Sets status tab</h4>
<p>New status view at Status &gt; MultiWAN Manager &gt; IP Sets (order 27, between Routing and Diagnostics).</p>
<ul>
<li>Page load calls <code>nftset_info</code> to get all user-defined mwan3 set metadata (type, counters flag, runtime element count) and loads mwan3 UCI config for each set's static parameters (entries, domains, loadfile, maxelem, timeout); element counts are shown in the panel header without requiring any user interaction</li>
</ul>
<ul>
<li>Each set is shown as a collapsible panel with an Expand/Collapse toggle; on first expand, configured domain names (from list domain) are shown immediately from UCI, then the runtime member table is loaded via the new nftset_elements RPC; subsequent collapse/expand cycles reuse the already-loaded DOM without re-fetching</li>
</ul>
<ul>
<li>Members are displayed in a consistent 3-column table (Address / Packets / Bytes) regardless of whether counters are enabled; Packets and Bytes cells are empty for sets without counters</li>
</ul>
<ul>
<li>Large sets: elements load up to 200 by default; Load more (1000) and Load all (5000) buttons appear when the result is truncated</li>
</ul>
<ul>
<li>ACL: add <code>nftset_info</code> and <code>nftset_elements</code> to both <code>luci-app-mwan3-status</code> and <code>luci-app-mwan3</code> ACL sections</li>
</ul>
<hr>
<h4 id="luci-app-mwan3-improve-ip-sets-network-configuration-tab">luci-app-mwan3: improve IP Sets network configuration tab</h4>
<ul>
<li>Block deletion of a set that is referenced by mwan3 rules; show a warning notification naming the referencing rules so the user knows what to fix first</li>
</ul>
<ul>
<li>Suppress premature validation red on the Name field when a new set modal opens with an empty name; the field is marked pristine until the user interacts with it or clicks Save, at which point the red border appears if the field is still empty</li>
</ul>
<ul>
<li>Change <code>maxelem</code> description and placeholder from 65536 to "unlimited" to match fw4 behaviour (sets have no size limit unless explicitly configured)</li>
</ul>
<hr>
<h4 id="luci-app-mwan3-add-ip-sets-configuration-tab">luci-app-mwan3: add IP Sets configuration tab</h4>
<p>Add a new IP Sets tab to the MultiWAN Manager network section (order 52, between Rule and Simulator). The tab provides a GridSection UI for managing config ipset sections in /etc/config/mwan3.</p>
<p>Fields: <code>name</code> (validated: safe chars, no <code>mwan3_</code> prefix, unique across sections), <code>family</code> (IPv4/IPv6), <code>entry</code> (DynamicList, ipaddr datatype, family cross-check validation), <code>domain</code> (DynamicList, for dnsmasq nftset population), <code>loadfile</code> (FileUpload to <code>/etc/luci-uploads</code>), <code>maxelem</code>, <code>timeout</code>, <code>counters</code>, enabled.</p>
<p>IP address entry fields use blur-only validation matching <code>rule.js</code> behaviour, implemented via <code>makeBlurOnlyList</code> which attaches a capture-phase keyup suppressor to existing inputs at render time and to dynamically added inputs via <code>MutationObserver</code>.</p>
<p>ACL updated to add <code>ubus file read/list</code> (read) and <code>file write/remove</code> (write) permissions required by <code>form.FileUpload</code>.</p>
<hr>
<h4 id="luci-app-mwan3-update-nftset-references-to-table-inet-mwan3">luci-app-mwan3: update nftset references to table inet mwan3</h4>
<p><code>nftset_dump</code> in luci-mwan3 now lists sets from <code>inet mwan3</code> rather than <code>inet fw4</code>; the <code>mwan3_</code> prefix filter correctly hides internal skeleton sets and surfaces only user-declared config ipset sections.</p>
<p>The ipset dropdown placeholder text in <code>rule.js</code> is updated to show the <code>inet#mwan3</code> nftset directive syntax and explain that fw4-side sets need a parallel config ipset declaration in <code>/etc/config/mwan3</code> to be usable in mwan3 rules.</p>
<hr>
<h4 id="luci-app-mwan3-update-port-range-hint-to-use-dash-separator">luci-app-mwan3: update port-range hint to use dash separator</h4>
<p>mwan3 now renders port ranges using <code>x-y</code> (nft native format); the colon separator <code>x:y</code> is still accepted in UCI for backwards compatibility but is no longer the recommended input format. Update the <code>src_port</code> and <code>dest_port</code> help text from <code>"1024:2048"</code> to <code>"1024-2048"</code> so new entries match nft syntax directly.</p>
<hr>
<h3 id="1815-version-341-unreleased">18.15 Version 3.4.1 (Unreleased)</h3>
<p><strong>Summary:</strong> Builds the per-interface <code>mwan3_iface_in_*</code> chains before <code>mwan3_set_general_nft()</code> activates <code>mwan3_prerouting</code> to avoid a race condition that leads to a wrong interface mark being assigned. Fixes bugs in the <code>nft list chains</code> syntax in <code>stop_service()</code> and a grep expression that was causing a too-broad match and resulting in traffic for interface <code>wan</code> bypassing mwan3 marking.</p>
<hr>
<h4 id="mwan3-fix-nft-list-chains-syntax-error-in-stopservice">mwan3: fix nft list chains syntax error in stop_service</h4>
<p><code>nft list chains</code> only accepts an optional family argument, not a table name. The two chain-enumeration loops in stop_service used <code>nft list chains inet fw4</code>, which is invalid syntax. The fw4 argument caused nft to exit with an error; <code>2&gt;/dev/null</code> suppressed it, producing no output. Both loops therefore silently iterated over nothing on every <code>service mwan3 stop</code> or <code>service mwan3 restart</code>.</p>
<p>Consequence: dynamic chains (<code>mwan3_iface_in_*</code>, <code>mwan3_policy_*</code>, <code>mwan3_or_meta_*</code>, <code>mwan3_or_ct_*</code>) were never flushed or deleted on stop. They persisted in <code>table inet fw4</code> as empty orphan objects. The final hardcoded skeleton-chain flush at the end of <code>stop_service</code> was unaffected and continued to work correctly, so functional impact was limited to residual chain objects after stop.</p>
<p>Fixed by changing both occurrences to <code>nft list chains inet</code>. The <code>grep "chain mwan3_"</code> filter is sufficient to scope results to mwan3's chains, which only exist in <code>table inet fw4</code>.</p>
<hr>
<h4 id="mwan3-fix-grep-substring-match-in-mwan3ifacesin-chain-wiring">mwan3: fix grep substring match in mwan3_ifaces_in chain wiring</h4>
<p><code>mwan3_create_iface_nft</code> used <code>grep -q "jump mwan3_iface_in_$1"</code> as an idempotency check before adding a jump from <code>mwan3_ifaces_in</code> to the per-interface chain. <code>grep -q</code> does substring matching, so for <code>$1=wan</code> the pattern also matches lines containing <code>mwan3_iface_in_wan2</code>, <code>mwan3_iface_in_wan6</code>, etc. If those interfaces had already added their jumps, wan's check returned a false positive and its jump was skipped, leaving wan absent from <code>mwan3_ifaces_in</code> and all wan traffic bypassing mwan3 marking entirely.</p>
<p>The same bug in <code>mwan3_delete_iface_nft</code> caused the handle lookup for wan's jump to return the handle of a different interface's rule.</p>
<p>The bug existed before this patch but was masked by the old sequential ordering, where wan (first in UCI order) always checked <code>mwan3_ifaces_in</code> before any other process had added jumps. The preceding commit changed to parallel execution, exposing the race.</p>
<p>Fixed with <code>grep -qw</code> (whole-word match) in both functions. Only affects configurations where one interface name is a prefix of another (e.g. wan/wan2, eth0/eth0b).</p>
<hr>
<h4 id="mwan3-build-ifacein-chains-before-activating-prerouting">mwan3: build iface_in chains before activating prerouting</h4>
<p>The per-interface <code>mwan3_iface_in_*</code> chains are now fully populated before <code>mwan3_set_general_nft()</code> activates <code>mwan3_prerouting</code>. Previously the ordering was reversed: prerouting became live first, then each <code>mwan3_ifup</code> background process added rules to <code>mwan3_ifaces_in</code> one by one. During that window, packets arriving on any interface whose chain was not yet wired got no ct mark. For DNAT traffic (<code>fib daddr type local</code> return with no mark save) the reply could hit numgen and acquire a random interface mark, which WireGuard PersistentKeepalive then locked in permanently by refreshing the conntrack entry every 25s.</p>
<p>Changed in all three rebuild code paths:</p>
<ul>
<li><code>init.d/mwan3 start_service</code>: moved <code>config_foreach mwan3_ifup</code> and <code>wait $hotplug_pids</code> before <code>mwan3_set_general_nft</code></li>
<li><code>25-mwan3</code> fw4-reload detection block: <code>config_foreach mwan3_rebuild_iface_nft</code> before <code>mwan3_set_general_nft</code></li>
<li><code>mwan3-fw-rebuild.sh</code>: same reorder as <code>25-mwan3</code></li>
</ul>
<p>Also removed the flush of <code>mwan3_postrouting</code> from <code>mwan3_set_general_nft</code>. That flush was a leftover from when <code>general_nft</code> ran before iface setup; with the new ordering it ran after <code>mwan3_create_iface_nft</code> had already written snat6 rules to postrouting, silently deleting them. <code>mwan3_delete_iface_nft</code> and the per-chain cleanup in <code>mwan3_create_iface_nft</code> already handle postrouting cleanup correctly.</p>
<hr>
<h3 id="1816-version-34">18.16 Version 3.4</h3>
<p><strong>Summary:</strong> Version 3.4 introduces mwan3evtd, a generalised ucode debounce daemon that coalesces rapid-fire events - such as simultaneous interface flaps triggering multiple fw4 reloads - into a single handler execution after the activity settles. This prevents the repeated dnsmasq SIGHUPs that previously caused cache thrash and, in tight-timing scenarios, dnsmasq crashes during concurrent startup.</p>
<hr>
<h4 id="mwan3-mwan3evtd-debounce-daemon">mwan3: mwan3evtd debounce daemon</h4>
<p>Add <code>mwan3evtd</code>, a generalised ucode debounce daemon that debounces rapid-fire events and executes a single safe handler for all identical events received during the debounce window.</p>
<p>Background: fw4 recreates the entire <code>inet fw4</code> nftables table on every reload, which flushes mwan3's nftsets. dnsmasq must be SIGHUPed after each rebuild to repopulate those sets. Without debouncing, simultaneous interface events (e.g. N-interface flap) would fire N independent mwan3 rebuilds in quick succession, each attempting to <code>SIGHUP</code> dnsmasq.</p>
<p><code>mwan3evtd</code> coalesces all pushes within a window into a single fire: <code>window_ms</code> (default 5 s) resets on each push, and the HUP is delivered only once after activity has settled or on <code>max_window_ms</code>.</p>
<p>The debounce daemon exists to avoid unnecessary cache flushes and to make dnsmasq HUPs safe, since in some cases, and with tight timing, a HUP on a starting instance was observed to crash dnsmasq, resulting in loss of connectivity for clients.</p>
<p><code>mwan3evtd</code> exposes a ubus object <code>mwan3evtd</code> with push/list/status/flush/reload/reset methods. Callers push named events via:</p>
<p><code>ubus call mwan3evtd push '{"event":"dnsmasq-hup"}'</code></p>
<p>or via the fast-path helper <code>/usr/sbin/mwan3evtd-push</code> (falls back to direct UCI config parsing when ubus is unavailable, e.g. early boot).</p>
<p>Each handler is configured in <code>/etc/config/mwan3evtd</code> with:</p>
<pre><code> window_ms: debounce window reset on each push
max_window_ms: hard cap, fires even under continuous pressure
command: shell command to run on fire
handler_timeout_ms: kill handler if it runs longer than this</code></pre>
<p>Default handlers: <code>dnsmasq-hup</code>, <code>dnsmasq-restart</code>, <code>rpcd-reload</code>, <code>rpcd-restart</code>, <code>firewall-reload</code>, <code>firewall-restart</code>, <code>unbound-restart</code>, <code>smartdns-restart</code>, <code>kresd-restart</code>, <code>named-restart</code>.</p>
<p>Shell injection in the handler fire path is prevented by passing the command through <code>EVTD_CMD</code> env var and using <code>eval "$EVTD_CMD"</code>, avoiding any brace-grouping that an adversarial <code>}</code> in command could escape.</p>
<p><code>mwan3evtd</code> is a generalised debounce daemon, capable of accepting any event type and any handler and can be used by other packages if desired.</p>
<hr>
<h3 id="1817-version-335">18.17 Version 3.3.5</h3>
<p><strong>Summary:</strong> Version 3.3.5 is a single-fix release that suppresses the per-deleted-entry output that <code>conntrack -D</code> writes to stdout, which was previously appearing on the console whenever mwan3 start or an fw4 reload triggered the zero-mark conntrack flush.</p>
<hr>
<h4 id="mwan3-suppress-conntrack--d-output-to-console">mwan3: suppress conntrack -D output to console</h4>
<p><code>conntrack -D</code> prints each deleted entry to stdout. The call in <code>mwan3_flush_stale_conntrack()</code> redirected only stderr, causing all deleted zero-mark conntrack entries to appear on the console whenever mwan3 start or fw4 reload triggered the flush.</p>
<p>Fix: redirect stdout to <code>/dev/null</code> alongside stderr.</p>
<hr>
<h3 id="1818-version-334">18.18 Version 3.3.4</h3>
<p><strong>Summary:</strong> Version 3.3.4 closes a class of misrouting bugs caused by the brief window between fw4 flushing <code>table inet fw4</code> and mwan3 completing its nft rebuild. Connections established during that window acquire <code>ct mark=0</code>; the new <code>mwan3_flush_stale_conntrack</code> helper removes all zero-mark conntrack entries after every rebuild and restart, preventing WireGuard persistent-keepalive and similar long-lived UDP from locking in a bad entry indefinitely. A double-rebuild race in <code>mwan3-fw-rebuild.sh</code> is also fixed by acquiring the procd lock before checking for empty chains.</p>
<hr>
<h4 id="mwan3-flush-zero-mark-conntrack-entries-after-fw4-rebuild-and-restart">mwan3: flush zero-mark conntrack entries after fw4 rebuild and restart</h4>
<p>During the brief window between fw4 wiping <code>table inet fw4</code> and the nft rebuild completing, new connections (DNAT, WireGuard) can be established with <code>ct mark=0</code>. The <code>iface_in</code> chains do not yet exist so incoming packets are not marked; the conntrack entry is created with <code>ct mark=0</code>.</p>
<p>For most protocols the bad entry expires within 120 seconds and self-heals on the next connection attempt. Long-lived UDP with persistent keepalives (WireGuard <code>persistent-keepalive</code>) refreshes the entry before expiry, keeping it alive indefinitely and causing persistent misrouting of DNAT replies.</p>
<p>Add <code>mwan3_flush_stale_conntrack()</code> to <code>mwan3.sh</code>. Flushes conntrack entries with no mwan3 mark (<code>0x0/MMX_MASK</code>) using the conntrack tool. Only zero-mark entries are removed; correctly-marked active connections are untouched. Even for users without the persistent-keepalive problem, the flush causes stale connections to re-establish immediately rather than waiting up to 120 seconds for natural conntrack expiry. If <code>conntrack-tools</code> is not installed, logs a notice suggesting installation.</p>
<p>Called from three sites:</p>
<ul>
<li><code>mwan3-fw-rebuild.sh</code>: covers fw4 reload triggered by any means</li>
<li><code>25-mwan3</code> fw4 rebuild detection block: covers ifup-triggered fw4 reload</li>
<li><code>start_service</code> in <code>init.d/mwan3</code>: covers <code>service mwan3 restart</code>, which runs <code>25-mwan3</code> with <code>MWAN3_STARTUP=init</code> bypassing both guarded blocks and does not invoke <code>26-mwan3-user</code> where a <code>mwan3.user</code> workaround would</li>
</ul>
<p>otherwise run</p>
<hr>
<h4 id="mwan3-fix-double-rebuild-race-in-mwan3-fw-rebuildsh">mwan3: fix double rebuild race in mwan3-fw-rebuild.sh</h4>
<p>mwan3-fw-rebuild.sh<code> checked for empty </code>mwan3_prerouting<code> before acquiring </code>procd_lock<code>. If </code>25-mwan3<code> was already holding the lock and rebuilding, the </code>fw-rebuild<code> script could pass the check, then queue behind </code>25-mwan3<code>, and proceed to rebuild and call </code>mwan3_dnsmasq_hup` a second time after the lock was released.</p>
<p>Fix by acquiring <code>procd_lock</code> first and re-checking under the lock, so only one of the two rebuild paths does the actual work.</p>
<hr>
<h3 id="1819-version-333">18.19 Version 3.3.3</h3>
<p><strong>Summary:</strong> Version 3.3.3 is a broad bug-fix release addressing several correctness issues: DNAT reply routing was broken by a misplaced <code>fib daddr type local return</code> rule that fired before DNAT translation, causing replies to exit via a randomly load-balanced interface; IPv6 ip rules were silently leaked on ifdown because <code>delete_iface_rules</code> queried the IPv4 rule table; <code>mwan3_dnsmasq_hup</code> never sent SIGHUP because <code>json_get_var</code> stores booleans as integers not strings; and the numgen counter was contaminated by inbound and reply traffic. Additional fixes cover a grep substring false-positive in iface chain wiring, unquoted regex variables, a dead function stub, a duplicate function, and missing <code>mwan3_postrouting</code> in the stop_service chain lists. A new <code>bypass_network</code> UCI option populates the dynamic bypass sets from config, and <code>mwan3-lb-test</code> gains fw4 reload detection.</p>
<hr>
<h4 id="mwan3-add-mwan3postrouting-to-stopservice-skeleton-chain-lists">mwan3: add mwan3_postrouting to stop_service skeleton chain lists</h4>
<p><code>mwan3_postrouting</code> is defined in the static <code>10-mwan3.nft</code> skeleton alongside the other named skeleton chains, but was absent from both the deletion exclusion list and the safety-flush list in <code>stop_service()</code>. This caused it to be deleted on stop rather than preserved, relying on <code>start_service</code>/<code>mwan3_ensure_nft_framework</code> to recreate it. Add it to both lists for consistency with the other skeleton chains.</p>
<hr>
<h4 id="mwan3-remove-duplicate-mwan3countonebits-from-mwan3sh">mwan3: remove duplicate mwan3_count_one_bits from mwan3.sh</h4>
<p>An identical copy of <code>mwan3_count_one_bits()</code> existed in both <code>mwan3.sh</code> and <code>common.sh</code>. Since <code>mwan3.sh</code> sources <code>common.sh</code>, the <code>mwan3.sh</code> copy shadowed the canonical definition without any functional difference. Remove the duplicate; all call sites continue to use the <code>common.sh</code> definition.</p>
<hr>
<h4 id="mwan3-remove-dead-mwan3setstickynft-function">mwan3: remove dead mwan3_set_sticky_nft function</h4>
<p><code>mwan3_set_sticky_nft()</code> was an incomplete stub from an earlier sticky routing design that was superseded by the current per-member ip-only set + OR-immediate vmap-dispatch implementation in <code>mwan3_set_user_nft_rule()</code>. The stub had a no-op loop body and was never called. Remove it.</p>
<p><code>mwan3_get_policy_members_for_family()</code> is retained - it is called by the active sticky implementation at line 1131.</p>
<hr>
<h4 id="mwan3-quote-cmdline-in-mwan3getmwan3trackstatus">mwan3: quote $cmdline in mwan3_get_mwan3track_status</h4>
<p><code>$cmdline</code> was unquoted in the <code>[ $cmdline != ... ]</code> test. If readfile returns nothing (narrow race where the tracked process exits between the PID file read and the <code>/proc/$pid/cmdline</code> read), the empty expansion produces a malformed two-argument test expression. In busybox ash this happens to evaluate correctly (falls through to <code>export -n "$1=down"</code>), but the unquoted form is fragile. Quote it.</p>
<hr>
<h4 id="mwan3-quote-ipv4regex-in-mwan3setusernftrule">mwan3: quote $IPv4_REGEX in mwan3_set_user_nft_rule</h4>
<p><code>$IPv4_REGEX</code> was unquoted in the <code>grep -qE</code> call on line 951 while the adjacent <code>$IPv6_REGEX</code> on line 950 was correctly quoted. The IPv4 regex contains characters (<code>?</code>, <code>[</code>, <code>]</code>) that the shell interprets as glob patterns before passing to grep if unquoted. Quote it for consistency and correctness.</p>
<hr>
<h4 id="mwan3-fix-ipv6-ip-rule-leak-in-mwan3deleteifacerules">mwan3: fix IPv6 ip rule leak in mwan3_delete_iface_rules</h4>
<p><code>mwan3_delete_iface_rules()</code> sets <code>IP="$IP6"</code> for IPv6 interfaces but used bare <code>ip rule list</code> (which defaults to IPv4) to find rule priorities to delete. No IPv6 rules were found, so the for loop never executed and all three ip rules (at priorities <code>IIF_BASE+id</code>, <code>FWMARK_BASE+id</code>, and <code>3000+id</code>) were leaked on every IPv6 interface ifdown.</p>
<p>On the next ifup, <code>mwan3_create_iface_rules</code> calls <code>mwan3_delete_iface_rules</code> before adding new rules, but the delete fails for the same reason, so stale rules accumulate on each ifdown/ifup cycle.</p>
<p>Fix: change <code>ip rule list</code> to <code>$IP rule list</code> so the correct address family is used.</p>
<hr>
<h4 id="mwan3-fix-dnsmasqhup-running-check---jsongetvar-returns-1-not-true-for-boolean-true">mwan3: fix dnsmasq_hup running check - json_get_var returns 1 not "true" for boolean true</h4>
<p><code>json_get_var</code> stores JSON boolean values as integers (<code>1</code> for true, <code>0</code> for false), not as strings. The previous check <code>[ "$running" = "true" ]</code> never matched, so <code>mwan3_dnsmasq_hup</code> never actually sent <code>SIGHUP</code> to dnsmasq after fw4 reloads. DNS cache was therefore not cleared, leaving stale nftset entries in place.</p>
<hr>
<h4 id="mwan3-fix-dnat-reply-routing-broken-by-misplaced-fib-local-return-rule">mwan3: fix DNAT reply routing broken by misplaced fib-local return rule</h4>
<p>The v3.3.1 numgen contamination fix added an unguarded <code>fib daddr type local return</code> as the first substantive rule in <code>mwan3_prerouting</code>. Because <code>mwan3_prerouting</code> runs at priority <code>mangle+1</code> (-149), before the <code>nat/prerouting</code> DNAT hook (-100), the rule fires on the original packet of a DNAT connection while its destination is still the router's own WAN IP. The packet is returned immediately with no ct mark saved. When the DNAT reply arrives from the internal host on the LAN interface, the ct mark restore finds zero, the packet falls to the policy chain, numgen assigns a random WAN mark, and the reply exits via whichever interface numgen picks rather than the one the original packet arrived on.</p>
<p>Fix: remove the unguarded <code>fib-local</code> return from its position before the <code>ifaces_in</code> dispatch and reinsert it after, guarded by <code>meta mark &amp; MMX_MASK == 0</code>. Traffic arriving on a mwan3 WAN interface is already marked by the <code>iface_in</code> catchall before reaching this rule, so the guard makes it a no-op for WAN traffic while still blocking non-WAN (LAN, loopback) local-destined traffic from reaching the policy chain. DNAT original packets are now marked by the <code>iface_in</code> catchall, ct mark is saved correctly, and DNAT replies restore it and route back via the correct interface.</p>
<p>Also remove <code>ct direction reply return</code> from <code>mwan3_output</code>. That rule was added to compensate for ct mark being zero (a consequence of the misplaced <code>fib-local</code> return). With the <code>fib-local</code> return correctly placed and guarded, ct mark is non-zero for inbound WAN connections, the ct mark restore in <code>mwan3_output</code> works correctly, the policy chain guard (<code>meta mark != 0</code>) prevents numgen from firing, and router-level service replies route back via the correct WAN table rather than the main routing table.</p>
<hr>
<h4 id="mwan3-add-bypassnetwork-uci-option-to-populate-dynamic-sets-from-globals-config">mwan3: add bypass_network UCI option to populate dynamic sets from globals config</h4>
<p>Add <code>mwan3_set_dynamic_network()</code> callback and update <code>mwan3_set_dynamic_sets()</code> to read the <code>bypass_network</code> list from UCI globals config and populate <code>mwan3_dynamic_v4</code>/<code>v6</code> from it at startup, rather than just flushing the sets empty. Each entry is classified as IPv4 (contains <code>.</code>) or IPv6 (contains <code>:</code>) and added to the appropriate set via the existing nft batch.</p>
<p>Add <code>mwan3_set_dynamic_sets</code> to the fw4 reload rebuild sequence in both <code>25-mwan3</code> and <code>mwan3-fw-rebuild.sh</code> so that <code>bypass_network</code> entries are restored after fw4 reload alongside the connected and custom sets.</p>
<p>The <code>mwan3_dynamic_v4</code>/<code>v6</code> sets and <code>mwan3_dynamic</code> chain already existed and were already referenced in <code>mwan3_prerouting</code>, <code>mwan3_output</code>, and <code>mwan3_iface_in_*</code> chains. This change makes the feature accessible via UCI rather than requiring direct nft element injection.</p>
<hr>
<h4 id="mwan3-add-fw4-reload-detection-and-timestamp-to-mwan3-lb-test">mwan3: add fw4 reload detection and timestamp to mwan3-lb-test</h4>
<p>When fw4 reload occurs during the test wait period, the <code>25-mwan3</code> hotplug rebuilds the policy chain without the injected counter rules. Previously this produced <code>TOTAL=unknown</code> and silent per-member zeros with a generic <code>FAIL</code>.</p>
<p>Now: detect missing counter rules explicitly and report "fw4 reload likely occurred during the test". Also add a human-readable completion timestamp to the summary output.</p>
<hr>
<h4 id="luci-app-mwan3-add-bypass-networks-field-to-globals-page-and-improve-rttablelookup-ui">luci-app-mwan3: add Bypass networks field to globals page and improve rt_table_lookup UI</h4>
<p>Add a Bypass networks <code>DynamicList</code> field (UCI option <code>bypass_network</code>) to the globals page. Accepts IPv4 and IPv6 CIDRs directly; traffic to these networks bypasses mwan3 policy routing and uses the default route. Validation uses the <code>cidr</code> datatype with blur-only firing (suppress keyup) via a capture-phase event delegation listener on the container node, covering dynamically added items without a MutationObserver.</p>
<p>Improve the <code>rt_table_lookup</code> field: rename label from "Routing table lookup" to "Routing table bypass" with a description that explains the bypass effect and that both table numbers and names are accepted. Remove the <code>uinteger</code> datatype constraint which incorrectly rejected table names, since the backend (<code>ip route list table</code>) accepts both.</p>
<hr>
<h3 id="1820-version-332">18.20 Version 3.3.2</h3>
<p><strong>Summary:</strong> Version 3.3.2 fixes a spurious tracked-IP entry in ubus status output caused by a naming collision between mwan3track's temporary output file and the <code>TRACK_*</code> glob used by rpcd. The <code>mwan3-lb-test</code> tool gains mandatory client isolation, a Windows test command, and a stale-artifact cleanup subcommand. LuCI receives cross-field family consistency validation in the rule editor, a fix for false "Present (unexpected)" health badges during interface bring-up, source nftset display in the overview rules column, and source nftset support in the traffic path simulator.</p>
<hr>
<h4 id="mwan3-rename-trackoutput-temp-file-to-pingoutput">mwan3: rename TRACK_OUTPUT temp file to PING_OUTPUT</h4>
<p>The rpcd status module globs <code>TRACK_*</code> files in each interface's status directory to discover tracked IPs, using the suffix as the IP address. The temporary output file used by <code>mwan3track</code> for ping/httping/nping/nslookup output was named <code>TRACK_OUTPUT</code>, causing the rpcd glob to yield <code>OUTPUT</code> as a spurious tracked IP address in <code>ubus call mwan3 status</code>.</p>
<p>Rename the variable and its backing file to <code>PING_OUTPUT</code> to avoid the collision with the <code>TRACK_*</code> namespace.</p>
<hr>
<h4 id="mwan3-enhance-mwan3-lb-test-with-client-isolation-and-windows-test-command">mwan3: enhance mwan3-lb-test with client isolation and Windows test command</h4>
<p>Add mandatory <code>-c &lt;client_ip&gt;</code> parameter: inserts a forward chain drop rule blocking pings to the test destination set from all LAN clients except the nominated test client, and an <code>mwan3_output</code> return rule bypassing mwan3 marking for router processes pinging the same IPs. Both rules are scoped to the test set and removed by <code>cleanup()</code>.</p>
<p>Add <code>cleanup</code> subcommand to remove stale rules and sets left by a <code>SIGKILL</code>-terminated run.</p>
<p>Add Windows <code>cmd.exe</code> test command output alongside the existing Linux shell loop. Windows ping uses a fixed ICMP identifier (<code>id=1</code>), causing conntrack entry reuse on repeated pings to the same destination. The generated Windows command uses a longer inter-ping delay (<code>30/TRACK_COUNT + 3</code> seconds) so that the full cycle through all test IPs exceeds the 30s ICMP conntrack timeout, ensuring each revisit creates a new conntrack entry.</p>
<hr>
<h4 id="luci-app-mwan3-fix-missing-cross-field-family-consistency-checks-in-rule-editor">luci-app-mwan3: fix missing cross-field family consistency checks in rule editor</h4>
<p><code>nftset_validate()</code> only checked set type against the explicit family field and returned true immediately when family was unset. This allowed invalid combinations such as an IPv6 source NFT set paired with an IPv4 destination address to pass validation silently.</p>
<p>Add cross-field checks:</p>
<ul>
<li><code>ipset_src</code> type vs <code>dest_ip</code> address family</li>
<li><code>ipset</code> type vs <code>src_ip</code> address family</li>
<li><code>ipset_src</code> type vs <code>ipset</code> type (mixed families on the same rule)</li>
</ul>
<p>Add <code>ip_family()</code> helper to derive <code>ipv4</code>/<code>ipv6</code> from an IP address string.</p>
<hr>
<h4 id="luci-app-mwan3-fix-routing-health-showing-present-unexpected-during-interface-bring-up">luci-app-mwan3: fix routing health showing Present (unexpected) during interface bring-up</h4>
<p>During interface bring-up, mwan3 adds ip rules on the connected hotplug event before mwan3track has confirmed <code>STATUS=online</code>. The routing health page was calling <code>renderStatusBadge</code> with <code>expectedPresent=false</code> (derived from <code>online=false</code>), causing rules to be labelled Present (unexpected) even though their presence is normal and correct in this state.</p>
<p>Fix by introducing a tri-state <code>expectedPresent</code> parameter:</p>
<p><code>true</code> - expected present: green Present / red Missing</p>
<p><code>false</code> - expected absent: orange Present (unexpected) / muted Absent</p>
<p><code>null</code> - no expectation: muted Present / muted Absent (neutral)</p>
<p>Pass <code>true</code> when online, <code>null</code> otherwise. The card border colour already conveys health for non-online interfaces; the individual badges no longer need to second-guess presence during transitional states.</p>
<hr>
<h4 id="luci-app-mwan3-add-src-ipset-to-overview-rules-match-column">luci-app-mwan3: add src ipset to overview rules match column</h4>
<p>Display <code>ipset_src</code> (source NFT set) in the Match column of the rules table on the Overview tab. Rename the existing <code>ipset:</code> label to <code>dst ipset:</code> to distinguish it from the new <code>src ipset:</code> label.</p>
<hr>
<h4 id="luci-app-mwan3-add-ipsetsrc-support-to-traffic-path-simulator">luci-app-mwan3: add ipset_src support to traffic path simulator</h4>
<p>Extend the rule matching simulation to handle the source NFT set option (<code>ipset_src</code>) added alongside the existing destination NFT set (<code>ipset</code>). Updates <code>ruleMatches()</code> to check <code>src_ip</code> membership in the source set, <code>matchSummary()</code> to display it, and the set fetch loop to collect <code>ipset_src</code> names alongside <code>ipset</code> names.</p>
<hr>
<h4 id="luci-app-mwan3-improve-rule-modal-family-consistency-and-nftset-support">luci-app-mwan3: improve rule modal family consistency and nftset support</h4>
<p>Family validation: <code>src_ip</code> and <code>dest_ip</code> now validate against the selected address family on blur, showing a clear error if an IPv4 address is entered with family set to IPv6 or vice versa.</p>
<p>nftset improvements: the nftset dropdown is now populated via the new <code>mwan3.nftset_info</code> rpcd method instead of <code>luci-mwan3 nftset dump</code>. Sets are annotated with their address family (<code>(IPv4)</code>/<code>(IPv6)</code>) in the dropdown label. Family consistency is validated on save. Combining a destination nftset with <code>dest_ip</code> or a source nftset with <code>src_ip</code> is flagged as an error since both match the same dimension.</p>
<p>Source nftset: new <code>ipset_src</code> field allows matching source addresses against an nftset, complementing the existing destination nftset. Both src and dst nftsets can be set on the same rule and are ANDed together.</p>
<p>Grid display: Source and Destination columns now show the nftset name when no IP address is configured, preventing the rule from appearing as a wildcard match in the listing.</p>
<hr>
<h3 id="1821-version-331">18.21 Version 3.3.1</h3>
<p><strong>Summary:</strong> Version 3.3.1 adds source nftset matching (<code>ipset_src</code>) as a complement to the existing destination nftset, fixes three distinct numgen counter contamination bugs that caused load-balancing distributions to skew under inbound or reply traffic, sweeps orphaned policy chains that accumulate when policies are removed from UCI without an fw4 reload, and corrects IPv6 ip rule detection in the routing health check. The release also adds <code>nftset_info</code> as an rpcd ubus method and introduces the <code>mwan3-lb-test</code> CLI tool for verifying load-balancing weight distributions against configured policy members.</p>
<hr>
<h4 id="mwan3-add-ipsetsrc-option-for-source-nftset-matching-in-rules">mwan3: add ipset_src option for source nftset matching in rules</h4>
<p>User rules previously supported only a destination nftset match via the <code>ipset</code> UCI option (<code>ip daddr @set</code>). Add <code>ipset_src</code> as an independent source nftset match (<code>ip saddr @set</code>), allowing both src and dst nftsets to be specified on the same rule and ANDed together.</p>
<p>The existing <code>ipset</code> option is unchanged for backward compatibility. <code>ipset_src</code> follows the same pre-creation and family handling logic as the destination set. The <code>nfproto</code> guard condition is updated to include <code>ipset_src</code> as an implicit family qualifier.</p>
<hr>
<h4 id="mwan3-add-nftsetinfo-rpcd-ubus-method">mwan3: add nftset_info rpcd ubus method</h4>
<p>Returns the name and address-family type of all non-mwan3 nftables sets in <code>table inet fw4</code>. Used by <code>luci-app-mwan3</code> to annotate the nftset dropdown with family information and validate that a selected set is consistent with the rule's configured address family.</p>
<hr>
<h4 id="mwan3-translate-icmp-to-ipv6-icmp-for-ipv6-family-rules">mwan3: translate icmp to ipv6-icmp for IPv6 family rules</h4>
<p>In nftables inet tables, <code>meta l4proto icmp</code> matches protocol 1 (ICMPv4) only. A UCI rule with <code>proto=icmp</code> and <code>family=ipv6</code> previously generated <code>meta nfproto ipv6 meta l4proto icmp</code>, a contradiction that silently matched nothing.</p>
<p>Translate <code>proto icmp</code> to <code>ipv6-icmp</code> (proto 58) when <code>family</code> is <code>ipv6</code> in <code>mwan3_set_user_nft_rule()</code> so the generated rule correctly matches IPv6 ICMP traffic. All other proto/family combinations are unaffected.</p>
<hr>
<h4 id="mwan3-add-mwan3-lb-test-load-balancing-distribution-verifier">mwan3: add mwan3-lb-test load balancing distribution verifier</h4>
<p>CLI tool that verifies a load-balancing policy distributes traffic according to configured member weights. Inserts temporary nft counter rules into the policy chain, generates a test ping command, waits for completion, then reports per-member actual vs expected hit counts with PASS/FAIL verdict.</p>
<p>Key design points:</p>
<ul>
<li>Computes N from member weights (LCM/GCD) so expected counts per member are always whole numbers</li>
<li>Temporary nft set + ICMP-only rule inserted into <code>mwan3_rules</code> ensures only test pings hit the policy chain; DNS and other traffic to the same IPs is not matched</li>
<li>Default destination pool excludes any IPs already configured as mwan3 tracking IPs (<code>mwan3track</code> pings those via <code>mwan3_output</code>, which would contaminate the counter)</li>
<li><code>-6</code> flag selects IPv6 mode (<code>ip6 daddr</code> + <code>ping6</code> + IPv6 well-known IPs)</li>
<li>Optional IP override arguments for sites with non-standard reachability</li>
<li>Cleanup on exit, <code>SIGINT</code>, <code>SIGTERM</code> and <code>SIGPIPE</code>; startup sweep removes stale artifacts from aborted previous runs</li>
<li>Installed as <code>/usr/sbin/mwan3-lb-test</code> alongside <code>mwan3track</code> and <code>mwan3rtmon</code></li>
</ul>
<hr>
<h4 id="mwan3-fix-numgen-counter-contamination-from-inbound-and-reply-traffic">mwan3: fix numgen counter contamination from inbound and reply traffic</h4>
<p>Three bugs caused the numgen load-balancing counter in policy chains to be incremented by traffic that should never reach a policy chain. The effect is non-strict alternation and a skewed distribution that does not match the configured member weights.</p>
<p>Bug 1: inbound internet traffic (port scanners, bots) destined for the router's own WAN IP traversed <code>mwan3_prerouting</code> with <code>mark=0</code>, found no matching user rule, fell to the default policy, and fired numgen. Fix: add <code>fib daddr type local return</code> in <code>mwan3_prerouting</code> after the ICMPv6 ND bypass.</p>
<p>Bug 2: the router's replies to inbound connections (ICMP echo replies, TCP responses) passed through <code>mwan3_output</code> with <code>ct mark=0</code> -- because bug 1's fix caused the inbound packet to bypass prerouting without saving a ct mark -- and fell through to the policy chain firing numgen. On a public IPv6 address receiving frequent external pings this generates ~10-20 spurious hits per second. Fix: add <code>ct direction reply return</code> as the first rule in <code>mwan3_output</code>.</p>
<p>Bug 3: a UCI rule with <code>family ipv4</code> or <code>ipv6</code> but no <code>src_ip</code>, <code>dest_ip</code>, or <code>ipset</code> generates a bare <code>meta mark ... jump policy</code> rule with no IP version restriction. Rules with address-based criteria get an implicit family qualifier from the <code>ip</code>/<code>ip6</code> keyword; rules without any address criteria do not. Combined with bug 2, the router's IPv6 ICMP replies fell through all <code>ip saddr</code>/<code>ip daddr</code> rules and hit the bare default rule, firing numgen at high rate. Fix: in <code>mwan3_set_user_nft_rule()</code>, if none of <code>src_ip</code>/<code>dest_ip</code>/<code>ipset_name</code> are set and <code>family</code> is explicitly <code>ipv4</code> or <code>ipv6</code>, prepend <code>meta nfproto ipv4</code>/<code>ipv6</code> to <code>nft_match</code>.</p>
<hr>
<h4 id="mwan3-sweep-orphaned-policy-chains-on-startup">mwan3: sweep orphaned policy chains on startup</h4>
<p>When a policy is removed from UCI config, its <code>mwan3_policy_*</code> chain persists in <code>table inet fw4</code> until the next fw4 reload flushes the table. mwan3 restart only manages chains it knows about from current config, leaving orphans indefinitely.</p>
<p><code>mwan3_set_policies_nft()</code> now enumerates all <code>mwan3_policy_*</code> chains in the live ruleset and deletes any that have no corresponding UCI policy config before rebuilding the policy chains from config.</p>
<hr>
<h4 id="mwan3-fix-routinghealth-ip-rule-detection-for-ipv6-interfaces">mwan3: fix routing_health ip rule detection for IPv6 interfaces</h4>
<p><code>get_ip_rules()</code> called <code>ip -j rule list</code> which is IPv4-only. mwan3 adds ip rules for IPv6 interfaces via <code>ip -6 rule add</code>, making them invisible to the IPv4 rule list. <code>routing_health()</code> therefore reported iif and fwmark rules as missing for any IPv6 mwan3 interface, showing a false red card even when mwan3 was functioning correctly.</p>
<p>Fix by querying both <code>-4</code> and <code>-6</code> rule tables and merging the results, matching the same pattern already used by <code>get_table_routes()</code>. The stale rule detection also benefits as it now sees IPv6 stale rules too.</p>
<hr>
<h3 id="1822-version-33">18.22 Version 3.3</h3>
<p><strong>Summary:</strong> Version 3.3 adds three major LuCI diagnostic tools - a traffic path Simulator, a static Configuration analyser, and a live Routing health view - backed by two new rpcd ubus methods (<code>nftset_members</code> and <code>routing_health</code>). The configuration analyser detects undefined references, orphaned sections, and rule shadowing including correct IPv6 CIDR containment checks. The routing health view colour-codes per-interface ip rule and routing table state against live kernel state. The <code>apk info</code> vs <code>apk list -I</code> version display bug is also fixed.</p>
<hr>
<h4 id="mwan3-add-nftsetmembers-and-routinghealth-rpcd-ubus-methods">mwan3: add nftset_members and routing_health rpcd ubus methods</h4>
<p><code>nftset_members { set: "&lt;name&gt;" }</code> returns the current members of a named nft set in <code>table inet fw4</code>. The set name is validated against <code>[a-zA-Z0-9_-]+</code> before being passed to nft. Used by the <code>luci-app-mwan3</code> Simulator tab for ipset rule matching and connected-network bypass detection.</p>
<p><code>routing_health {}</code> compares the UCI configuration against live kernel state. For each mwan3 interface (by 1-based UCI order index N) it checks for ip rules at priorities <code>1000+N</code> (iif) and <code>2000+N</code> (fwmark), checks routing table N for a default route, and reads the mwan3track <code>STATUS</code> file. Reports stale ip rules in mwan3's priority range that have no matching UCI interface. Built-in blackhole (<code>FWMARK_BASE + MAX_IFACES - 2 = 2061</code>) and unreachable (<code>FWMARK_BASE + MAX_IFACES - 1 = 2062</code>) policy rules are explicitly excluded from stale detection.</p>
<p>Both methods are declared in the <code>luci-app-mwan3</code> ACL files.</p>
<hr>
<h4 id="mwan3-fix-software-version-display-in-troubleshooting-output">mwan3: fix software version display in troubleshooting output</h4>
<p><code>apk info mwan3</code> returns the version from the repository index, not the installed version. When the installed package version differs from the repo version, the Software-Version section of <code>mwan3 internal</code> output shows the wrong version.</p>
<p>Use <code>apk list -I</code> to query installed packages only, ensuring the correct installed version is displayed.</p>
<hr>
<h4 id="luci-app-mwan3-extend-rule-shadowing-check-to-cover-ipv6-cidrs">luci-app-mwan3: extend rule-shadowing check to cover IPv6 CIDRs</h4>
<p>The Configuration tab's rule-shadowing analysis previously skipped all IPv6 addresses, treating every IPv6 pair as 'may shadow'. Add proper IPv6 CIDR containment using BigInt arithmetic (<code>expandIPv6</code>, <code>ipv6ToBigInt</code>, <code>ipv6CidrContains</code>) so shadowed IPv6 rules are correctly identified. Refactor the IPv4 path into <code>ipv4CidrContains</code> to match the new structure.</p>
<hr>
<h4 id="luci-app-mwan3-add-simulator-configuration-and-routing-diagnostic-tabs">luci-app-mwan3: add Simulator, Configuration, and Routing diagnostic tabs</h4>
<p>Three new tabs backed by mwan3 internals:</p>
<p>Simulator (Network &gt; MultiWAN Manager &gt; Simulator): simulates which mwan3 rule matches a described packet and shows live policy state. Supports IPv4 and IPv6 CIDR matching, port ranges, nft set membership via <code>nftset_members</code> ubus method, and connected-network bypass detection via <code>mwan3_connected_v4</code>/<code>v6</code> sets. Enter key in any field triggers simulation.</p>
<p>Configuration (Network &gt; MultiWAN Manager &gt; Configuration): static analysis of the mwan3 UCI configuration. Detects undefined references, orphaned sections, policies with no members, all-same-interface policies, and rule shadowing via IPv4 and IPv6 CIDR containment.</p>
<p>Routing (Status &gt; MultiWAN Manager &gt; Routing): live comparison of ip rules and routing tables against UCI configuration via <code>routing_health</code> ubus method. Per-interface cards show iif rule, fwmark rule, and routing table state with colour-coded health badges. Includes a field reference explaining the ip rule priority scheme for non-expert users. Stale rule detection excludes built-in blackhole/unreachable policy priorities (<code>2061</code>/<code>2062</code>).</p>
<p>Also adds <code>nftset_members</code> and <code>routing_health</code> methods to the rpcd module with appropriate ACL entries in both <code>luci-app-mwan3</code> and <code>luci-app-mwan3-status</code>.</p>
<hr>
<h3 id="1823-version-323">18.23 Version 3.2.3</h3>
<p><strong>Summary:</strong> Version 3.2.3 improves tracking status visibility by adding per-IP latency and packet-loss detail to <code>mwan3 status</code> output and fixing the <code>check_quality</code> display to derive its state from mwan3track's runtime files rather than UCI, so changes to UCI without a restart no longer cause the status page to disagree with what is actually running. Stale gateway <code>TRACK_*</code>/<code>LATENCY_*</code>/<code>LOSS_*</code> files from previous PPPoE sessions are cleaned up on each probe list rebuild. The <code>luci-app-mwan3</code> PKG_VERSION scheme is fixed to prevent <code>apk upgrade</code> from reverting to the official package, and the GitHub Actions APK rename step is corrected to avoid i18n sub-packages overwriting the main package.</p>
<hr>
<h4 id="mwan3-add-per-ip-tracking-detail-to-interfaces-status-output">mwan3: add per-IP tracking detail to interfaces status output</h4>
<p><code>mwan3_report_iface_status</code> now prints each tracking IP below the interface status line, showing its status and (when <code>check_quality=1</code>) latency and packet loss:</p>
<pre><code> interface wan is online and tracking is active
track 8.8.4.4: up (12ms, 0% loss)
track 8.8.8.8: down (-, 100% loss)
track 192.168.1.1: ignored</code></pre>
<p><code>check_quality</code> is derived from <code>LATENCY_*</code> file presence (not UCI) to reflect actual mwan3track runtime state. The kernel status value "skipped" is displayed as "ignored" for consistency with the LuCI status page.</p>
<hr>
<h4 id="mwan3-fix-checkquality-display-inconsistency-when-uci-is-changed-without-restart">mwan3: fix check_quality display inconsistency when UCI is changed without restart</h4>
<p>When <code>check_quality</code> is changed in UCI without restarting mwan3track, the rpcd status reported the new UCI value immediately while mwan3track continued running with the old setting, causing the status page to show "Not enabled" for latency/loss even though measurements were still being taken (or vice versa).</p>
<p>Fix in two parts:</p>
<p>rpcd: derive <code>check_quality</code> from <code>LATENCY_*</code> file content rather than UCI. mwan3track only writes <code>LATENCY_*</code> files when <code>check_quality=1</code>, so file presence with content is the authoritative indicator of actual runtime state. A fresh read of UCI is not needed on the rpcd side.</p>
<p>mwan3track: in <code>mwan3_load_track_ips()</code>, re-read <code>check_quality</code> from the current UCI config and remove any stale <code>LATENCY_*</code>/<code>LOSS_*</code> files when it is 0. This runs at startup and on every ifup event, so stale files from a previous check_quality=1<code> run are cleaned up as soon as the interface next comes up with </code>check_quality=0`, ensuring the rpcd file-based detection also returns the correct result after restart.</p>
<hr>
<h4 id="mwan3-remove-stale-gateway-tracklatencyloss-files-on-probe-list-rebuild">mwan3: remove stale gateway TRACK_<em>/LATENCY_</em>/LOSS_* files on probe list rebuild</h4>
<p>When <code>track_gateway=1</code> and the gateway IP changes (e.g. PPPoE reconnect), <code>mwan3_load_track_ips</code> builds a new probe list with the new gateway but leaves <code>TRACK_*</code>, <code>LATENCY_*</code>, and <code>LOSS_*</code> files from the old gateway IP on disk. The rpcd status function globs all <code>TRACK_*</code> files to build the tracking IP list, so stale files from previous gateways appear in <code>ubus call mwan3 status</code> alongside the current ones.</p>
<p>Fix: after building <code>track_ips</code>, scan the interface directory and delete any <code>TRACK_*</code>/<code>LATENCY_*</code>/<code>LOSS_*</code> files for IPs not in the current probe list. Runs at startup and on every ifup event, covering both initial cleanup of files left by a previous mwan3track instance and gateway changes during operation.</p>
<hr>
<h4 id="luci-app-mwan3-set-explicit-pkgversion-and-fix-release-workflow">luci-app-mwan3: set explicit PKG_VERSION and fix release workflow</h4>
<p>Without explicit versioning, <code>luci.mk</code> auto-generates the package version from git commit metadata. In the GitHub Actions SDK build environment the feed directory has no <code>.git</code> (excluded by rsync), so git walks up to the SDK repo and picks up the SDK commit hash and timestamp. The resulting version (e.g. <code>26.099.76803~8a085e7</code>) is indistinguishable from the official OpenWrt package version format, causing apk to install the official repo version.</p>
<p>Set <code>PKG_VERSION</code> to <code>$(PKG_SRC_PREFIX).$(PKG_SRC_SUFFIX)</code> where <code>PKG_SRC_PREFIX</code> is derived from the build year and a fixed day value of <code>999</code> (not a valid calendar day), ensuring it is always higher than any real date-based official package version for that year. <code>PKG_SRC_SUFFIX</code> encodes the mwan3 version number for readability.</p>
<p>Also fix the APK rename step to only match the main <code>luci-app-mwan3</code> package (pattern: <code>PKG_NAME[_-][0-9]*.ext</code>) instead of all APKs in the feed directory. The previous pattern matched all i18n sub-packages as well; since they all got renamed to the same target filename, the last one alphabetically clobbered the main package and was what actually got uploaded to the release.</p>
<hr>
<h4 id="luci-app-mwan3-improve-tracking-ip-latencyloss-display-for-downskippeddisabled-states">luci-app-mwan3: improve tracking IP latency/loss display for down/skipped/disabled states</h4>
<p>When <code>check_quality=1</code>, tracker latency/loss sentinel values (<code>999999ms</code>, <code>100%</code>) are replaced with more informative indicators:</p>
<ul>
<li>Tracker down: latency shows <code>∞</code> (scaled to be legible), packet loss retains <code>100%</code> since it is accurate</li>
<li>Tracker skipped: both show <code>-</code> since no probe was run that round</li>
<li>Interface disabled: tracker status shows 'Disabled' (muted) instead of 'Down', and both latency and loss show <code>-</code></li>
</ul>
<hr>
<h3 id="1824-version-322">18.24 Version 3.2.2</h3>
<p><strong>Summary:</strong> Version 3.2.2 fixes two misrouting bugs: duplicate jump rules accumulating from repeated fw4 reload cycles caused iface_in chain deletion to fail with "Resource busy", and the unguarded catchall rule in each <code>mwan3_iface_in_*</code> chain was stamping IPv6 packets with the IPv4 interface mark on dual-stack physical devices, breaking QUIC/HTTP3 streams that resumed after conntrack expiry. The gateway IP is moved to the front of the tracking probe list so it is always tested. LuCI receives a visual redesign replacing solid alert cards with bordered flex cards, and adds latency and packet-loss columns to the tracking IP table.</p>
<hr>
<h4 id="mwan3-fix-ifacein-jump-rule-deletion-failing-with-multiple-handles">mwan3: fix iface_in jump rule deletion failing with multiple handles</h4>
<p><code>mwan3_delete_iface_nft</code> used a single-shot handle lookup to remove the jump rule for an interface from <code>mwan3_ifaces_in</code>. If repeated fw4 reload cycles caused duplicate jump rules to accumulate, the handle variable received multiple space-separated values and the resulting nft command was syntactically invalid. The chain delete that followed then failed with "Resource busy" since the jump rule was still present.</p>
<p>Replace the single-shot lookup with a while loop using <code>head -n1</code>, matching the pattern already used for SNAT rule deletion in the same function. This correctly removes all copies of the jump rule one at a time and self-heals any pre-existing duplicates on the next ifdown.</p>
<hr>
<h4 id="mwan3-fix-ifacein-catchall-misclassifying-ipv6-on-shared-device-interfaces">mwan3: fix iface_in catchall misclassifying IPv6 on shared-device interfaces</h4>
<p>The catchall rule in each <code>mwan3_iface_in_*</code> chain lacked a <code>meta nfproto</code> filter. When an IPv4 and IPv6 mwan3 interface share the same physical device (e.g. a dual-stack PPPoE or L2TP WAN), <code>mwan3_ifaces_in</code> dispatches to the IPv4 chain first. The IPv4-specific <code>ip saddr</code> bypass rules do not match IPv6 packets, but the unguarded catchall does, stamping incoming IPv6 packets with the IPv4 interface mark instead of the IPv6 mark.</p>
<p>The ct mark is then saved with the wrong value. Subsequent packets for that connection restore the IPv4 mark, find no matching <code>ip -6</code> rule, and fall to the main routing table. For established TCP connections the bug is invisible because ct mark is set on the first outbound packet before any inbound packet arrives. For QUIC (UDP), conntrack entries expire after ~120-180s of inactivity; when the server resumes first the entry has <code>ct mark 0</code>, the catchall fires, and the connection is disrupted - manifesting as stalled streams or a gray page on sites using HTTP/3.</p>
<p>Fix by adding <code>meta nfproto ipv4</code>/<code>ipv6</code> to the catchall based on interface family, consistent with the protocol filters already applied to the <code>connected</code>/<code>custom</code>/<code>dynamic</code> bypass rules above it.</p>
<hr>
<h4 id="mwan3-ping-gateway-ip-first-when-trackgateway1">mwan3: ping gateway IP first when track_gateway=1</h4>
<p><code>mwan3_load_track_ips()</code> previously appended the gateway IP to the end of the <code>track_ips</code> list. Since mwan3track stops probing once <code>host_up_count</code> reaches the reliability threshold, the gateway would be skipped whenever an earlier IP responded.</p>
<p>Prepend the gateway instead so it is always probed first. This gives the gateway probe priority and ensures its reachability is tested on every round regardless of <code>reliability</code> setting.</p>
<hr>
<h4 id="mwan3-fix-rpcd-status-to-include-gateway-tracking-ip-and-restore-latencyloss">mwan3: fix rpcd status to include gateway tracking IP and restore latency/loss</h4>
<p><code>interfaces_status()</code> built the <code>track_ip</code> list from UCI <code>track_ip</code> options only, so the gateway IP (added dynamically by mwan3track when <code>track_gateway=1</code>) was never visible in ubus status output.</p>
<p>Replace the UCI iteration with a glob scan of the <code>TRACK_*</code> files that mwan3track actually writes, so all tracked IPs appear regardless of whether they come from UCI config or the dynamic gateway.</p>
<p>Also correct <code>get_mwan3track_status()</code> to treat <code>track_gateway=1</code> as an active tracking configuration (not disabled), and restore the latency/packetloss fields to the per-IP output (populated when <code>check_quality=1</code> is configured).</p>
<hr>
<h4 id="luci-app-mwan3-visual-redesign-of-status-cards-across-overview-and-main-status-page">luci-app-mwan3: visual redesign of status cards across overview and main status page</h4>
<p>Replace solid alert-message background cards with bordered flex cards throughout: interface status boxes now use a 2px border coloured to match the interface status (green/red/orange/grey) with no background fill.</p>
<p><code>detail.js</code>: status box border colour matches interface status; paused tracking state changed from warning orange to muted grey; tracking IP table rows sorted by status (up first, then down, then ignored).</p>
<p><code>overview.js</code>: policy section replaced with per-policy flex cards matching the interface card style; interface card borders coloured by status.</p>
<p><code>90_mwan3.js</code>: main LuCI status overview page cards converted from alert-message solid backgrounds to the same bordered card style.</p>
<hr>
<h4 id="luci-app-mwan3-add-latency-and-packet-loss-columns-to-tracking-ip-status-table">luci-app-mwan3: add latency and packet loss columns to tracking IP status table</h4>
<p>The rpcd mwan3 status now returns <code>latency</code>, <code>packetloss</code> and <code>check_quality</code> fields per interface. Extend the Status tab tracking IP table with Latency and Packet Loss columns.</p>
<p>When <code>check_quality</code> is disabled (the default), the columns display "Not enabled" in muted text rather than zeroed values.</p>
<hr>
<h3 id="1825-version-321">18.25 Version 3.2.1</h3>
<p><strong>Summary:</strong> Version 3.2.1 fixes policy status reporting to include all members with their live traffic share percentages (not just the currently-routing member), replaces <code>killall -HUP dnsmasq</code> with a procd-aware targeted SIGHUP to avoid crashing instances still in the startup phase, adds the installed mwan3 package version to <code>mwan3 internal</code> output, and redesigns the LuCI status pages with structured collapsible sections and an IPv6 troubleshooting pane. LuCI also exposes the <code>snat6</code> IPv6 SNAT option on interface configuration.</p>
<hr>
<h4 id="mwan3-show-mwan3-package-version-in-internal-troubleshooting-output">mwan3: show mwan3 package version in internal troubleshooting output</h4>
<p>Replace the OpenWrt release string in the Software-Version section of <code>mwan3 internal</code> with the installed mwan3 package version from apk.</p>
<hr>
<h4 id="mwan3-fix-policy-status-to-show-all-members-with-traffic-share">mwan3: fix policy status to show all members with traffic share</h4>
<p><code>mwan3_report_policies()</code> read the live nftables policy chain to determine members, which only contains the currently-routing member. Metric-2 standby members have no nft rules while metric-1 is up, so they were invisible in the status output.</p>
<p>The rpcd ucode <code>get_policies()</code> had the same problem, and additionally returned the raw nft mark value instead of the interface name.</p>
<p>Fix both by reading policy membership from UCI config and cross-referencing with mwan3track <code>STATUS</code> files to determine which metric is active and what the traffic share is for each member.</p>
<p>Every member now shows a percentage: <code>100%</code> for a sole active member, the load-balanced share for equal-priority members, and <code>0%</code> for standby/offline members. This makes failover and load-balancing configuration immediately readable at a glance.</p>
<p>The shell status command now calls ubus rather than duplicating the metric/weight/status logic in shell.</p>
<hr>
<h4 id="mwan3-fix-dnsmasq-sighup-race-during-concurrent-startup">mwan3: fix dnsmasq SIGHUP race during concurrent startup</h4>
<p><code>killall -HUP dnsmasq</code> signals every process named dnsmasq, including instances that are mid-initialisation. A dnsmasq receiving <code>SIGHUP</code> before it has completed startup exits, which can happen when mwan3's fw4-reload recovery (<code>25-mwan3</code> at position 25) fires concurrently with anything else restarting dnsmasq.</p>
<p>Replace <code>killall -HUP dnsmasq</code> with <code>mwan3_dnsmasq_hup()</code>, which queries procd via ubus for the dnsmasq service instances and sends <code>SIGHUP</code> only to PIDs that procd reports as <code>running=true</code>. Instances still in the startup phase are invisible to the function and are not signalled.</p>
<p><code>json_set_namespace</code> is used to protect the caller's jshn state since <code>mwan3.sh</code> uses jshn elsewhere.</p>
<hr>
<h4 id="luci-app-mwan3-redesign-status-pages-with-structured-views">luci-app-mwan3: redesign status pages with structured views</h4>
<p>Overview tab: replace simple interface cards with full operational view showing interface status cards (flex layout), policies table with per-member traffic share percentages, and rules table.</p>
<p>Status tab: replace static interface cards with per-interface tracking health panels showing tracking mode, score, and a table of probe IPs with up/down/ignored status using coloured text indicators.</p>
<p>Troubleshooting tab: replace raw pre-formatted text dump with collapsible sections (collapsed by default with expand arrow), IPv6 sections alongside IPv4, and vmap-dispatch boilerplate chains filtered from the nftables output. Add ACL entry for <code>mwan3 internal ipv6</code>.</p>
<hr>
<h4 id="luci-app-mwan3-expose-mwan3-snat6-option-for-ipv6-interfaces">luci-app-mwan3: expose mwan3 snat6 option for IPv6 interfaces</h4>
<p>Adds an "IPv6 SNAT" form field to the interface configuration modal, visible only when <code>family</code> is set to <code>ipv6</code>. Mirrors the placement of the existing "Track gateway" field. Accepts unset/0 (off, default), <code>1</code> (SNAT to the interface's primary GUA), or a literal IPv6 address for NPTv6-style fixed-source pinning. The control is opt-in by design -- see the mwan3 package for the rationale (RFC 6724 source-address selection, NAT66 harm in PA/ULA designs, fixed-saddr upstream requirements).</p>
<hr>
<h3 id="1826-version-32">18.26 Version 3.2</h3>
<p><strong>Summary:</strong> Version 3.2 adds two significant features. First, opt-in per-interface IPv6 SNAT via the <code>snat6</code> UCI option, which corrects BCP38/uRPF drops for router-originated traffic rerouted by <code>mwan3_output</code> onto a different WAN than the kernel initially selected at <code>sendto()</code>. Second, non-destructive vmap-dispatch mark save/restore: 126 per-mark OR-immediate setter chains replace the previous unmasked connmark operations, making mwan3 fully order-independent with respect to pbr and other fwmark-using packages without requiring coordinated chain priority ordering.</p>
<hr>
<h4 id="mwan3-add-opt-in-ipv6-postrouting-snat-via-snat6-uci-option">mwan3: add opt-in IPv6 postrouting SNAT via snat6 UCI option</h4>
<p>Adds a <code>mwan3_postrouting</code> base chain (<code>type nat hook postrouting priority srcnat - 1</code>) and opt-in per-interface IPv6 SNAT for router-originated traffic that <code>mwan3_output</code> has rerouted onto a different WAN.</p>
<p>The kernel binds saddr at <code>sendto()</code> using the unmarked initial route; <code>mwan3_output</code> then reroutes oif onto a different WAN; without SNAT the packet egresses with the wrong source prefix and is dropped upstream by BCP38/uRPF. fw4 does not masquerade IPv6 by default so there is no automatic fallback. <code>mwan3track</code> is unaffected because it uses <code>SO_BINDTODEVICE</code>. IPv4 router-originated traffic does not require explicit SNAT -- fw4's masquerade handles it.</p>
<p>IPv6 SNAT is opt-in per interface via the new <code>snat6</code> UCI option, defaulting to off because RFC 6724 source-address selection / SADR routing can solve the same problem without translation, NAT66 is harmful in PA/ULA designs, and some upstreams require a specific saddr. <code>snat6</code> values: unset/0 (off), <code>1</code> (SNAT to interface primary GUA via <code>mwan3_get_src_ip</code>), or a literal v6 address (NPTv6-style fixed-source pinning). Cleanup is via comment-tag matching (<code>"mwan3_snat_&lt;iface&gt;"</code>).</p>
<hr>
<h4 id="mwan3-non-destructive-vmap-dispatch-saverestore">mwan3: non-destructive vmap-dispatch save/restore</h4>
<p>Restores iptables-equivalent non-destructive masked CONNMARK semantics by synthesising masked save/restore via vmap dispatch into 126 per-mark OR-immediate setter chains (<code>mwan3_or_meta_*</code>, <code>mwan3_or_ct_*</code>, 63 each). Each setter chain is a 2-statement skeleton <code>"meta/ct mark set ... | &lt;imm&gt;; return"</code>. The kernel rejects the obvious compound form <code>"(meta mark &amp; ~M) | (ct mark &amp; M)"</code> because a set-statement may reference at most one runtime source register; vmap-dispatch routes around this by materialising the constant immediate at rule-emit time.</p>
<p>Because the save and restore operations are now non-destructive, mwan3's fwmark bits are ORed in and out of the conntrack mark without disturbing bits owned by other packages. This makes mwan3 order-independent with respect to other fwmark-using packages such as pbr, regardless of chain priority ordering.</p>
<p>The same vmap-dispatch primitive is reused by <code>mwan3_create_policies_nft</code> for load-balancing numgen rules and by the sticky implementation in <code>mwan3_set_user_nft_rule</code>, both of which were previously destructive in unmasked bits. Sticky routing is rebuilt around per-(rule, family, member) ip-only sets that pair with the per-mark setter chains, replacing the legacy <code>ip-&gt;mark</code> map. <code>mwan3_delete_iface_map_entries</code> and <code>mwan3_report_policies</code> are updated for the new sticky and numgen forms.</p>
<hr>
<h3 id="1827-version-314">18.27 Version 3.1.4</h3>
<p><strong>Summary:</strong> Version 3.1.4 fixes interoperability with pbr by moving mwan3's prerouting and output chains from priority <code>mangle + 1</code> to <code>mangle - 1</code>, so mwan3 restores and saves its ct mark bits before pbr injects its own marks at <code>mangle</code> priority. With the previous ordering pbr's marks were zeroed before the routing decision and its ip rules never matched.</p>
<hr>
<h4 id="mwan3-fix-pbr-interoperability-by-running-at-priority-mangle---1">mwan3: fix pbr interoperability by running at priority mangle - 1</h4>
<p>Move <code>mwan3_prerouting</code> and <code>mwan3_output</code> from priority <code>mangle + 1</code> to <code>mangle - 1</code> so mwan3 runs before pbr, which injects into fw4's <code>mangle_prerouting</code> at priority <code>mangle</code> (-150).</p>
<p>With the old priority, pbr set marks in <code>0x00ff0000</code> first, then mwan3's unmasked restore (<code>meta mark set ct mark &amp; MMX_MASK</code>) zeroed them before the routing decision, causing pbr's ip rules to never match.</p>
<p>With <code>mangle - 1</code>, mwan3 restores and saves its mark before pbr runs. pbr then adds its bits on top, and both sets of marks are present at the routing decision -- matching the coexistence behaviour of the original iptables implementation.</p>
<p>Add <code>postinst</code> migration to flush and delete the old chains on upgrade, since nftables rejects a chain redeclaration with a different priority.</p>
<hr>
<h3 id="1828-version-313">18.28 Version 3.1.3</h3>
<p><strong>Summary:</strong> Version 3.1.3 fixes three status and policy rendering bugs: single-member policies were emitting spurious "unreachable" entries because the empty-string guard on <code>mwan3_mark_to_name</code> never matched; mixed IPv4/IPv6 policies lost one family's members because both shared a single reset list; and equal-weight load-balancing entries were invisible in <code>mwan3 status</code> because nft normalises single-element numgen ranges to plain values that the reporting regex did not match.</p>
<hr>
<h4 id="mwan3-fix-status-reporting-for-single-member-and-mixed-family-policies">mwan3: fix status reporting for single-member and mixed-family policies</h4>
<p><code>mwan3_report_policies</code> used <code>mwan3_mark_to_name</code> on the mark from the single-member branch and tested <code>[ -n "$iface_name" ]</code> to skip special marks. However <code>mwan3_mark_to_name</code> never returns an empty string: it returns "unreachable", "blackhole", "default", or the raw hex value as a fallthrough for unrecognised marks. The empty-string test therefore never filtered anything, causing spurious "unreachable" output.</p>
<p>Replace the empty-string guard with a <code>case</code> statement that explicitly skips the known non-interface values (<code>unreachable</code>, <code>blackhole</code>, <code>default</code>, and raw <code>0x...</code> hex fallthrough). Also iterate all <code>"meta mark set"</code> rules in the chain rather than only the first, so mixed-family policies with one IPv4 and one IPv6 member both appear in the status output.</p>
<hr>
<h4 id="mwan3-fix-cross-family-member-reset-in-mixed-ipv4ipv6-policies">mwan3: fix cross-family member reset in mixed IPv4/IPv6 policies</h4>
<p>When a policy contains members from both IPv4 and IPv6 interfaces, the previous code used a single <code>policy_members</code> list that was reset on each new lowest-metric member regardless of address family. This caused IPv4 members to be erased when a lower-metric IPv6 member was processed (or vice versa), leaving the policy with only the last-processed family.</p>
<p>Fix by maintaining separate <code>policy_members_v4</code> and <code>policy_members_v6</code> lists, each reset only when a new lowest-metric member of the same family is encountered. When both families have members, emit per-family nft rules guarded with <code>meta nfproto ipv4</code>/<code>ipv6</code> so traffic is only directed to members of the matching address family.</p>
<hr>
<h4 id="mwan3-fix-load-balancing-policy-not-shown-in-status">mwan3: fix load balancing policy not shown in status</h4>
<p>nft normalizes single-element ranges (e.g. <code>0-0</code>) to plain values (e.g. <code>0</code>) when listing rules. The reporting regex only matched the range format <code>N-M : 0xMARK</code>, so equal-weight load balancing entries (weight=1, displayed as <code>N : 0xMARK</code>) were silently skipped, causing the policy to appear empty in <code>mwan3 status</code> despite working correctly.</p>
<p>Handle both <code>N-M : 0xMARK</code> (weight&gt;1, range preserved by nft) and <code>N : 0xMARK</code> (weight=1, normalized by nft) formats.</p>
<hr>
<h3 id="1829-version-312">18.29 Version 3.1.2</h3>
<p><strong>Summary:</strong> Version 3.1.2 improves mwan3rtmon with two fixes: an in-memory route cache replaces the per-event <code>RTM_GETROUTE</code> dump for O(1) ECMP path checks, and a ucode-mod-rtnl double-destructor bug that caused a reliable segfault on clean shutdown is eliminated by letting the GC collect the route listener rather than calling <code>close()</code> explicitly.</p>
<hr>
<h4 id="mwan3rtmon-replace-routestillexists-with-an-in-memory-route-cache">mwan3rtmon: replace route_still_exists() with an in-memory route cache</h4>
<p><code>route_still_exists()</code> issued a full <code>RTM_GETROUTE</code> dump on every route-delete event to check whether an ECMP path still existed before removing per-interface table entries. This is unnecessary overhead.</p>
<p>Replace with <code>main_route_cache</code>: a <code>{ route_key: count }</code> map built from the initial route snapshot in <code>populate_iface_routes()</code> and maintained incrementally in <code>handle_route_event()</code>. The ECMP check becomes an O(1) cache lookup with no rtnl round-trip.</p>
<hr>
<h4 id="mwan3rtmon-fix-segfault-on-exit-caused-by-ucode-mod-rtnl-double-destructor-bug">mwan3rtmon: fix segfault on exit caused by ucode-mod-rtnl double-destructor bug</h4>
<p>Calling <code>route_listener.close()</code> explicitly zeroes the resource data pointer while the ucode variable still holds a live reference. When that reference is later released at scope exit, <code>uc_nl_listener_free()</code> fires a second time with <code>arg=NULL</code> and reads <code>uc_nl_listener_t.index</code> at <code>NULL+0x10</code>, producing a reliable "segfault at 10" on every clean shutdown.</p>
<p>Fix: omit the explicit <code>close()</code> call and let the GC collect the listener naturally. The destructor then fires exactly once with a valid pointer.</p>
<hr>
<h3 id="1830-version-311">18.30 Version 3.1.1</h3>
<p><strong>Summary:</strong> Version 3.1.1 is a broad mwan3track hardening release: the disconnecting threshold is raised to suppress false alarms from single transient ping losses, an exclusive flock prevents ghost duplicate tracker processes per interface, <code>sockopt_wrap</code> replaces <code>exit()</code> with graceful error returns so a stale source IP or disappearing interface does not abruptly terminate the tracked process, per-host failure logs are suppressed when the reliability threshold is still met, and interface events are processed at the top of the main loop before pinging to avoid a spurious disconnecting state on wakeup from disabled. LuCI adds a track_gateway checkbox to the interface modal and clarifies the flush_conntrack help text.</p>
<hr>
<h4 id="mwan3track-raise-disconnecting-threshold-and-log-recovery">mwan3track: raise disconnecting threshold and log recovery</h4>
<p>Raise the threshold at which the disconnecting state fires from the first score drop to <code>ceil(down/3)</code> failures below the maximum score. This prevents single or double transient ping losses (e.g. to a public DNS server) from spuriously triggering the disconnecting state.</p>
<p>Log an explicit notice when the score recovers out of the disconnecting state, making the transition back to online visible in the log.</p>
<hr>
<h4 id="mwan3track-use-flock-to-prevent-ghost-processes-for-same-interface">mwan3track: use flock to prevent ghost processes for same interface</h4>
<p>Use an exclusive flock on a per-interface lock file to ensure only one mwan3track instance runs per interface at a time. The lock is held for the lifetime of the process and released unconditionally on exit.</p>
<hr>
<h4 id="mwan3-sockoptwrap-replace-exit-with-graceful-error-returns">mwan3: sockopt_wrap: replace exit() with graceful error returns</h4>
<p>A <code>LD_PRELOAD</code> shim must not call <code>exit()</code> on recoverable errors; doing so terminates the tracked process abruptly with no log from mwan3track, making it indistinguishable from a genuine connectivity failure.</p>
<p>Three cases are fixed:</p>
<ul>
<li>Source IP bind failure in <code>dobind()</code>: can occur when <code>SRC_IP</code> becomes stale after a DHCP address change that does not generate an ifup event. Close the socket and return; the subsequent <code>sendto()</code>/<code>connect()</code> call will fail with <code>EBADF</code>, causing the ping to exit with a normal error code that mwan3track records as a ping failure.</li>
</ul>
<ul>
<li><code>SO_BINDTODEVICE</code> failure in <code>socket()</code>: can occur if the interface disappears between m### mwan3track: suppress per-host failure logs when reliability threshold is met</li>
</ul>
<p>With multiple track IPs and <code>reliability &lt; number of IPs</code>, a single host failure was logged as "Check failed for target X" even when a subsequent host met the reliability threshold and the round succeeded. 95% of all logged failures are false alarms of this kind, making it appear the interfaces are degrading when they are in fact healthy.</p>
<p>Fix the excessive log noise by accumulating failed host names during the probe loop and logging them in a single message after the loop, when <code>host_up_count</code> is still below the reliability threshold (round genuinely failed). Rounds that succeed via a later host produce no failure log. The success log during recovery remains per-host and unchanged.</p>
<p>For <code>check_quality</code> mode the accumulated entry includes per-host latency and loss: <code>"target(s) \"1.2.3.4(999999ms/100%)\""</code>.</p>
<hr>
<h4 id="mwan3track-process-interface-events-before-ping-round-on-wakeup">mwan3track: process interface events before ping round on wakeup</h4>
<p>When <code>USR2</code> (ifup) wakes mwan3track from disabled state, the main loop resumes past the <code>MAX_SLEEP</code> wait and immediately starts a ping round before reaching the <code>IFUP_EVENT</code> handler at the bottom of the loop. At that point <code>DEVICE</code> is still stale (empty string at first start), so <code>sockopt_wrap</code> skips <code>SO_BINDTODEVICE</code>, the unbound ping fails, and a spurious "disconnecting" state is logged.</p>
<p>Fix this by checking <code>IFDOWN_EVENT</code>/<code>IFUP_EVENT</code> at the top of the loop, before any pinging, and using <code>continue</code> to restart the iteration cleanly after <code>firstconnect()</code> has refreshed <code>DEVICE</code> and <code>SRC_IP</code>. The existing handlers at the bottom of the loop are retained for events that arrive during a ping round or the inter-round sleep.</p>
<hr>
<h4 id="luci-app-mwan3-clarify-flushconntrack-help-text">luci-app-mwan3: clarify flush_conntrack help text</h4>
<p>Update the help text for the <code>flush_conntrack</code> option to clarify that it flushes the entire global conntrack table, and that per-interface conntrack entries are already flushed automatically on ifdown since commit ("mwan3: selectively flush conntrack entries for failed WAN interface on ifdown").</p>
<hr>
<h4 id="luci-app-mwan3-add-trackgateway-option-to-interface-settings">luci-app-mwan3: add track_gateway option to interface settings</h4>
<p>Add a "Track gateway" checkbox to the interface configuration modal, visible only when the internet protocol is set to IPv4. This exposes the <code>track_gateway</code> UCI option added to the mwan3 backend for automatic point-to-point peer/gateway tracking.</p>
<hr>
<h2 id="19-specific-use-cases">19. Specific use-cases</h2>
<h3 id="191-tailscale">19.1 Tailscale</h3>
<p>Tailscale's specific routing architecture is incompatible with legacy mwan3, meaning that Tailscale had to explicitly bypass mwan3 and to that end, has specific detection code and a bypass. Tailscale cannot be managed with legacy mwan3 policies.</p>
<p>mwan3 nf tables, however, has the requisite functionality to enable peaceful co-existence between the two packages. If mwan3 is left in default configuration, Tailscale will continue to bypass mwan3 as it has always done and users will notice no difference.</p>
<p>However, mwan3 can be specifically configured to policy route Tailscale bypass-marked traffic. The rest of this section explains how to do that.</p>
<h4 id="configuring-mwan3-failover-for-tailscale-control-traffic">Configuring mwan3 failover for tailscale control traffic</h4>
<p>Tailscale marks every socket it opens for its own outbound connections (DERP relay servers, coordination server, STUN) with <code>SO_MARK = 0x80000</code> at socket creation time via <code>SetsockoptInt</code>. In normal operation with mwan3, it then installs an ip rule at priority 1310 that intercepts these marked packets and routes them to the main routing table (table 254, managed by netifd), bypassing mwan3's per-interface tables entirely:</p>
<pre><code>1310: fwmark 0x80000/0xff0000 lookup main (254)
1330: fwmark 0x80000/0xff0000 lookup default (253)
1350: fwmark 0x80000/0xff0000 unreachable
1370: (unconditional) lookup tailscale (52)</code></pre>
<p>Tailscale installs these rules at the 1300 base specifically because it detects OpenWrt with mwan3 active (<code>checkOpenWRTUsingMWAN3()</code> in <code>wgengine/router/osrouter/router_linux.go</code>) and shifts its default base from 5200 to 1300. The 1300 base is chosen to fit between mwan3's default iif rules (1001-1060) and mwan3's default fwmark lookup rules (2001-2060). The rules at 1310-1350 are an anti-loop mechanism: if tailscaled's own outbound traffic fell through to rule 1370 and was routed into <code>tailscale0</code>, it would re-enter the overlay indefinitely.</p>
<p>The problem is that rule 1310 routes tailscale's bypass-marked traffic via the main routing table, which is maintained exclusively by netifd and is not subject to mwan3's failover machinery. In the most common real-world WAN outage (upstream unreachable with carrier still up), netifd sees carrier and leaves the primary WAN route in the main table, so tailscale's control traffic continues trying to reach DERP servers via the dead upstream while mwan3 has already failed LAN traffic over to the backup WAN.</p>
<p>Tailscale requires this in order to prevent a routing loop, whereby bypass-marked packets enter the tailscale device via routing table 52 and was necessary when using Tailscale with legacy mwan3.</p>
<p>Making use of three features in mwan3 nftables, it is possible to get tailscale to benefit from mwan3 while also avoiding the aforementioned loop.</p>
<p>The solution has three parts:</p>
<ol>
<li>A new mwan3 rule type that matches on a fwmark/fwmask.</li>
</ol>
<ol>
<li>The ability to shift the mwan3 rule priorities to a new base</li>
</ol>
<ol>
<li>Dynamic route monitoring by mwan3rtmon that mirrors routes from Tailscale's table 52 defined with <code>list rt_table_lookup '52'</code> in the globals section of <code>/etc/config/mwan3</code>. mwan3rtmon will dynamically mirror peer IP/CIDRs in table 52 to the mwan3 sets <code>mwan3_custom_v4/v6</code>. Anything present in these sets will thus bypass mwan3's rules, fall through to the next rules in the list and get correctly routed by Tailscale's table 52. </li>
</ol>
<h4 id="how-the-three-parts-work-to-integrate-mwan3-and-tailscale">How the three parts work to integrate mwan3 and tailscale</h4>
<ol>
<li>The mwan3 rule matches on <code>fwmark 0x80000/0xff0000</code> and assigns the tailscale bypass marked packets to an mwan3 policy</li>
<li>The mwan3 rule fires before tailscale's bypass because of the route ordering priority change and routes out whichever is the active WAN. This gives the failover: the bypass marked packets are the outer tunnel / control plane</li>
<li>Traffic destined for the tailscale inner tunnel, which is normally looked up in table 52, has matching IPs and CIDRs in the <code>mwan3_custom_v4/6</code> sets put there by mwan3rtmon. Any destination IP in those sets will now bypass mwan3's specific handling and fall through to tailscale's rule that looks them up in table 52, routing them out the <code>tailscale0</code> device.</li>
</ol>
<h4 id="mwan3s-fwmark-and-tailscales-bypass-mark-do-not-interfere-with-each-other">mwan3's fwmark and tailscale's bypass mark do not interfere with each other</h4>
<p>mwan3 uses bits 8-13 of the packet mark (<code>0x3f00</code> default mask). Tailscale's bypass mark uses bits 16-23 (<code>0xff0000</code>). The two ranges do not overlap. mwan3's mark-set operation is non-destructive: it uses a bitwise OR via vmap-dispatch setter chains, so it writes only to the bits it owns and leaves all other bits intact.</p>
<p>After <code>mwan3_output</code> fires on a tailscale bypass packet, the packet carries both marks simultaneously in separate bit fields:</p>
<ul>
<li>bits 16-23: <code>0x80000</code> (tailscale bypass mark, unchanged)</li>
<li>bits 8-13: <code>0x100</code> (mwan3 policy mark for interface id 1, ORed in)</li>
<li>final mark: <code>0x80100</code></li>
</ul>
<h4 id="choosing-the-priority-values">Choosing the priority values</h4>
<p>The only constraint is that mwan3's fwmark rules must fall outside of tailscale's mwan3 detection window. <code>checkOpenWRTUsingMWAN3()</code> looks for ip rules at priorities 2001-2004 with a non-zero fwmark. If any fwmark rule falls in that range, tailscale detects mwan3 and shifts its rules to a 1300 base, which would interleave with mwan3's fwmark rules. The constraint is:</p>
<pre><code>fwmark_rule_base + MWAN3_INTERFACE_MAX &lt; 2001</code></pre>
<p>With the default mask <code>0x3f00</code> (6 bits, 60 maximum interfaces), this gives <code>fwmark_rule_base &lt; 1941</code>. The recommended value is <code>fwmark_rule_base = 1100</code>, which places the highest-numbered fwmark lookup rule at 1160, well clear of the detection window. Tailscale therefore does not detect mwan3 and installs at its default 5210 base instead.</p>
<p>Because <code>iif_rule_base</code> defaults to 1000 and can stay there, only two UCI options need to be changed from their defaults:</p>
<table>
<thead><tr><th>Option</th><th>Default</th><th>Recommended for tailscale</th></tr></thead><tbody>
<tr><td><code>iif_rule_base</code></td><td>1000</td><td>1000 (unchanged)</td></tr>
<tr><td><code>fwmark_rule_base</code></td><td>2000</td><td>1100</td></tr>
<tr><td><code>unreachable_rule_base</code></td><td>3000</td><td>1200</td></tr>
</tbody></table>
<p>The ordering constraints validated by mwan3 at startup both hold:</p>
<ul>
<li><code>1000 + 60 = 1060 &lt; 1100</code> (iif tier clears fwmark tier)</li>
<li><code>1100 + 60 + 1 = 1161 &lt; 1200</code> (fwmark tier including global blackhole/unreachable at 1161/1162 clears unreachable tier)</li>
</ul>
<h4 id="why-rule-5270-table-52-still-works-correctly">Why rule 5270 (table 52) still works correctly</h4>
<p>Tailscale's rule 5270 is unconditional - it fires for every packet that reaches it. With mwan3's fwmark rules at 1101-1160, rule 5270 is reached only by packets whose <code>mark &amp; 0x3f00</code> did not match any live per-interface id mark. The relevant case is packets carrying <code>MMX_DEFAULT</code> (mark = <code>0x3f00</code>).</p>
<p><code>mwan3_prerouting</code> marks packets destined for networks in <code>mwan3_connected_v4</code> and <code>mwan3_custom_v4</code> with <code>MMX_DEFAULT</code> as an early-return bypass. <code>MMX_DEFAULT</code> does not match any per-interface fwmark rule (those require specific id marks, not all-bits-set), so these packets fall through to rule 5270, which looks up table 52 and routes them to <code>tailscale0</code>.</p>
<p>Ensuring tailscale peer destinations appear in <code>mwan3_custom_v4</code> is therefore the prerequisite for correct tailnet routing after the <code>fwmark_rule_base</code> change. This is described in the next section.</p>
<h4 id="ensuring-tailnet-destined-traffic-reaches-tailscale0">Ensuring tailnet-destined traffic reaches <code>tailscale0</code></h4>
<p>Tailscale installs peer routes exclusively into table 52 via netlink. The main routing table receives no tailscale peer routes. With mwan3's fwmark rules moved to 1101-1160, the default catch-all mwan3 rule assigns a WAN policy mark to all LAN traffic including packets destined for tailscale peers. That marked traffic hits rule 1101 before it reaches rule 5270. The per-interface routing table has a default route to the internet gateway but no route to the tailscale peer. The packet exits via the WAN unencrypted.</p>
<p>Under the default mwan3 configuration (fwmark rules at 2001+), tailscale rule 1370 fires at priority 1370, before the fwmark lookup rules at 2001+. The WAN policy mark is shadowed by rule 1370 and the packet reaches tailscale0 correctly. Moving the fwmark base to 1100 removes that shadowing.</p>
<p>The fix does not require touching routing tables. mwan3 provides <code>mwan3_custom_v4</code> and <code>mwan3_custom_v6</code>, nftables sets whose members receive <code>MMX_DEFAULT</code> in <code>mwan3_prerouting</code> and thereby bypass WAN policy selection entirely. The <code>rt_table_lookup</code> option in mwan3 globals instructs mwan3 to populate <code>mwan3_custom_v4/v6</code> from the routes in a specified routing table at start. Adding table 52 to this list:</p>
<pre><code>list rt_table_lookup '52'</code></pre>
<p>causes mwan3 to read all current table 52 routes at start and add their destinations to <code>mwan3_custom_v4/v6</code>. Packets to tailscale peers receive <code>MMX_DEFAULT</code>, pass through all per-interface fwmark rules without matching, and reach rule 5270 (<code>from all lookup tailscale</code>), which routes them to <code>tailscale0</code> via table 52. No per-interface routing table needs a tailscale route.</p>
<p>This is the correct treatment for the inner tailnet packets. <code>tailscale0</code> is a locally-connected TUN device: the inner packets are delivered to the tailscale daemon, which wireguard-encapsulates them into UDP. Those outer packets carry SO_MARK=0x80000 and are subject to mwan3 WAN failover via the dual-mark mechanism. <code>MMX_DEFAULT</code> on the inner packets correctly expresses that WAN selection does not apply at this layer; it is applied at the outer encrypted layer instead.</p>
<p>Tailscale adds and removes peer routes from table 52 dynamically as peers join and leave the tailnet. mwan3rtmon watches for <code>RTM_NEWROUTE</code> and <code>RTM_DELROUTE</code> events on tables listed in <code>rt_table_lookup</code> and updates <code>mwan3_custom_v4/v6</code> accordingly, so the set remains current without any external script or daemon.</p>
<p>If mwan3 starts before tailscale has populated table 52 (for example, on boot), the initial <code>rt_table_lookup</code> pass reads an empty or partial table 52 and <code>mwan3_custom_v4</code> is initially partially complete. mwan3rtmon's event handler fills in the missing entries as tailscale adds routes after its own startup, so the set converges to the correct state within seconds of tailscale coming up.</p>
<h4 id="the-resulting-rule-layout">The resulting rule layout</h4>
<p>With <code>fwmark_rule_base = 1100</code> and <code>unreachable_rule_base = 1200</code>, and four mwan3 interfaces for illustration. Tailscale's ip rules are shown in their steady-state position, explained below. The moving of mwan3's base priority means that tailscale won't actually detect mwan3 to be running and will revert its own rule priority to the default of 5210 instead of installing at 1310.</p>
<pre><code> 0: from all lookup local
1001: iif &lt;wan1-dev&gt; lookup 1 # mwan3 iif
1002: iif &lt;wan2-dev&gt; lookup 2 # mwan3 iif
1003: iif &lt;wan3-dev&gt; lookup 3 # mwan3 iif
1004: iif &lt;wan4-dev&gt; lookup 4 # mwan3 iif
1101: fwmark 0x100/0x3f00 lookup 1 # mwan3 fwmark
1102: fwmark 0x200/0x3f00 lookup 2 # mwan3 fwmark
1103: fwmark 0x300/0x3f00 lookup 3 # mwan3 fwmark
1104: fwmark 0x400/0x3f00 lookup 4 # mwan3 fwmark
1161: fwmark 0x3d00/0x3f00 blackhole # mwan3 blackhole
1162: fwmark 0x3e00/0x3f00 unreachable # mwan3 unreachable
1201: fwmark 0x100/0x3f00 unreachable # mwan3 unreachable
1202: fwmark 0x200/0x3f00 unreachable # mwan3 unreachable
1203: fwmark 0x300/0x3f00 unreachable # mwan3 unreachable
1204: fwmark 0x400/0x3f00 unreachable # mwan3 unreachable
5210: fwmark 0x80000/0xff0000 lookup main (254) # tailscale anti-loop
5230: fwmark 0x80000/0xff0000 lookup default (253) # tailscale anti-loop
5250: fwmark 0x80000/0xff0000 unreachable # tailscale anti-loop
5270: from all lookup tailscale (52) # tailscale overlay routing
32766: from all lookup main
32767: from all lookup default</code></pre>
<p>Tailscale's mwan3 detection (<code>checkOpenWRTUsingMWAN3()</code> in <code>wgengine/router/osrouter/router_linux.go</code>) identifies mwan3 by looking for ip rules at priorities 2001-2004 with a non-zero fwmark. With <code>fwmark_rule_base = 1100</code> those rules are at 1101-1104, outside the detection window. Tailscale therefore does not detect mwan3 and installs its rules at its default base of 5200 (priorities 5210, 5230, 5250, 5270) .</p>
<p>This does not affect correctness. Tailscale bypass-marked packets (<code>0x80000</code>) acquire an mwan3 policy mark in <code>mwan3_output</code> and are caught at 1101-1104, well before tailscale's rules at 5210. The anti-loop concern that motivated rules 5210-5250 does not apply: the per-interface routing tables contain internet routes, not tailscale overlay routes, so there is no loop path. Tailnet-destined LAN traffic carries <code>MMX_DEFAULT</code> (via <code>mwan3_custom_v4</code>, described above) and passes through all fwmark rules without matching before reaching rule 5270, which routes via table 52 to <code>tailscale0</code>.</p>
<h4 id="configuration">Configuration</h4>
<p>In <code>/etc/config/mwan3</code>, in the <code>config globals</code> section, change the two rule base options and add the <code>rt_table_lookup</code> entry for tailscale's routing table (this can also be done on the Globals tab in Luci, as well as a bypass network <code>list bypass_network '100.64.0.0/10'</code>:</p>
<pre><code>config globals 'globals'
option mmx_mask '0x3F00'
option fwmark_rule_base '1100'
option unreachable_rule_base '1200'
list rt_table_lookup '52'
list bypass_network '100.64.0.0/10'</code></pre>
<p>Add a rule section for tailscale bypass traffic, placed before any catch-all default rules, preferably as the very first rule in the list:</p>
<pre><code>config rule 'tailscale_bypass'
option fwmark '0x80000'
option fwmask '0xff0000'
option family 'any'
option use_policy '&lt;my_failover_policy&gt;'
option enabled '1'</code></pre>
<p>Replace <code>&lt;my_failover_policy&gt;</code> with whichever mwan3 policy should carry tailscale's control traffic. Recommend you use a failover policy and not a balanced policy for this purpose. After saving, run:</p>
<pre><code>service mwan3 restart
service tailscale restart</code></pre>
<p>A reload (<code>service mwan3 reload</code>) updates the nftables rules and the ip rules atomically via <code>mwan3_delete_iface_rules</code> (which finds and removes old rules by routing table id regardless of their previous priority), so old rules at the legacy 2000/3000 bases are correctly cleaned up.</p>
<p>After restart, verify:</p>
<pre><code># mwan3's fwmark rules at 1101-1160, tailscale's rules at 5210-5270
ip rule list
# The tailscale bypass rule appears in mwan3_rules
nft list chain inet mwan3 mwan3_rules | grep fwmark
# Expected: meta mark &amp; 0x00ff0000 == 0x00080000 ... jump mwan3_policy_&lt;name&gt;
# Tailscale peer destinations are in mwan3_custom_v4
nft list set inet mwan3 mwan3_custom_v4</code></pre>
<h4 id="forcing-immediate-failover-of-tailscale-derp-connections">Forcing immediate failover of tailscale DERP connections</h4>
<p>DERP connections that were open at the moment of WAN failover remain on the old WAN until they close naturally. Only sockets opened after tailscale reconnects pick up the new fwmark ip rule. They should failover quite quickly anyway, but to force immediate failover, put the following into <code>/etc/mwan3.user</code></p>
<pre><code>#!/bin/sh
[ "$ACTION" = "ifup" ] || exit 0
. /lib/functions.sh
. /lib/mwan3/common.sh
. /lib/mwan3/mwan3.sh
config_load mwan3
mwan3_init
mwan3_flush_marked_conntrack</code></pre>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink