cwx/op-packages
1
0 Fork 0
mirror of https://github.com/kiddin9/op-packages.git synced 2026-10-29 23:15:13 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

💋 Sync 2026-10-04 04:16:09

Browse Source
This commit is contained in:
github-actions[bot] committed 2026-10-04 04:16:09 +08:00
1 parent f6b2df73ec
commit f62992fd80
745 files changed
+212416 -1173
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

luci-app-appflow/docs/DESIGN.md
+984
View File
@@ -0,0 +1,984 @@
# appflow: Design
Status: v1 spec LOCKED (2026-08-21). Remaining open items tracked in §8.
## 1. Goal
A LuCI-native, fully open per-application traffic dashboard for OpenWrt, built
on the Netify Agent. Stock OpenWrt can tell you how much traffic crossed an
interface; it cannot tell you which application produced it. Several vendor
firmwares ship a DPI traffic-statistics screen that can, and this is an open
equivalent for stock OpenWrt.
Primary scope (v1): **live view**, per-application and per-device rates and
session totals, category grouping, live charts.
Explicit non-goal for v1, but designed-for: **historical accounting** (see §7).
## 2. Ground truth (verified on hardware, 2026-08-21)
Test platform: Linksys EA8500, OpenWrt 25.12.5, netifyd 4.4.7 (nDPI 5.0-based),
arch `arm_cortex-a15_neon-vfpv4`. All facts below observed live, not assumed.
### 2.1 Data source: netifyd JSON export socket
- netifyd **listens** on `unix:/var/run/netifyd/netifyd.sock`
(`[socket] listen_path[0]`, enabled by default in the OpenWrt package).
- ucode connects with plain `socket.connect(path)`, verified.
- **Wire framing**: newline-delimited JSON where each payload line is preceded
by a header line `{"length": N}`. Parser rule: parse each line; a lone
`length` key = frame header (skip); anything else dispatches on `type`.
- Event types observed: `agent_hello`, `definitions`, `agent_status` (15 s
cadence), `flow`, `flow_stats`.
- `/var/run/netifyd/status.json` contains **agent status only** (plus a useful
MAC→IP `devices` map), no flow data. The socket is the only flow source.
### 2.2 Event schemas (as observed)
`flow` (fires on detection; `detection_updated` re-fires on refinement):
```json
{ "type": "flow", "interface": "wan", "internal": false,
"flow": {
"digest": "9c2d…", // unique flow key
"detected_application": 10465,
"detected_application_name": "netify.openwrt",
"detected_protocol": 5,
"detected_protocol_name": "DNS",
"category": { "application": 31, "protocol": 14, "domain": 0 },
"host_server_name": "openwrt.org", // SNI/hostname when known
"local_ip": "…", "local_mac": "…", "local_port": 53,
"other_ip": "…", "other_mac": "…", "other_port": 57052,
"other_type": "local" | "remote",
"local_origin": bool, "ip_nat": bool, "ip_version": 4, "ip_protocol": 17,
"first_seen_at": 1787325075857, "last_seen_at": …, // ms epoch
"risks": { "ndpi_risk_score": 10, "risks": [46] } // bonus signal
} }
```
`flow_stats` (periodic per-flow counter updates):
```json
{ "type": "flow_stats", "interface": "wan", "internal": false,
"flow": { "digest": "…",
"local_bytes": 0, "other_bytes": 240, "total_bytes": 480,
"local_packets": 0, "other_packets": 2, "total_packets": 4,
"last_seen_at": 1787325070685 } }
```
**Correction (implementation ground truth, 2026-08-21):** `local_bytes` /
`other_bytes` are **per-update deltas** during a flow's life, but on the
final `flow_purge` event for that flow they are **always zero** (confirmed
on 10+ live purge events), while `total_bytes` is the flow's final
cumulative value. Since the purge event cannot supply a final directional
split, appflowd recovers it by distributing the `total_bytes` delta since the
last known total across up/down using the flow's established up/down ratio
(accumulated from that flow's prior `flow_stats` deltas), rather than
accounting `local_bytes`/`other_bytes` from the purge event itself.
### 2.3 Naming and taxonomy
- `definitions` event carries id→tag maps: 416 applications, 217 protocols.
Applications and protocols are SEPARATE id spaces that can collide numerically
(id 206 is both `WireGuard` the protocol and `netify.cloudflare` the app), and
legacy netify apps exist below 10000 (netflix=133), the `netify.` tag prefix,
not the id range, is the discriminator. Details: docs/netifyd-4.4.7-interface.md.
- Category **names** live in `/etc/netify.d/netify-categories.json`:
`application_tag_index` (32 tags: `streaming-media`, `social-media`, `games`,
`voip`, `cdn`, `advertiser`, `os-software-updates`, `malware`, …) and
`protocol_tag_index` (18 tags), plus `application_index` /
`protocol_index` mapping category-id → member ids.
- Display-name policy: application name if detected (strip `netify.` prefix,
title-case), else protocol name, else `Unknown`; hostname (`host_server_name`)
shown as secondary detail.
- **Licensing**: the signature DB (`netify-apps.conf`) and generated category
file are eGloo-proprietary data shipped by the `netifyd` package. appflow
reads them at runtime and **vendors none of this data** in the repo.
### 2.4 Additional confirmed facts (research pass, 2026-08-21)
- `flow_purge` events carry `reason: closed|expired|terminated` and the
flow's final cumulative `total_bytes`/`total_packets`, but
`local_bytes`/`other_bytes` are always zero on purge (confirmed live, see
§2.2); the final up/down split must be recovered from the accumulated
ratio, not read off the purge event, before flow removal.
- **Corrected:** `dump_established_flows` does NOT exist in netifyd 4.4.7 (string absent from the binary; a research citation of v5-era docs was wrong). On (re)connect a
client starts blind. The daemon
tolerates `flow_stats` for unknown digests (stub "Unknown" entry). **Corrected
2026-08-22:** that stub is backfilled only when a `flow` event actually follows,
which happens for a genuine stats-before-flow race but NOT for a flow that was
already established at connect time, netifyd never emits a `flow` event for
those, so they stay Unknown for their remaining lifetime. Measured in §2.5.
- App detections are `netify.<slug>` strings; protocol names are plain.
- OpenWrt ships netifyd **4.4.7** while upstream is v5.x with a different
plugin architecture, all behavior here is pinned to 4.4.7-as-packaged.
- netifyd bundles its own nDPI statically: `libndpi` is NOT a dependency.
### 2.5 Byte-attribution behavior (measured 2026-08-22)
Three questions about where bytes land were settled on the bench (EA8500,
netifyd 4.4.7). **Method matters here:** all earlier attempts were confounded by
upstream hosts throttling the test router's WAN IP under repeated fetches, and a
refused connection carries no SNI, so netifyd cannot classify it and it lands as
Unknown for reasons unrelated to this daemon. The measurements below therefore
use **LAN-local traffic only** (a file served from the router over `br-lan` via
`socat`, pulled by a LAN client with `curl --limit-rate`), so no upstream can
participate and throttling is impossible by construction.
- **Byte conservation through the Unknown path: HOLDS.** 25 short flows of 2 MB
= 50,000,000 bytes of client-side ground truth against 51,127,054 bytes
accounted (the excess is TCP/HTTP overhead, both directions). Repeated with
150 *tiny* 1 KB flows (the case most likely to complete inside one
`update_interval`, and so most likely to be purged before any `flow_stats`):
all 150 accounted, `no_total` = 0, nothing lost. A plausible reading of the
`fr.total > 0` gate in `flow_update()` predicts silent loss for such flows;
that prediction is **refuted** on this hardware, because netifyd supplies
accountable byte counts for sub-interval flows.
**CORRECTED 2026-08-25. The two paragraphs below understated this, and the
review panel was right.** Once `purge_only_rescued` was instrumented and left
running against ordinary traffic, it reported **13 purge-only flows out of
1,341 seen, 0.97%**. Flows purged without ever receiving a `flow_stats` DO
occur on netifyd 4.4.7. Before the defensive path was added, every one of them
lost its entire byte count silently, which is exactly what three carriers
predicted and what this section briefly called refuted.
The refutation was under-powered, not wrong-headed: a 38-flow capture at a
0.97% rate has an expected count of 0.4, so observing none said almost
nothing. Reading "0 of 38" as "netifyd never does this" was the error, and the
lesson is that a negative result needs its sample size checked against the
rate it claims to exclude. The event-level evidence below is still accurate
about what it saw; it simply did not see enough.
**Re-tested at the EVENT level 2026-08-24, after an adversarial review panel
independently reached the same prediction three times.** The 2026-08-22 result
above is outcome-based: it shows the bytes landed, not why. Tapping netifyd's
export socket directly for 170 s (`flow`/`flow_stats`/`flow_purge` per digest,
UDP DNS and short TCP both driven deliberately), **64 of 64 flows whose full
lifecycle was observed carried a `flow_stats` before their `flow_purge`.**
Zero went `flow` -> `flow_purge` directly. The premise the prediction rests on
does not occur on netifyd 4.4.7.
Two flows in that capture did show a purge with no preceding stats, and both
were flows that predated the tap: their birth was simply not observed. That is
§2.4 connect-blindness, not this. Mistaking one for the other is easy and is
why the analysis is stated at the level of *fully observed* lifecycles.
The gate is defended: `flow_update()` takes a `final` flag from `ev_purge()`
and attributes `total_bytes` for a purge-only flow, counted in
`stats.purge_only` and reported as `status.bytes.purge_only_rescued`. That
counter is **not** expected to stay 0 -- it runs at about 1% of flows -- and
it is the reason those bytes are now counted at all. Watch its rate rather
than its presence: a sharp rise means netifyd's event contract has shifted,
and a zero across a busy run means the rescue has regressed.
- **Connect/reconnect blindness: REAL, bounded, not fixable here.** A flow that
is already established when the daemon connects gets `flow_stats` but never a
`flow` event (§2.4), so its identity never arrives and its ongoing bytes
accumulate under Unknown until it ends. Forced deterministically by restarting
the daemon mid-transfer: 10.7 MB of a correctly-classified HTTP flow moved to
Unknown while the app's own total stayed flat, and a fresh socket client
observed 5 `flow_stats` and 0 `flow` events for that flow. Confined to the
window after a daemon restart or a netifyd restart, self-heals as flows cycle,
and the bytes are parked rather than lost. No fix exists on our side without
an established-flow dump from netifyd; see §8.
- **Late re-point stranding: OPEN, see §8.** Distinct from the above and not yet
observed in the wild.
Counters relevant to all three (`status.flows`) behaved consistently: `dropped`
stayed 0 and `late` stayed flat under sustained load, so the daemon is not
losing events under load; the Unknown mass seen during earlier testing was the
upstream confound plus the restart blindness above, not an event-handling defect.
## 3. Architecture
```
netifyd (DPI, procd service; nDPI engine)
│ unix stream, length-framed JSON events
▼
appflowd — ucode daemon (procd service)
• connects to netifyd.sock, reconnects with backoff
• parses frames, maintains flow table keyed by digest
• aggregates: totals + windowed rates per (application × device),
per category, per device; prunes idle flows
• serves snapshots over ubus ←— single writer, many readers
▼
rpcd ACL (usr/share/rpcd/acl.d) gates LuCI session access
▼
LuCI JS view (htdocs/…/view/appflow/*.js)
• L.poll every 3–5 s → ubus call → render
• hand-drawn inline SVG charts (theme-aware via CSS custom properties);
standard LuCI tables/widgets
```
Two implementation options for the ubus surface:
- **A (preferred): appflowd registers a ubus object directly** via
`ucode-mod-ubus` (`conn.publish()`); rpcd ACL still governs LuCI access.
- **B (fallback): snapshot file**, appflowd atomically writes
`/var/run/appflow/state.json`; a stateless rpcd-ucode plugin serves reads.
**DECIDED: Option A shipped**, `publish()` proven reliable on 25.12 (typed args,
status returns, int64 all verified on-device). Option B text retained above only as design rationale.
### 3.1 Why a persistent daemon (not an on-demand reader)
The socket is a **stream**: flows announce once, then only counter deltas
follow. Any on-demand reader would miss all context before it connected.
State must live in a long-running consumer; ucode + uloop keeps that consumer
tiny (no C, no external deps beyond packaged ucode modules).
### 3.2 Flow attribution and the dual-capture problem
netifyd watches `br-lan` (internal) and `wan` (external) simultaneously, so a
NAT'd client flow can be captured **twice** (pre- and post-NAT, different
digests, one per interface). The device a flow belongs to and whether it is a
duplicate must be decided without double-counting.
**The shipping rule (implemented in `flow_identify`).** An earlier design paired
the two captures on the remote endpoint and keyed the dedup on `ip_nat`; it was
replaced before v1 because that pairing was fragile (see §10). The rule that
ships is deterministic and needs no cross-flow pairing state:
- **Device side.** On an `internal` capture the device is the non-router
endpoint; on an `external` capture the device is the router-owned endpoint
(upstream infrastructure is the peer). `is_router()` decides which side is the
router, from the interface MAC/IP maps learned from `agent_status`.
- **Multicast/broadcast** endpoints collapse to a single `multicast`
pseudo-device (via netifyd's `other_type` when present, else the group-bit MAC
test), so they cannot mint device aggregates.
- **Single-count (shadow) rule.** An `internal` capture is always counted (it is
the device's own copy). An `external` capture is counted only when the device
is the router itself; an external capture of a *client* flow is a NAT twin and
is **shadowed** (tracked, never counted). This is gated on `saw_internal`, a
process-wide latch meaning "internal capture is active at all": on an
external-only netifyd (`-E` with no `-I`) nothing is shadowed, so every flow is
counted rather than dropped to zero.
**What this rule is and is not.** It is a *mode* test ("is internal capture
running"), not a *per-flow twin* test ("does this specific flow have an internal
twin"). The two differ only for flows that appear on the WAN side alone, and
`fr.nat` (`ip_nat`) is retained as a diagnostic field in the `flows` method but
drives no decision.
**FIXED 2026-08-24 by consulting conntrack. The history below is kept because
the mistakes are the instructive part.**
netifyd cannot answer this question, so appflowd now asks the kernel. For each
external capture whose device resolves to the router, `ct_nat_twin()` looks the
flow up in `/proc/net/nf_conntrack` on its reply tuple and compares the ORIGINAL
source against the reply destination: they differ for a masqueraded client and
match for router-originated traffic. Snapshots are reused for `CT_TTL_MS`, and a
lookup miss forces one refresh (floored at `CT_FORCE_MIN_MS`) because a flow is
normally younger than the current snapshot. If conntrack cannot be read at all
the code shadows on the safe default and says so once in the log, which keeps
client accounting correct at the cost of not attributing the router's own
traffic.
Measured after the fix, against wire ground truth from interface counters:
| | wire truth | attributed | |
|---|---|---|---|
| NAT-ed client | 21,323,454 | 21,287,298 | 99.8% |
| router's own | 6,290,088 | 6,157,849 | 97.9% |
with `conntrack: available true, twins 20, router_local 2, unresolved 2`. Both
are counted once. The fallback path was tested by pointing `CONNTRACK_FILE` at a
non-existent path: totals stayed at 97% rather than doubling, `unresolved` rose
to 16, and the warning fired.
**A defect found in the first version of this very fix**, worth recording because
it would have been invisible: caching the snapshot for 2 s meant new flows,
which are exactly the ones needing classification, usually missed it. It looked
like it worked (totals were right, because everything got shadowed) while
silently degrading to the no-conntrack behavior with conntrack fully available.
`unresolved: 20` against `router_local: 0` was the tell.
---
**The original defect, for the record. This used to double-count.** An earlier revision of
this section, committed the same day, claimed the opposite and said the twin
was correctly shadowed. That claim rested on a single measurement and was
wrong; the correction and how it happened are recorded at the end of this
section, because the mistake is more instructive than the result.
The deciding fact was what netifyd 4.4.7 puts in `local_mac`/`other_mac` when
`ip_nat` is true. Measured: **it carries the router's own WAN identity, not the
client's.** A NAT'd flow's external
capture reports
```json
{ "ip_nat": true, "local_ip": "192.168.72.10",
"local_mac": "c0:56:27:4e:3e:92", "local_origin": true,
"other_ip": "172.66.0.218" }
```
where `192.168.72.10` / `c0:56:27:4e:3e:92` are this router's WAN address and
MAC, while the internal capture of the same flow carries the real client
(`192.168.1.50`).
That makes `dev_is_router` **true** on the twin, and the rule reads
```
want_shadow = (saw_internal && !fr.internal && !dev_is_router && !group)
```
so `!dev_is_router` is false and the twin is **NOT shadowed**. It is counted a
second time, under the `router` pseudo-device, on top of the internal capture
already counted against the real client. The global total roughly doubles for
NAT'd client traffic.
Measured with a genuine NAT'd client: a veth pair into a network namespace
attached to `br-lan`, addressed `192.168.1.50/24` with a default route through
the router, so its traffic is really forwarded and really NATed. Ground truth
is the veth interface byte counter, which is **wire** bytes, the same quantity
appflow counts.
Per-device attribution of the client itself is essentially exact. 25 flows of
2 MB:
| | bytes |
|---|---|
| wire ground truth to/from the client | 53,145,843 |
| appflow, client device | 53,145,347 |
496 bytes of difference in 53 MB. The client's own row is trustworthy.
**The global total is not.** The same run credited the `router` pseudo-device a
further 51,785,387 bytes, the external twin of the very same traffic. Three
clean single-flow trials, each restarting `appflowd` first and letting the flow
purge:
| run | wire truth | appflow grand total | ratio |
|---|---|---|---|
| 1 | 10,479,105 | 17,089,074 | 163% |
| 2 | 10,478,155 | 20,639,013 | 196% |
| 3 | 10,478,879 | 20,877,799 | 199% |
> **The rest of this subsection is HISTORICAL and describes the state before
> the fix above. It is kept because the reasoning is what led to the fix, but
> nothing below is an open problem.** A review panel reading this document cold
> reported it as a live double-count, having entered mid-section, so if you are
> skimming, stop here and go to §3.3.
**Fix direction, at the time this was written, since implemented.** The `!dev_is_router` term exists for a
real reason: traffic the router genuinely originates belongs under `router` and
must not be shadowed. So it cannot simply be dropped. What is needed is a way
to tell a router-originated flow from a NAT'd client's twin, both of which
present the router's own MAC on the external capture. `ip_nat` looks like the
obvious discriminator and `fr.nat` already carries it, but it is **not
reliable**: a capture taken while generating both kinds of traffic reported
`ip_nat: false` for every flow including the NAT'd client's, even though
`ip_nat: true` was definitely observed on a router-WAN flow earlier the same
day. Any fix must first establish when netifyd actually sets that field.
**How the wrong conclusion got committed, because this is the useful part.**
The first measurement was a single 20 MB flow that reported the client at
20,950,037 bytes and `router` at 452. That looked decisive and was written up
as resolved. It was not reproducible: repeating the identical single-flow test
gave `router` 20,572,398 and a grand total of 167%. One measurement, no
repetition, and a result that happened to match the hoped-for branch. The same
failure this project has caught in reviewers repeatedly, committed by the
author, on the day it was caught elsewhere.
**Methodology warning for anyone repeating this.** An earlier run of the same
test reported 69% and then a spread of 46-101%, which would have been a serious
false finding. Both were the measurement, not the daemon: the first sampled
before the flow had purged and immediately after an `appflowd` restart (so it
also measured connect-blindness, §2.5), and the spread came from fragile shell
parsing picking the wrong record out of the JSON. Let `appflowd` settle before
the flow starts, let the flow purge before sampling, and read the numbers
directly rather than through `grep`.
**Related sub-case, now mitigated.** A pre-existing external flow captured only
as a stub (`flow_stats` before any `flow` event) is identified once, before
`saw_internal` latches, and would otherwise stay counted alongside its internal
twin for its whole life, inflating the central total. `reshadow()` re-applies
the shadow rule on later stats updates, closing that window to at most one stats
interval. The residual sign question for an unidentifiable stub (a genuine NAT
twin vs. pre-existing router-origin traffic, both landing in the "unknown"
bucket) folds into the same bench test above.
### 3.3 Memory bounds (runs on 128 MB devices)
Flow table capped (default 4096 flows, uci-tunable); LRU-pruned on purge
events, idle timeout, and cap pressure. No unbounded growth by construction,
but the real worst case for the aggregate tables is not small: each of
`AGG_MAX` (384) aggregates per class can carry up to `APP_DEV_MAX` (64)
per-device ring objects, a ceiling of 384 x 64 = 24,576 per-(app x device)
ring objects, plus up to `APP_HOST_MAX` (16) host ring entries per aggregate,
384 x 16 = 6,144 more. Observed RSS was approximately 2 MB at a light load of
33 apps and 11 devices (§8); the structural ceiling above is one to two
orders of magnitude higher than that observed point, not a number the box is
expected to reach under normal use, but worth stating plainly rather than
waved off as "naturally small".
**Every bound must be VISIBLE when it binds, and two of the three were not.**
A table that silently stops accepting data reports the same status as a table
with nothing to accept, which is the worst reading an operator can be given:
the dashboard looks healthy at exactly the moment it has become incomplete.
| bound | counter | added |
|---|---|---|
| aggregate table full of live entries | `aggregates.refused` | 2026-08-25 |
| flow table at `flow_max` | `flows.shed` | 2026-08-26 |
| unknown event-type map | (bounded at 16, not counted) | |
`flows.shed` counts evictions caused by **cap pressure**, separately from
`flows.pruned`, which is ordinary idle housekeeping and is expected to be
large. They shared one counter until 2026-08-26, so cap pressure was invisible
underneath normal behavior.
**`flows.dropped` is a genuine last-resort backstop and reads 0 in practice.**
`flow_new()` increments it only if the table is still full *after*
`prune(now, true)`, and that prune unconditionally sheds `int(flow_max/10) + 1`
entries, so it always frees space and the fall-through does not occur.
Measured by `tests/protocol-suite.sh`: 200 distinct digests against
`flow_max=64` gave `tracked=60 pruned=143 shed=143 dropped=0`. It is kept
because the guard is correct and cheap; it is not the number to watch.
The conventions this screen follows, which are the ones a DPI traffic screen
conventionally uses and which users of such screens already expect: a
**range-based** view (past hour, with day and week as later phases), a **15 s**
poll, table columns of Application / Download / Upload / Total sorted by total
descending, searchable, with a synthetic "All traffic" first row, a Top-10 bar
chart, and a per-app detail drawer carrying a time series plus per-client rows
(name, MAC, last seen, share %).
Classification here is **entirely local**: netifyd runs on the router, reads a
local socket, and nothing about the traffic leaves the device. Measured on the
development router, netifyd held 16 sockets over 4 days 23 hours of uptime, 12
unix and 3 packet-capture, and **zero TCP or UDP sockets**, with no outbound
connection at any point.
appflow v1 ships two tabs:
1. **Overview (live)**, the live view, which range-based DPI screens do not
generally provide. Live total
throughput chart, top apps by current rate, category breakdown, top
devices. Poll 5 s.
2. **Statistics (Past hour)**, the range view, from in-RAM buckets
(12 × 5-min per app): Top-10 chart, the Application/Download/Upload/Total
table (sorted, searchable, "All traffic" row), per-app detail drawer with
12-point time series + per-device rows (name, MAC, last seen, DL/UL,
share %), and a Clear button (`reset`). Poll 15 s.
**Past day / Past week tabs are phase 2** (require persistence; the UI shows
the tabs disabled with a tooltip, so the seam is visible rather than hidden).
Deliberate departures from the conventional layout, documented as choices:
- **No app icons in core, by design.** Vendor screens' per-app icons ship
from their CDN and are proprietary (netify/eGloo data). Core
`luci-app-appflow` renders letter-tile avatars colored by category on its
own, zero licensing exposure, still scannable. A separate, optional
package, `luci-app-appflow-icons` (§9), adds brand icons for the ~100 most
recognizable detectable apps; core probes for that package's icon
directory at render time and falls back cleanly when it is not installed.
- **No per-app Block toggle** (vendor screens wire it to a content-filter product;
out of scope for a statistics package).
- **No enable/disable toggle**, that gates a heavy NFQUEUE pipeline elsewhere; appflowd
is a lightweight socket consumer, service control via standard init.
- Device names resolved from `/tmp/dhcp.leases` + netifyd status.json MAC map
(vendor implementations use a proprietary client registry socket).
## 5. ubus contract (v1 draft)
Object `appflow`:
| method | args | returns |
|---|---|---|
| `summary` | - | totals, rates, top_apps[], top_categories[], top_devices[], window meta |
| `apps` | `{sort, limit}` | full per-app aggregate list |
| `devices` | - | per-device aggregates (+resolved names) |
| `app_detail` | `{app}` | per-device + per-hostname breakdown + hour time series for one app |
| `stats` | `{range:"hour"}` | range totals: all/applications[]/top_apps[] with 12×5-min time series (only `hour` in v1; other ranges rejected until phase 2) |
| `flows` | `{limit}` | live flow list (debug/power-user view) |
| `status` | - | daemon health: uptime, flows tracked, events/s, socket state |
| `reset` | - | zero all aggregates/buckets (the "Clear" function; **write-scope ACL**, separate from read grants) |
All byte values in **bytes**, rates in **bytes/sec** (UI formats). Timestamps
ms epoch. Every response carries `generated_at` + `agent_connected`.
## 6. Package layout
```
luci-app-appflow/
├── Makefile # LuCI feed style, PKGARCH:=all
├── htdocs/luci-static/resources/view/appflow/{overview,statistics,common}.js
├── root/etc/init.d/appflowd # procd wrapper
├── root/etc/config/appflow # uci: caps, window sizes, top-N
├── root/usr/sbin/appflowd # ucode daemon (shebang /usr/bin/ucode)
├── root/usr/share/rpcd/acl.d/luci-app-appflow.json
├── root/usr/share/luci/menu.d/luci-app-appflow.json
├── po/templates/appflow.pot # string catalog, GENERATED from source
├── tests/protocol-suite.sh # 54 checks, fake agent, no traffic needed
├── tests/hardware-suite.sh # 15 checks, real traffic vs wire truth
├── tests/frontend-suite.js # 31 checks, Node, runs in CI
├── tests/luci-module.js # loads a real LuCI view module under Node
└── docs/DESIGN.md # this file
```
`po/` is not hand-maintained. `luci.mk` discovers languages by globbing `po/*`
and generates a `luci-i18n-appflow-<lang>` subpackage per directory, so adding
a translation needs no Makefile change. See CONTRIBUTING.md.
Depends (from `Makefile` `LUCI_DEPENDS`): `netifyd`, `luci-base`,
`ucode-mod-socket`, `ucode-mod-uloop`, `ucode-mod-ubus`, `ucode-mod-uci`,
`ucode-mod-fs`.
## 7. History extension seams (deliberate, v1 ships them dormant)
1. **Single aggregation choke point**: every counter delta flows through one
`account(flow, delta_bytes, ts)` function, the only place history taps in.
2. **Time-bucketed accumulators**: aggregates are structured as
`{cur, buckets[]}` from day one, and the buckets are load-bearing in v1,
not a stub: every aggregate (totals, each app, each device, each
category) carries a 12-slot, 5-minute ring covering the trailing hour
(backs the `stats`/`app_detail` time series), and every
per-(app × device) and per-host breakdown carries a coarser 4-slot,
15-minute ring covering the same trailing window at a quarter of the
memory. History (§9) extends retention and adds persistence of that same
shape; it does not introduce the buckets themselves.
3. **Persistence interface**: `store.flush(buckets)` no-op stub in v1; later
implementations (RAM-rotating file / collectd exec / sqlite on USB) plug in
without touching accounting. Flash-wear rule: never write per-sample to
overlay; flush coarse buckets only.
4. **ubus contract reserves** `history` method name + `since/until` args.
## 8. Risks / open items
- ~~UI reference spec~~, landed 2026-08-21. It lived here as section 4 and
now lives in [frontend.md](frontend.md); the numbering below still skips
4 rather than renumber every cross-reference in this file.
- ~~netifyd conf details~~, landed: multi-client listen socket, `dump_established_flows`,
`flow_stats` cadence config. Purge event confirmed as `flow_purge` (extracted from the netifyd binary, 2026-08-21).
- ~~ucode `publish()` reliability → decides §3 option A/B~~: resolved, §3
states Option A shipped, proven reliable on 25.12.
- ~~Dual-capture dedup double-counts NAT'd client traffic~~, FIXED 2026-08-24
(§3.2) by resolving the ambiguity through conntrack rather than guessing.
Client 99.8% and router 97.9% of wire truth, each counted once. Degrades
safely when conntrack is unreadable. Original defect description follows.
- **(historical) The defect this replaced (§3.2).**
netifyd 4.4.7 reports the router's own WAN MAC on the external capture of a
NAT'd flow, so `dev_is_router` is true and `want_shadow` is false: the twin
is counted again under `router`. Measured 163%, 196% and 199% of wire ground
truth across three trials. The per-device row for the client is accurate to
496 bytes in 53 MB; it is the **grand total** and the `router` row that are
wrong. No fix yet: `!dev_is_router` protects genuinely router-originated
traffic and cannot simply be removed, and `ip_nat` is not a reliable
discriminator. See §3.2 for the fix direction.
- ~~**CT_MAX described as a memory bound**~~, FIXED 2026-08-24, also from the
panel. `ct_load()` called `fs.readfile()` and then `split(raw, "
")`, which
materialised the whole conntrack table and the whole line array *before*
`CT_MAX` was consulted, so the constant bounded the retained map but not peak
allocation, and its own comment calling it a memory bound was wrong. It now
reads a line at a time via `fs.open()` / `read("line")`, so peak is one line
and the cap stops the work as well as the retention.
A second defect in the same function: the non-forced throttle read
`conntrack.entries && (now - conntrack.at) < CT_TTL_MS`. A snapshot that
parsed to zero rows while the file was readable made that falsy and disabled
the throttle entirely, and the non-forced path has no `CT_FORCE_MIN_MS` floor,
so every external flow identification re-read the whole file. Now gated on
`conntrack.at`, i.e. on having read recently rather than on having read
recently *and* found something.
- **Byte conservation is now instrumented, not just asserted** (2026-08-24,
suggested by a panel juror). `status.bytes` reports `reported` (what netifyd
said moved, before any clamp or shadow decision), `attributed` (what reached
an aggregate), `shadowed` (what was deliberately dropped as a NAT twin) and
`leaked` (the residual, which should be 0). Both silent byte-loss defects this
daemon has had would have been visible here from the first minute. Measured 0
across every run since. In a NAT-ed transfer `reported` lands at roughly twice
wire truth with about half shadowed, which is the dual capture being
deduplicated, visible for the first time.
- EA8500 resource baseline MEASURED (2026-08-21, light load, 12 flows,
synthetic traffic): netifyd ~16.5 MB VSZ, ~0% CPU, box 95% idle, 397 MB
free. Heavy-load measurement (real client routed through) still pending.
- ~~Byte conservation through the Unknown/late-stub path~~, VERIFIED sound
2026-08-22 (§2.5), including the sub-`update_interval` case.
- **Connect/reconnect blindness (KNOWN LIMITATION, will ship).** Flows already
established when the daemon or netifyd (re)starts have their remaining bytes
attributed to Unknown, because netifyd 4.4.7 emits no `flow` event for them
(§2.4, §2.5). Bounded to that window, self-healing, non-destructive. Cannot be
fixed without an established-flow dump upstream. Documented in the
README rather than hidden.
- ~~**reshadow() exempted the flows it existed to correct**~~, FOUND by an
adversarial panel and FIXED 2026-08-24. `reshadow()` returned early on
`dev_key == "router"`, which is exactly how a WAN-side capture of a NAT-ed
client flow is classified, because netifyd reports the post-translation
identity (§3.2). The conntrack twin resolution added earlier that day went
into `flow_identify()` and was never wired into `reshadow()`, so the one class
of flow it must correct was the one it skipped. It now calls `ct_nat_twin()`
and defaults the same way `flow_identify()` does.
Measured on hardware, NAT-ed client transfer with `appflowd` restarted
mid-flight, appflow total against the client veth's own tx counter:
| run | before fix | after fix |
|---|---|---|
| 1 | 0.701 | 0.892 |
| 2 | 0.791 | 1.013 |
No-restart control after the fix: 1.012. **A caveat worth keeping:** the panel
predicted a persistent *double* count of the central total. That is not what
was measured. The old code *under*-counted after a restart (0.70-0.79) and the
fix restores accuracy; it does not remove an inflation that was never
observed. The code defect was real, the predicted magnitude and sign were not.
The residual gap in run 1 is connect-blindness, below.
- **Late re-point stranding (OPEN, unquantified).** If netifyd first reports a
flow unclassified (`app_id <= 0`) and only later re-points it via a detection
update, bytes counted before the re-point stay under Unknown, because the
accounting seam is append-only by design (see the re-point comment in
`classify()`). No counter currently exposes this, so its real-world magnitude
is unknown. It could not be induced deliberately on the bench (it needs a
protocol netifyd classifies only after several packets on a stable
connection). Cheapest settling test: a real-traffic socket capture correlating
per-digest `flow`/`flow_stats`/detection-update ordering, excluding
connect-time orphans by `first_seen_at`; alternatively instrument the re-point
in `classify()` to accumulate stranded mass into a `status` counter. Deferred
rather than guessed at.
## 9. Roadmap (queued, post-v1)
1. **v1.1, `luci-app-appflow-icons`, a SEPARATE package.** Hand-curated mapping
(netify app id → icon) for the ~100 most-recognizable detectable apps, using
monochrome SVGs from CC0/Apache-licensed collections (simple-icons /
dashboard-icons), bundled locally, tinted by category color. Deliberately
split from core, i18n-style, for legal blast-radius isolation: brand icons
are trademarks regardless of collection license, so if anyone ever objects
the icon package is delisted on its own and core is untouched. Coupling is
runtime-soft in BOTH directions: core probes the icon directory at render
time and falls back to letter-tile avatars; the icon package depends on
nothing and breaks nothing when absent or removed. Same repo, second
package directory.
2. **v2, Past day / Past week ranges** (persistence per §7 seams; flash-wear
policy: coarse-bucket flushes only, tmpfs-first with periodic backup,
tiered downsampling).
## 10. As-implemented contract notes (v1 final)
§5 sketched the method set; the implemented field naming is authoritative:
live aggregate rows use `bytes_up/bytes_down/bytes_total` +
`rate_up/rate_down/rate_total` (+ `key/name/tag/category`), while the
The `stats`/`app_detail` range views use `download/upload/total`, which is the
conventional vocabulary for that screen. `status` is nested
(`socket.*`, `flows.*`, `aggregates.*`, `agent.*`, `memory.*`).
`app_detail` takes the aggregate `key` or the raw tag or the display
label (resolver tries all three). The frontend ships normalizers for
both dialects in `view/appflow/common.js`. luci-lib-chartjs was dropped:
the packaged build is a Chart.js 1.x fragment (Doughnut/Pie only), and
inline SVG follows LuCI theme variables, which canvas cannot.
## 11. AI service breakout
### 11.1 Why netifyd cannot do this
Measured on two routers, 2026-08-27. `/etc/netify.d/netify-apps.conf` is the
Netify Agent's signature set. On both it is dated **10 August 2023**, carries
**199** application signatures, and contains **zero** entries matching
anthropic, claude, openai, chatgpt, huggingface, perplexity, mistral, copilot,
bard or gemini.
So AI traffic arrives with `detected_application` 0 and aggregates into generic
HTTP/S alongside everything else. This is vendor data three years stale, not a
configuration problem, and nothing on the router can change it.
### 11.2 Why appflow can
netifyd already reports the TLS SNI as `host_server_name`, and appflow already
stores it. Traffic generated to the real endpoints on the bench carried
`api.anthropic.com`, `claude.ai` and `api.openai.com` with no code change at
all. The identifying data was there the whole time; nothing was making use of
it.
The daemon is the right layer because aggregation happens there. The browser
only ever receives summarized rows, so a frontend fix would arrive after the
decision that matters had already been made.
### 11.3 What it does
In `flow_identify()`, after netifyd's identity is resolved and after the
category is derived, the SNI is matched by **domain suffix** against a built-in
table. On a hit the application key, label, tag and category are replaced.
```
key ai:anthropic-claude
name Anthropic (Claude)
tag appflow.anthropic-claude
cat ai-assistants
```
Four categories: `ai-assistants`, `ai-media`, `ai-developer`,
`ai-infrastructure`. The last covers aggregators, inference APIs, GPU rental
and vector stores, which are different businesses but one thing from a
network's point of view.
**The category is a SLUG on the wire, and 1.1.0 got this wrong.** It shipped
`AI assistants` and its three siblings as display strings, which read correctly
in English and could therefore never be translated: the browser renders a
category by looking the slug up in a table of `_()` literals, and a string that
arrives already rendered has no key to look up. Every netifyd category travels
as a lowercase slug; these four were the only ones that did not, and they were
ours. Corrected in 1.1.1. `frontend-suite.js` now reads this table out of the
daemon source and fails if a category is emitted that the frontend cannot
translate, so the two halves cannot drift apart again.
**Keyed on the vendor label, not the matched domain.** `anthropic.com` and
`claude.ai` are one vendor and must aggregate into one row; keying on the
suffix would produce two identically-named rows and split the total.
**The `appflow.` tag prefix is deliberate.** netifyd's own tags are `netify.*`.
Nothing here pretends to have come from netifyd.
### 11.4 The gate, and why it makes the feature self-retiring
**netifyd's own answer wins.** The override runs only when
`detected_application` is 0, with one exception: entries flagged `strong`,
which override even a positive identification.
`strong` is set only where netifyd is known to recognize the PARENT brand and
would otherwise swallow the AI service inside it: Gemini reported as Google,
Copilot as GitHub, Qwen as Alibaba. Everywhere else the rule is conservative,
which means **the day Netify ship AI signatures this table falls silent by
itself** rather than permanently shadowing better data.
### 11.5 Matching is anchored on label boundaries
The SNI is lowercased, a trailing dot is stripped, and the last two, three and
four labels are rebuilt and looked up in turn. Never a substring test.
`anthropic.com.attacker.example` is a hostname anyone can register and point
anywhere. A substring match on `anthropic.com` matches it. Rebuilding the last
N labels makes that impossible by construction rather than by care, and the
suite asserts it with a flow carrying exactly that hostname.
### 11.6 Accounting is untouched
This changes an identity, never a byte. `account()` never sees it, byte
conservation is exactly as it was, and the existing conservation checks still
hold. `tests/protocol-suite.sh ai` asserts a hand-computed total across a batch
containing AI flows for precisely this reason.
### 11.7 Limitations, stated rather than discovered
- **This is hostname matching, not DPI.** It asserts a classification netifyd
did not make. The technique is legitimate, matching the SNI is what netifyd
itself does for HTTP/S, but the accurate description is "we recognize the
name", never "we inspected the protocol".
- **The list rots.** New services appear constantly and nothing here updates
itself. `option ai_breakout '0'` turns it off entirely.
- **ECH would end it.** Encrypted Client Hello encrypts the SNI. netifyd has
the identical exposure so this is no worse than the status quo, but it is not
durable and should not be sold as such.
- **Region-encoded hostnames under a generic cloud domain cannot be matched.**
`bedrock-runtime.us-east-1.amazonaws.com` has no suffix specific to Bedrock,
and matching `amazonaws.com` would label every S3 bucket on the network as
AI. Deliberately absent rather than approximated.
- **A service behind a shared CDN hostname is invisible**, because the SNI
names the CDN.
- **No icons ship for these entries.** They render as letter tiles until the
icon pack gains matching `appflow.*` keys.
## 12. Extensibility: the hostname table is not a constant
The AI breakout (section 11) shipped its table as a constant in the daemon,
which meant adding a service required editing the source and rebuilding the
package. For a tool whose central claim is that netifyd's signature set is three
years stale, being un-extendable is the wrong shape, and the question that
exposed it was the obvious one: "I have a service I want measured, what do I
do?"
`config hostmap` sections in `/etc/config/appflow` are read at config load and
consulted BEFORE the built-in table, so one mechanism does both jobs: adding a
service that is missing, and correcting one that is wrong.
```
config hostmap
option suffix 'torproject.org'
option name 'Tor'
option category 'Privacy'
option strong '0'
```
**A malformed entry is refused and named in the log.** A suffix with no dot can
never match, because the matcher rebuilds whole labels rather than comparing
substrings, so accepting one produces a rule that silently does nothing and
looks identical to the feature being broken.
That guard has no effect on classification, only on whether the user is told,
and the test for it had to be written accordingly. Two earlier versions were
tautologies: one grepped the whole log and passed forever once the message had
ever appeared, the next counted log lines and read past the count, which failed
because `logread` is a RING whose line count stops growing once it is full. A
unique marker written with `logger` is immune to both. See
`tests/protocol-suite.sh hostmap`.
## 13. Shipped in 1.1.2: the Router row had never counted a byte
**The most serious defect this package has had, and it was invisible by
construction.** On both routers checked, the `router` pseudo-device reported
exactly zero bytes. Production had logged **107,476** flows where conntrack
positively confirmed the router originated the traffic, against a device row
reading zero for 2.6 days and 52 GB. The dashboard correctly hides a zero-byte
device, so the feature was absent rather than visibly broken, and nobody
noticed for the life of the package.
**The cause is one line and it is worth stating precisely.** `ct_nat_twin()`
built its lookup key from the event it was handed:
other_ip | local_ip | other_port | local_port
netifyd puts the ports on the initial `flow` event and on nothing afterwards. A
`flow_stats` payload carries addresses and byte counters only. So the key built
from an update had two empty fields, matched nothing, and returned `null`.
`reshadow()` treats `null` as grounds to discard, so every router-originated
flow was shadowed on its first stats update, before a single byte reached the
aggregate. `flow_identify()` resolved it correctly, once, and the next event
threw the answer away.
The fix has two halves, and they are independently sufficient, which is why the
tests distinguish them:
- **The ports are stored on the flow record** and the lookup prefers them. This
is the root cause and the check on `conntrack.unresolved` isolates it.
- **A positive finding is remembered.** `fr.ct_local` records that conntrack
once said the router originated this flow, and a later lookup that cannot
answer no longer overturns it. Only a positive "this is a NAT twin" may.
Absent evidence is not positive evidence, the same rule §5 already states
about `statusStrip()`.
**The dedup is deliberately unchanged.** A fix that simply stopped shadowing
would restore the 163-199% over-count this mechanism exists to prevent, so the
suite asserts that a router flow conntrack cannot confirm is still discarded.
`conntrack_path` became a config option purely so this is testable, exactly as
`socket_path` already was. The `router` group in `protocol-suite.sh` supplies a
conntrack fixture and drives the real daemon.
**Also in 1.1.2: tile letters outside ASCII.** `tile()` stripped with
`/^[^0-9A-Za-z]+/`, which consumes a name written in any non-Latin script
entirely and falls through to `?`. Every Chinese, Japanese, Korean, Russian,
Greek, Arabic and Hebrew name rendered an identical question mark. Not a
translation bug: any LAN client with a non-Latin DHCP hostname hit it on every
release. Now stripped on `\p{L}\p{N}` with the `/u` flag.
## 14. Shipped in 1.1.1: labels that could not be translated
Every item that stood in this section as "queued for 1.1.1" shipped in 1.1.0
instead (commit `b8ca31d`): the search placeholder, exclude terms, and the
clickable donut. The section was left describing them as pending for three
weeks. What actually became 1.1.1 is below.
**A label is only translatable if a literal reaches `_()`.** That is what the
`.pot` extractor scans for, and three separate paths produced labels that never
did. All three were invisible to a green test run, and all three were reported,
directly or indirectly, by a first-time contributor's translation pull request.
**`catLabel()` held 12 entries and title-cased the rest at runtime.** That is
string manipulation, not translation: `web` became `Web` by algorithm, in
English, in every language. The live netifyd category file carries 43 tags
across its application and protocol indexes. The table now holds all of them.
**The daemon names its three synthetic pseudo-devices in English**, because a
daemon has no idea what language a browser wants. The frontend now maps the
stable key it sends, not the name, so a daemon-side rewording cannot silently
revert the translation.
**The AI categories were ours, not inherited.** See section 12.
The interesting part is why no test caught it. `frontend-suite.js` asserted
that `catLabel('networking')` returns `'Networking'`, which is satisfied
identically by a `_()` lookup and by a title-caser, and it asserted that every
string reaching `_()` is in the catalogue, which is blind by construction to a
string that never reaches `_()`. The new group asserts **membership in the set
of strings the module actually passed to `_()`**, recorded by the harness as it
loads. Six product edits were executed against it to confirm each check can
fail; two of those runs found defects in the test rather than the product.
## 15. Filtering and the drill-down
### 14.1 Where the logic lives, and why
`parseFilter()` and `matchFilter()` are in `common.js`, not inside either
view's render function. Both views share them so the two cannot drift, and Node
can load `common.js`, so the filter is covered by `tests/frontend-suite.js`
rather than by looking at a browser. The sibling package learned the same
lesson the same week: logic that decides something belongs where a test can
reach it.
Terms are space separated and every one must pass, so terms narrow. A leading
minus excludes. Exclusion is not decoration: the question that produced this
feature was "one streaming session is burying everything else", and an
include-only filter cannot express it, because you would have to already know
what you wanted to keep.
A bare `-` is dropped rather than treated as exclude-all, because that is what
a half-typed term looks like and blanking the table between two keystrokes is
not acceptable.
### 14.2 Fetching past the top-N cut
The overview asks the daemon for `FETCH_LIMIT` (50) application rows and
displays `TOP_APPS` (8). The extra rows are the filter's raw material. Without
them, filtering a top-10 list to "AI" returns nothing the moment a streaming
session pushes AI out of the top 10, which looks broken and is worse than
having no filter at all.
### 14.3 The drill-down needs no new aggregate
Both directions are one pass over the live flow table:
devices_for_apps(keys) which devices carry these applications
apps_for_device(dev) which applications is this device using
Every flow record already carries `dev_key`, `app_key` and `cat_key`, so the
join exists on every flow, and the table is already bounded by `flow_max`. No
new storage.
**This was first answered wrongly.** The initial response was that it required
a per-device-per-application aggregate and was therefore a bounded-memory
design question. Checking the flow record disproved that in two minutes. The
episode is recorded because the shape of the mistake matters more than the
mistake: an architectural-sounding objection offered instead of a two-minute
check.
### 14.4 Attributed bytes, not lifetime counters
Both functions sum `fr.up + fr.down`, the bytes appflow attributed, and never
`fr.total`, which is netifyd's lifetime counter for the connection. A stub flow
adopted mid-transfer has a large lifetime and small attributed bytes, so the
two differ exactly where it matters.
The test fixture carries one flow with a lifetime of 9000 against 1000
attributed, for one reason: without it every flow had the two equal, summing
the wrong field was invisible, and the sabotage for that check passed against
broken code.
### 14.5 Three bugs of one shape
Getting this working produced three defects that presented identically: wired
up, no error, no effect.
- **`null` for a string ubus argument.** ubus rejects it with code 2 rather
than treating it as absent, and `L.resolveDefault` swallowed the rejection.
- **The poll repainted over the filtered result.** A side call filtered the
device card and the next five-second poll overwrote it. Filtering is view
state, so every fetch carries it rather than one special path.
- **`normDevice` dropped the daemon's `key`.** Clicking a device passed
`undefined`, which *cleared* the selection instead of setting it.
None of these are catchable by a test of logic, because in each case the logic
was right and the wire was not connected. They were found by driving the page.
That is the limit of this package's test strategy and it is written here
rather than left implicit.
### 14.6 One ACL grant was widened, deliberately
`apps` was unbound in the frontend and ungranted in the ACL after an August
security review. The device drill-down needs it, so it was bound and granted,
in that order, exactly as that review's own note required.
`apps {device}` returns an aggregate per device and application from the live
flow table. It is more revealing than either list alone, since it says which
device used which application, and it is still far short of `flows`: no remote
address, no port, no per-connection record. **`flows` remains unbound and
ungranted.**
luci-app-appflow/docs/ai-filter.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 117 KiB

luci-app-appflow/docs/frontend.md
+11
View File
@@ -0,0 +1,11 @@
# AppFlow frontend
- `common.js`: shared class (`require view.appflow.common as appflow`); ubus bindings, payload normalizers, byte/rate formatting, category colors, the icon probe, SVG chart primitives, injected CSS.
- `overview.js`: Overview (live). Polls `summary`+`devices`+`status` every 5 s; the 60-sample chart window lives in the view, not the daemon.
- `statistics.js`: Statistics (past hour). Polls `stats {range:"hour"}` every 15 s; search, sort and the detail drawer are client-side; `reset` sits behind a confirm dialog.
- Menu: `root/usr/share/luci/menu.d/luci-app-appflow.json`, top-level `admin/appflow` firstchild node gated on the `luci-app-appflow` ACL group.
Conventions: charts are hand-drawn SVG (`lineChart`/`columnChart`/`donut`) so they inherit the theme's CSS variables and read on both bootstrap themes; `luci-lib-chartjs` is unused (its shipped build ships Doughnut/Pie only, no Line/Bar). All CSS is prefixed `af-` and injected once via `appflow.injectCSS()`.
Every payload field is read through `normApp`/`normDevice`/`normTotals`/`normSeries`, which absorb the daemon's two dialects (`bytes_down`/`rate_down` on live aggregates, `download`/`upload` on range views); tighten those alias lists once the DESIGN §5 contract freezes instead of reaching into payloads from the views.
Icons are an optional soft dependency (DESIGN §9): `loadIcons()` probes `/luci-static/resources/appflow-icons/manifest.json` once per page load from each view's `load()`, keyed by the row's netify `tag`; if the sibling package is absent the probe resolves silently and every tile stays a letter tile. `simple-icons` entries are fill-less glyphs and get whitened over the category-colored tile; other sources render as-is. Category colors come from the fixed `CATEGORY_SLOT` table (hash fallback) so the doughnut legend never shows two identical swatches.
Both views build their skeleton once and repaint only inner nodes, so the search box keeps focus and caret across polls; each keeps a `fails` counter so one dropped poll does not flash the "appflowd is not running" screen.
luci-app-appflow/docs/netifyd-4.4.7-interface.md
+624
View File
@@ -0,0 +1,624 @@
# The netifyd 4.4.7 JSON export interface, as shipped by OpenWrt
## Scope
This document describes the JSON socket interface exposed by **netifyd 4.4.7**,
the exact version OpenWrt packages and installs (`netifyd-4.4.7-r2.apk` at the
time of writing). It does **not** describe the current upstream Netify Agent
(v5.x), which has moved to a plugin (`plugins.d`) architecture with a
different sink model, additional fields (`digest_prev`, `risks.*` extensions,
`app_ip_override`), and options that do not exist in 4.4.7. Netify's own
public documentation site currently documents v5, which is a meaningful
source of confusion if you go looking for confirmation of anything here: most
of what you'll find online describes a daemon that is not the one running on
your router.
Everything stated as fact below is grounded in one of two sources:
1. **Direct inspection of the 4.4.7 source and OpenWrt packaging**
(`gitlab.com/netify.ai/public/netify-agent` at tag `v4.4.7`, and the
`openwrt/packages` build recipe).
2. **A real captured stream from a running 4.4.7 instance** on a Linksys
EA8500 (OpenWrt 25.12.5, `arm_cortex-a15_neon-vfpv4`), taken over a UNIX
socket connection immediately following daemon start. Excerpts from that
capture are reproduced verbatim throughout this document (only field
*values* like MAC/IP addresses and timestamps are real lab data, not
synthetic).
Where the two disagree, or where a claim could only be confirmed from source
and not observed live, this is called out explicitly. An "Unverified /
unknown" section at the end collects every open question directly rather
than papering over it.
**A note on `dump_established_flows`.** You may see this option referenced in
blog posts, forum threads, or even in comments in older netifyd
configuration examples. It does not exist in 4.4.7: the string is entirely
absent from the compiled binary and from the 4.4.7 man page source. It is a
v5-era `plugins.d` option. Practically, this means a client that connects to
the socket mid-stream (which, for most consumers, is every connection) gets
**no backfill of already-tracked flows**. It starts blind. See "Connection
lifecycle" and "Practical consumer guidance" below for what this means and
how to handle it; the real capture used throughout this document directly
demonstrates the resulting behavior.
## 1. Socket and wire framing
netifyd exposes a UNIX domain socket, configured in `/etc/netifyd.conf`:
```ini
[socket]
listen_path[0] = /var/run/netifyd/netifyd.sock
```
On stock OpenWrt this is active by default as soon as the `netifyd` init
script starts the daemon; no additional configuration is required to get a
live stream. (A TCP listener is also supported via `listen_address[n]` /
`listen_port[n]`, but is not used by default on OpenWrt and was not
exercised for this document.)
**Wire format**: every message is a JSON object terminated by `\n`, and every
payload line is preceded by a one-line header of the form `{"length": N}`.
This is newline-delimited JSON with an explicit length prefix, not bare NDJSON.
Real captured pair (the very first two lines netifyd writes to a freshly
connected socket):
```
{"length": 165}
{"agent_version":"4.4.7","build_version":"Netify Agent/4.4.7 (openwrt; arm; conntrack; netlink; dns-cache; plugins; regex)","json_version":1.9,"type":"agent_hello"}
```
**Framing detail confirmed by direct measurement**: `N` is the byte length of
the payload line **including its trailing `\n`**, not the length of the raw
JSON content alone. Checked across all 17 header/payload pairs in the capture
used for this document, the declared `length` matched `byteLength(payload) + 1`
in every case, and never matched `byteLength(payload)` alone. A parser must
account for the newline when deciding how many bytes to read for a given
frame (or, more robustly, simply read line-delimited text and treat the
length header as a frame-boundary hint rather than the sole read-size
authority).
Parser rule, in short: read a line; if its only key is `length`, it is a
frame header, discard it; otherwise the line is a complete JSON event and its
`type` field determines how to interpret it.
## 2. Connection lifecycle
The real capture shows this exact sequence on connect (daemon `update_interval`
was 15 seconds for this run):
1. `agent_hello` -- sent once, immediately, before anything else.
2. `agent_status` -- sent immediately after `agent_hello`. This first status
frame is a **reduced** one; see the field-presence note in section 3.3.
3. `definitions` -- sent once per connection, a large one-time dump of the
application and protocol id-to-name tables (416 applications / 217
protocols in this capture).
4. A burst of `flow_stats` events (11 of them in this capture) for flows that
were **already being tracked** before this client connected. **None of
these digests were ever announced via a `flow` event on this connection.**
This is the live, directly observable consequence of
`dump_established_flows` not existing in 4.4.7: the client is receiving
counter updates for flows whose identity (application, protocol, hostname,
IPs) it was never told.
5. A second `agent_status`, 15 seconds after the first (matching
`update_interval`), this time carrying the fuller field set (interfaces,
devices, per-interface capture stats).
6. Two `flow` events, for newly detected DNS queries to `openwrt.org`,
showing the daemon has now genuinely observed new traffic from
first packet.
No `flow_purge` event occurred during this capture window (the two new flows
were still active when the capture ended). Its shape is documented in
section 3.6, confirmed live via 10+ purge events captured separately during
this project's hardware testing (`DESIGN.md`), even though none happened to
land in the specific capture excerpted in this document.
Practical implication for a long-running consumer (this is what
`appflowd` in this repo does): on every connect or reconnect, assume you know
nothing about flows already in progress. You will start receiving
`flow_stats` (and eventually `flow_purge`) for digests you have never seen a
`flow` event for. Treat that as the normal, permanent case, not a transient
startup race.
## 3. Event type reference
Every event on the socket, except `agent_hello` and `definitions`, shares the
same top-level envelope:
```json
{"type": "<event-type>", "interface": "<ifname>", "internal": <bool>, "established": <bool>, "flow": { ... }}
```
`interface` is the capturing interface name (`br-lan`, `wan`, etc.). `internal`
reflects which capture point saw the packet (true for internal/LAN-side
capture, false for external/WAN-side), which matters because netifyd watches
both `br-lan` and `wan` simultaneously and a NAT'd client's traffic can appear
as two distinct flows with two distinct digests (see `DESIGN.md` section 3.2
for this project's dedup policy; that is an application-level concern, not
part of the wire protocol itself).
`established` is present on every `flow`/`flow_stats` event in the capture
and was `false` in all observed instances. Its precise semantics were not
confirmed from source in the research this document is based on, and the
capture is too short and too uniform to infer it empirically; see
"Unverified / unknown."
### 3.1 `agent_hello`
Sent once, immediately on connect, before any other event.
```json
{"agent_version":"4.4.7","build_version":"Netify Agent/4.4.7 (openwrt; arm; conntrack; netlink; dns-cache; plugins; regex)","json_version":1.9,"type":"agent_hello"}
```
| Field | Type | Notes |
|---|---|---|
| `agent_version` | string | e.g. `"4.4.7"` |
| `build_version` | string | Human-readable build string; the parenthesized list is compile-time capability flags. This build has `conntrack` compiled in (relevant to `ct_id`/`ct_mark`, see 3.4). |
| `json_version` | number | Schema/protocol version for the JSON payloads on this socket, `1.9` in this capture. Matches the `version` field later seen in `agent_status`. |
| `type` | string | Always `"agent_hello"` |
### 3.2 `definitions`
Sent once per connection, immediately after the first `agent_status`. Carries
the full id-to-name tables for both id spaces used elsewhere in the stream
(`detected_protocol` and `detected_application`). This is a large message
(18,704 bytes in this capture, framed with the `{"length": N}` header like
everything else).
Truncated real excerpt (full arrays omitted for length; see section 5 for a
detailed discussion of what's in them):
```json
{"applications":[{"id":248,"tag":"ZOOM"},{"id":177,"tag":"ZMQ"}, ... ,{"id":10126,"tag":"netify.zendesk"}],
"protocols":[{"id":248,"tag":"ZOOM"},{"id":177,"tag":"ZMQ"}, ... ,{"id":191,"tag":"OOKLA"}],
"type":"definitions"}
```
| Field | Type | Notes |
|---|---|---|
| `applications` | array of `{id: uint, tag: string}` | 416 entries in this capture. Resolves `flow.detected_application`. **Do not** also use this array to resolve `detected_protocol` -- see the id-collision warning in section 5. |
| `protocols` | array of `{id: uint, tag: string}` | 217 entries in this capture. Resolves `flow.detected_protocol` only. |
| `type` | string | Always `"definitions"` |
The captured counts (416 / 217) match `DESIGN.md`'s figures exactly, which is
good independent corroboration that this capture is representative and not a
truncated or unusual snapshot.
### 3.3 `agent_status`
Agent-level telemetry: uptime, CPU, memory, flow-table size, sink (upload)
state, and (on richer frames) per-interface identity and packet-capture
counters. **Never contains per-flow data.** This is the same payload also
written periodically to `/var/run/netifyd/status.json`; on the socket it is
additionally interleaved with flow events, once per `update_interval`.
**Two different shapes were observed for this event type in the same
capture, and this is worth flagging explicitly because it is easy to design
a parser around only the richer one and then break on the first frame.**
First `agent_status` (sent immediately after `agent_hello`, before
`definitions`):
```json
{"cpu_cores":2,"cpu_system":0.075803,"cpu_system_prev":0.028308,"cpu_user":1.212848,"cpu_user_prev":0.113235,"dhc_size":1,"dhc_status":true,"flow_count":16,"flow_count_prev":0,"maxrss_kb":15372,"maxrss_kb_prev":9332,"sink_status":false,"sink_uploads":false,"timestamp":1787325073,"type":"agent_status","update_imf":1,"update_interval":15,"uptime":15,"version":1.9}
```
Second `agent_status` (15 seconds later, on the regular cadence):
```json
{"agent_version":"4.4.7","cpu_cores":2,"cpu_system":0.077808,"cpu_system_prev":0.075803,"cpu_user":1.256057,"cpu_user_prev":1.212848,"devices":{"b4:2e:99:3a:d6:e4":["192.168.1.10"],"c0:56:27:4e:3e:92":["192.168.72.10"],"c0:56:27:4e:3e:93":["192.168.1.1"],"dc:b5:4f:ad:0d:6e":["192.168.72.21","fe80::18f3:bf8d:dc6e:980f"]},"dhc_size":1,"dhc_status":true,"flow_count":21,"flow_count_prev":16,"interfaces":{"br-lan":{"addr":["fe80::c256:27ff:fe4e:3e93","fda8:3a0e:1478::1","192.168.1.1"],"mac":"c0:56:27:4e:3e:93","role":"LAN","state":1},"wan":{"addr":["fe80::c256:27ff:fe4e:3e92","192.168.72.10"],"mac":"c0:56:27:4e:3e:92","role":"WAN","state":1}},"maxrss_kb":15372,"maxrss_kb_prev":15372,"sink_status":false,"sink_uploads":false,"stats":{"br-lan":{"capture_dropped":0,"capture_filtered":0,"discarded":0,"discarded_bytes":0,"ethernet":35,"fragmented":0,"icmp":0,"igmp":0,"ip":35,"ip_bytes":10806,"largest_bytes":1718,"mpls":0,"pcap_drop":0,"pcap_ifdrop":0,"pcap_recv":35,"pppoe":0,"queue_dropped":0,"raw":35,"tcp":35,"tcp_resets":0,"tcp_seq_error":0,"udp":0,"vlan":0,"wire_bytes":11646},"wan":{"capture_dropped":0,"capture_filtered":0,"discarded":2,"discarded_bytes":102,"ethernet":158,"fragmented":0,"icmp":2,"igmp":0,"ip":156,"ip_bytes":149662,"largest_bytes":4338,"mpls":0,"pcap_drop":0,"pcap_ifdrop":0,"pcap_recv":158,"pppoe":0,"queue_dropped":0,"raw":158,"tcp":134,"tcp_resets":2,"tcp_seq_error":0,"udp":20,"vlan":0,"wire_bytes":153406}},"timestamp":1787325074,"type":"agent_status","update_imf":1,"update_interval":15,"uptime":30,"version":1.9}
```
| Field | Type | Present on | Notes |
|---|---|---|---|
| `type` | string | both | Always `"agent_status"` |
| `version` | number | both | JSON schema version, `1.9` here (same value as `agent_hello.json_version`) |
| `timestamp` | uint | both | Epoch **seconds**. See the timing anomaly noted below; don't lean on this for interval math. |
| `uptime` | uint | both | Seconds since agent start. Reliable for computing elapsed time between status frames. |
| `update_interval` | uint | both | Configured status/stats cadence in seconds, `15` here |
| `update_imf` | uint | both | Present, value `1` in both frames. Meaning not confirmed by any source consulted; see "Unverified / unknown." |
| `cpu_cores` | uint | both | |
| `cpu_user`, `cpu_user_prev`, `cpu_system`, `cpu_system_prev` | number (seconds) | both | Cumulative CPU time, current and previous sample |
| `flow_count`, `flow_count_prev` | uint | both | Size of the daemon's live flow table, current and previous sample. This is a **count**, never the flows themselves. |
| `maxrss_kb`, `maxrss_kb_prev` | uint | both | Resident memory high-water mark, current and previous sample |
| `dhc_status` | bool | both | DNS hint cache active |
| `dhc_size` | uint | both | DNS hint cache entry count |
| `sink_status` | bool | both | Cloud upload sink connected. `false` throughout this capture (sink disabled). |
| `sink_uploads` | bool | both | Whether upload is compiled/enabled at all |
| `agent_version` | string | second frame only | e.g. `"4.4.7"` |
| `devices` | object, MAC string to array of IP strings | second frame only | MAC to IP(s) map; useful for device-name resolution independent of DHCP leases |
| `interfaces` | object keyed by ifname | second frame only | Per-interface `addr[]`, `mac`, `role` (`"LAN"`/`"WAN"`), `state` (observed `1`, presumed up/down) |
| `stats` | object keyed by ifname | second frame only | Per-interface packet-capture counters (drops, protocol breakdown, byte totals). Not per-flow. |
**Consumer guidance from this asymmetry**: do not assume `agent_version`,
`devices`, `interfaces`, or `stats` are present on every `agent_status`
frame. Merge additively into whatever state you're building rather than
replacing it wholesale on frames that omit these keys, or you will discard
good state on the first frame after every reconnect.
**Timing anomaly, flagged rather than explained**: between the two captured
`agent_status` frames, `uptime` advanced by exactly 15 seconds (matching
`update_interval`, as expected), but the wall-clock `timestamp` field only
advanced by 1 second (`1787325073` to `1787325074`). This is inconsistent
with both fields tracking real elapsed time. It may be a genuine daemon
quirk, or an artifact of how this particular lab capture was recorded (the
JSON was captured live but the header rows were relatively hand-timed for
this exercise). No independent source was available to confirm which.
**Do not use `timestamp` deltas to infer the update cadence; use `uptime`
deltas or the configured `update_interval` instead.**
### 3.4 `flow`
Fires when a flow's Layer-7 identity is (re)determined: first classification,
or a later refinement flagged via `detection_updated`. Carries full identity
metadata. **Carries no byte or packet counters** -- confirmed directly by
this capture, where neither of the two captured `flow` events has
`local_bytes`, `other_bytes`, `total_bytes`, or any packet counter field.
Counters arrive separately via `flow_stats`.
Real captured example (a router-originated DNS query, unedited beyond
formatting):
```json
{"established":false,"flow":{"category":{"application":31,"domain":0,"protocol":14},"detected_application":10465,"detected_application_name":"netify.openwrt","detected_protocol":5,"detected_protocol_name":"DNS","detection_guessed":false,"detection_updated":false,"dhc_hit":false,"digest":"9c2d7b56c4571e146f65b0d15c12cec661d56b3f","first_seen_at":1787325075857,"first_update_at":1787325075857,"host_server_name":"openwrt.org","ip_nat":false,"ip_protocol":17,"ip_version":4,"last_seen_at":1787325075869,"local_ip":"192.168.72.1","local_mac":"94:83:c4:a8:8f:f1","local_origin":false,"local_port":53,"other_ip":"192.168.72.10","other_mac":"c0:56:27:4e:3e:92","other_port":57052,"other_type":"local","risks":{"ndpi_risk_score":10,"ndpi_risk_score_client":5,"ndpi_risk_score_server":5,"risks":[46]},"soft_dissector":false,"vlan_id":0},"interface":"wan","internal":false,"type":"flow"}
```
Reading this example concretely: `local_ip` (192.168.72.1, port 53) is the
upstream DNS resolver; `other_ip` (192.168.72.10, port 57052) is this
router's own WAN address. `local_origin: false` means the "local" side (the
DNS resolver, from netifyd's point of view) did not initiate; the "other"
side (the router) did -- i.e. this is the router itself making a DNS query,
captured on the `wan` interface (`internal: false`). `other_type: "local"`
classifies the *other* endpoint's address as being on a locally-adjacent
network, independent of which capture point (`internal`/`interface`) saw the
packet. This distinction (address locality vs. capture-point locality) is
exactly what `DESIGN.md` section 3.2's router-origin-traffic attribution
policy depends on.
| Field | Type | Notes |
|---|---|---|
| `digest` | string | 40-char lowercase hex (SHA1). Correlation key; unconditionally present on every event type that carries a `flow` object. |
| `first_seen_at`, `last_seen_at` | uint64 | Epoch **milliseconds** |
| `first_update_at` | uint64 | Present in this capture, not documented in prior source-only notes on this interface. In both captured examples it is numerically identical to `first_seen_at`. Whether it diverges once `detection_updated` fires was not observable in this capture (neither example had `detection_updated: true`). See "Unverified / unknown." |
| `detection_guessed` | bool | |
| `detection_updated` | bool | `false` in both captured examples (both are first-detection events, not re-detections) |
| `dhc_hit` | bool | DNS hint cache was used for this detection |
| `soft_dissector` | bool | |
| `ip_version` | uint | 4 or 6 |
| `ip_protocol` | uint | IANA protocol number (`17` = UDP in the example) |
| `vlan_id` | uint | |
| `ip_nat` | bool | `false` in the example |
| `other_type` | string | Observed value: `"local"`. Source also documents `"remote"`, `"broadcast"`, `"multicast"`, `"unsupported"`, `"error"`, `"unknown"` -- not independently confirmed live in this capture. |
| `local_origin` | bool | Which side initiated; see worked example above |
| `local_mac` / `other_mac` | string | MAC addresses |
| `local_ip` / `other_ip` | string | |
| `local_port` / `other_port` | uint | |
| `detected_protocol` | uint | Id into the `definitions.protocols[]` table |
| `detected_protocol_name` | string | Plain string, not namespaced (`"DNS"` here) |
| `detected_application` | uint | Id into the `definitions.applications[]` table |
| `detected_application_name` | string | `"netify.<slug>"` format when an app-level identification was made. Source indicates this can be absent/empty when only a protocol, not a specific app, was identified; not observed empty in this capture (both examples had it set). |
| `category.application`, `category.protocol`, `category.domain` | uint | Category ids. Resolved via `/etc/netify.d/netify-categories.json`'s index tables, **not** via the `definitions` applications/protocols name arrays (different id space entirely; see section 5). |
| `dns_host_name` | string | Not present in either captured example (both flows already had `host_server_name` populated from the query itself) |
| `host_server_name` | string | TLS SNI or DNS/HTTP hostname, when available. `"openwrt.org"` in the example. |
| `http.user_agent`, `http.url` | -- | Not observed (neither captured flow is HTTP); documented from source only |
| `dhcp.fingerprint` | -- | Not observed; documented from source only |
| `gtp.*` | -- | Not observed; documented from source only, GTP tunnel inner addressing |
| `risks` | object | Present in both captured examples: `ndpi_risk_score`, `ndpi_risk_score_client`, `ndpi_risk_score_server` (uint scores) and `risks` (array of numeric risk-flag ids, e.g. `[46]`). Not present in the source-derived field list this document's predecessor research was built from -- this is a real, confirmed field that prior source-reading missed. No id-to-name mapping for the values in `risks[]` was found in any source consulted here; see "Unverified / unknown." |
| `ct_id`, `ct_mark` | -- | **Not present** in either captured `flow` example, despite this build having `conntrack` compiled in (per the `agent_hello.build_version` string). Whether this is because these are UDP/no-NAT flows, or because emitting these fields needs an encode flag not set by default, is unconfirmed. |
Notably absent from `flow` in this capture: `detection_packets` and every
byte/packet counter. Those appear only in `flow_stats` (below), confirming
the two-event split cleanly.
### 3.5 `flow_stats`
Periodic per-flow counter update, once per `update_interval`, for flows past
initial detection. No identity metadata (app/protocol names, hostnames,
etc.) -- only the digest, timestamp, and counters.
Real captured example:
```json
{"established":false,"flow":{"detection_packets":16,"digest":"f663f9df9734cbaf815438f6e4b7361794729816","last_seen_at":1787325073851,"local_bytes":5336,"local_packets":19,"other_bytes":5470,"other_packets":16,"total_bytes":10806,"total_packets":35},"interface":"br-lan","internal":true,"type":"flow_stats"}
```
| Field | Type | Notes |
|---|---|---|
| `digest` | string | Correlation key |
| `last_seen_at` | uint64 | Epoch milliseconds |
| `detection_packets` | uint | Packets DPI examined before finalizing classification (a detection-time count, not a running traffic count) |
| `local_bytes`, `other_bytes` | uint | **Per-update deltas, not cumulative.** See the correction below. |
| `local_packets`, `other_packets` | uint | Same: per-update deltas |
| `total_bytes`, `total_packets` | uint | **Cumulative** since flow start. This is the only running total in the record. |
**Correction to a prior assumption, stated here as confirmed fact**:
`local_bytes`/`other_bytes` (and the matching packet counters) are
per-update deltas, not running totals; only `total_bytes`/`total_packets`
are cumulative. This was verified on hardware by observing directional
counters return to zero in the final `flow_purge` for a flow while
`total_bytes` did not (documented in this project's `DESIGN.md`; that
specific purge event was not part of the shorter capture excerpted in this
document). It also holds up as independent arithmetic in this capture's own
`flow_stats` records, without needing a purge event: for several digests,
`local_bytes + other_bytes` is an exact multiple of `total_bytes`
(consistent with equal-sized prior deltas already having accumulated into
the total), and never simply equal to it by coincidence across the whole
set. Two examples from the real data:
| digest (first 8 hex) | local+other bytes (this update) | total_bytes | local+other pkts | total_packets | Reading |
|---|---|---|---|---|---|
| `7f3838be` | 240 | 480 | 2 | 4 | This update's delta is exactly half the cumulative total: one earlier, equal-sized update already happened. |
| `37be5b0b` | 189 | 378 | 3 | 6 | Same pattern: exactly 2x, one prior equal delta already folded into `total_bytes`. |
| `f663f9df` | 10806 | 10806 | 35 | 35 | Delta equals total exactly: consistent with this being the flow's first `flow_stats` update (nothing prior to sum in). |
If `local_bytes`/`other_bytes` were themselves cumulative, the delta sum
would always equal the total, never a clean multiple of it. Records like
`7f3838be` and `37be5b0b` rule that out directly from this capture's data.
### 3.6 `flow_purge`
**Confirmed live**, across 10+ purge events captured separately during this
project's hardware testing (`DESIGN.md`), though none happened to fall
inside the specific capture session excerpted in this document (the two
newly detected flows were still active when that capture ended, and none of
the already-tracked flows from the initial `flow_stats` burst closed during
the window). The confirmed live behavior: `local_bytes`/`other_bytes`
(and the matching packet counters) are **always zero** on the closing purge
event; `total_bytes`/`total_packets` carry the flow's final cumulative
totals, unchanged from the flow's last known value going into the purge.
The example below is reconstructed for illustration, since this document's
own capture did not happen to include one, reusing a real digest from this
capture with counters set to that confirmed pattern:
```json
{"established":false,"flow":{"digest":"9c2d7b56c4571e146f65b0d15c12cec661d56b3f","last_seen_at":1787325199999,"local_bytes":0,"other_bytes":0,"total_bytes":642,"local_packets":0,"other_packets":0,"total_packets":6,"reason":"closed"},"interface":"wan","internal":false,"type":"flow_purge"}
```
**(Reconstructed for illustration: this document's own capture did not include a purge event, but the local_bytes/other_bytes=0, total_bytes=final pattern shown is confirmed live behavior, not speculation.)**
| Field | Type | Notes |
|---|---|---|
| `digest` | string | Correlation key; the flow being removed |
| `last_seen_at` | uint64 | Epoch milliseconds |
| `local_bytes`, `other_bytes`, `local_packets`, `other_packets` | uint | **Always zero** on `flow_purge`, confirmed live across 10+ purge events (`DESIGN.md`). The flow's final directional split is not available on this event; see the accounting note below. |
| `total_bytes`, `total_packets` | uint | Final cumulative totals for the flow's whole lifetime |
| `reason` | string | One of `"closed"` (e.g. TCP FIN), `"expired"` (idle timeout), `"terminated"` (e.g. agent shutdown) |
Emitted when a flow leaves the tracking table: normal close, idle expiry, or
agent shutdown. This is the last event you will see for a given digest.
Because the directional counters are zeroed on purge, a consumer cannot read
a flow's final up/down split off this event directly; this project's daemon
instead recovers it by distributing the `total_bytes` delta since the last
known total across up/down using the flow's established ratio from its
prior `flow_stats` updates.
## 4. Consumer-relevant caveat on `agent_status`
Repeated from section 3.3 for visibility: never rely on `agent_status`
carrying `agent_version`/`devices`/`interfaces`/`stats`. Only the base
process-telemetry fields are guaranteed on every frame.
## 5. Naming and id spaces
Two id spaces coexist in `detected_protocol`/`detected_application`, and
they are **not** globally unique against each other -- this is a genuine
footgun confirmed directly from the real `definitions` payload.
- **nDPI-native protocol detections** (`detected_protocol` / `protocols[]`
in `definitions`): plain, unnamespaced strings -- `"DNS"`, `"HTTP"`,
`"HTTP/S"`, `"BitTorrent"`, `"WireGuard"`, `"Tailscale"`, `"ZOOM"`, etc.
217 entries in this capture, ids observed from `0` (`"Unknown"`) up to
`316` (`"Meraki/Cloud"`), plus a sentinel id `4294967295` (`0xFFFFFFFF`,
uint32 max) tagged `"TODO"` whose meaning is not confirmed by any source
consulted (placeholder for an unassigned/pending signature, presumably,
but unverified).
- **Netify application-layer detections** (`detected_application` /
`applications[]` in `definitions`): use the `netify.<slug>` naming
convention when a specific app/service was identified on top of the
transport protocol -- `netify.netflix`, `netify.tailscale`,
`netify.openwrt`, etc. 416 entries in this capture, 199 of which are
`netify.`-prefixed; the remaining 217 non-`netify.`-prefixed entries in
the *applications* array are the same low-numbered nDPI protocol tags
mirrored into that array as well (e.g. id `156` appears in `applications[]`
tagged `"Spotify"` *and* separately in `applications[]` tagged... no --
more precisely: id `156` is `"Spotify"` in `protocols[]` and
`"netify.spotify"` in `applications[]`. These are two different arrays
with independently assigned ids that happen to collide numerically.
**The id-collision gotcha, confirmed directly from the real captured
`definitions` payload:**
| Numeric id | Meaning in `protocols[]` | Meaning in `applications[]` |
|---|---|---|
| 121 | `Dropbox` | `netify.dropbox` |
| 156 | `Spotify` | `netify.spotify` |
| 206 | `WireGuard` | `netify.cloudflare` |
| 207 | `SMPP` | `netify.microsoft-onedrive` |
`id 206` is the sharpest example: it is `WireGuard` in the protocol table
and `netify.cloudflare` in the application table -- two completely unrelated
things sharing a number. **Always resolve `detected_protocol` only against
`definitions.protocols[]`, and `detected_application` only against
`definitions.applications[]`. Never cross-reference an id between the two
arrays.**
A second, related nuance: a widely-repeated simplification (also present in
this project's own `DESIGN.md`, stated there as a general rule of thumb) is
that netify application ids are `>= 10000` and nDPI protocol ids are
`< 10000`. That is the dominant pattern for *recently added* app signatures,
but it is not a hard rule: this capture's `applications[]` array contains at
least 26 legacy `netify.<slug>` entries with ids well under 10000, including
some very common ones:
```
68 netify.msn
70 netify.yahoo
119 netify.facebook
120 netify.twitter
121 netify.dropbox
122 netify.gmail
124 netify.youtube
133 netify.netflix
156 netify.spotify
195 netify.twitch
199 netify.snapchat
201 netify.instagram
202 netify.microsoft
206 netify.cloudflare
```
The reliable discriminator for "is this an app-level detection, not just a
protocol-level one" is the literal `netify.` string prefix on
`detected_application_name`, not the numeric id range. Use the prefix test,
not an id threshold.
Cross-checked against the actual `netify-apps.conf` examples cited in prior
source-reading of this project (`app:133:netify.netflix`,
`app:10119:netify.adobe`, `app:10552:netify.tesla`, `app:199:netify.snapchat`):
all four id-to-slug pairs match exactly what's in this capture's
`definitions.applications[]` array, which is good cross-validation that the
signature-file mapping and the live socket's id table are the same data.
## 6. Category files on disk
`/etc/netify.d/netify-categories.json` (installed by the `netifyd` package,
not vendored by this repo) provides the category taxonomy:
- `application_tag_index` / `protocol_tag_index` -- numeric category id to
name, e.g. `streaming-media`, `social-media`, `games`, `voip`, `cdn`,
`advertiser`, `malware`, `os-software-updates`.
- `application_index` / `protocol_index` -- numeric category id to the
member application/protocol ids in that category.
These map to `flow.category.application` / `flow.category.protocol` /
`flow.category.domain` (see section 3.4). No category object was captured in
enough detail in this session to re-derive the full tag list independently;
figures below are as previously documented and are called out where they
disagree.
**Count discrepancy, unresolved**: prior source-reading of this project
enumerated exactly 31 application-category tags and 17 protocol-category
tags by name. This project's `DESIGN.md` states 32 and 18 respectively
(without enumerating the 32nd/18th). Neither the category JSON file itself
nor a `flow.category.*` value dense enough to reverse-engineer the missing
entry was available for this document. Treat `DESIGN.md`'s counts as
authoritative per this project's convention, but be aware the exact
32nd application tag and 18th protocol tag were not independently confirmed
here.
**Licensing**: both the signature database that drives application/protocol
detection and the generated category file are proprietary data owned by
eGloo/Netify, shipped as part of the `netifyd` package itself, not part of
netifyd's GPLv3 source distribution in the way the daemon code is. This
project reads them at runtime from the paths above and vendors none of this
data in its own repository. (Prior source-level reading found the explicit
proprietary-license header specifically inside a signature pattern-matching
file distinct from the plain id-to-slug `netify-apps.conf`; that specific
filename-level detail was not independently re-confirmed for this document
and is listed under "Unverified / unknown." The operative conclusion, that
none of this data should be vendored or redistributed, is confirmed by both
sources independently and is not in question.)
## 7. Practical consumer guidance
**Join on digest.** `flow.digest` and `flow.last_seen_at` are the only two
fields serialized unconditionally across every event type that carries a
`flow` object. Key your flow table by `digest`. Do not assume any other
field is always present -- `flow` carries identity and no counters,
`flow_stats` carries counters and no identity, `flow_purge` carries a final
cumulative `total_bytes`/`total_packets` plus a reason, but zeroed
directional counters (§3.6): it does not carry a final directional split.
**Stub-flow pattern.** Because `dump_established_flows` does not exist in
4.4.7 (section "Scope" above), and because this is directly demonstrated by
the real capture (11 `flow_stats` records for digests never announced via
`flow` on this connection), a consumer must tolerate `flow_stats` or
`flow_purge` arriving for an unknown digest. Do not drop these events.
Instead, create a placeholder entry (interface/internal known from the
envelope, identity fields set to an "Unknown" sentinel) and backfill
identity if and when a `flow` event for that same digest eventually arrives.
It may never arrive, if the flow was already fully classified before this
connection started and its classification does not get re-announced.
**Delta clamping rationale.** Because `local_bytes`/`other_bytes` are
per-update deltas while `total_bytes` is the only cumulative field, a
consumer accounting a running rate should sum the directional deltas per
interval directly. But right after a reconnect (where a stub flow has no
prior history), or after any missed update, naively summing deltas risks
overshooting the flow's real observed growth. Clamp the accounted delta so
the consumer's own running total for that digest never exceeds
`total_bytes`'s growth: `accounted_delta = min(local_bytes + other_bytes,
total_bytes - last_known_total)`. This is the approach this project's daemon
takes, and it is the correct general-purpose defense against the blind-spot
created by the missing backfill option.
**Resolve ids against the matching array only.** `detected_protocol` against
`definitions.protocols[]`; `detected_application` against
`definitions.applications[]`; `flow.category.*` against
`netify-categories.json`'s index tables. All three are separate id spaces
that can and do collide numerically (section 5).
## 8. Unverified / unknown
Collected here in one place rather than scattered as caveats only:
- **`established` (top-level boolean)** -- precise semantics not confirmed
by source reading or by this capture (always `false` in the sample, which
is too uniform to infer meaning from).
- **`update_imf`** (in `agent_status`) -- present, value `1` in both captured
frames, meaning not confirmed by any source consulted.
- **`first_update_at`** (in `flow`) -- a real field, not previously
documented for this interface. Identical to `first_seen_at` in both
captured examples (both first-detection events); whether/how it diverges
when `detection_updated` fires is unconfirmed.
- **The `agent_status` timing anomaly** -- `uptime` advanced 15s between two
captured frames while `timestamp` advanced only 1s. Not explained; may be
a genuine daemon quirk or a capture-environment artifact. Flagged, not
asserted as behavior to depend on.
- **`risks.risks[]` numeric values** (e.g. `46` observed) -- no id-to-name
mapping found in any netifyd-side source or doc consulted. These are
presumably nDPI's own risk-flag enum (documented upstream in nDPI, not in
netifyd), which was out of scope to chase down for this document.
- **`other_type` full enum** -- only `"local"` was observed live. The other
values (`remote`, `broadcast`, `multicast`, `unsupported`, `error`,
`unknown`) come from source reading only, not from this capture.
- **`ct_id`/`ct_mark`/conntrack correlation** -- absent from both captured
`flow` examples despite `conntrack` being a compiled-in capability per
`build_version`. Unconfirmed whether that's because these are UDP/no-NAT
flows or because emission requires a config flag not on by default.
- **Category tag counts** -- 31/17 (prior source enumeration, with names) vs
32/18 (`DESIGN.md`, without full enumeration). Not reconciled; see
section 6.
- **Whether `detected_application_name` can be absent/empty** for
protocol-only detections -- asserted by source reading, not observed
empty in this capture (both examples had an app name set).
- **The specific filename carrying the proprietary-license header** on the
signature data (`netify-apps.conf` vs. a separate pattern-matching file) --
see the licensing note in section 6.
- **TCP socket default port** -- irrelevant to OpenWrt's default
UNIX-socket-only configuration, so omitted from the main body, but never
resolved in prior research: conflicting values (`7150` in man-page prose,
`2100` in a shipped sample config) were found and neither was checked
against a running instance with TCP enabled.
- **`http.*`, `dhcp.fingerprint`, `gtp.*` sub-objects** -- none observed in
this capture (no flows of the relevant type were present); documented from
source only.
## 9. Sources
- Direct hardware capture: Linksys EA8500, OpenWrt 25.12.5, netifyd 4.4.7,
UNIX socket capture immediately following daemon start (this document's
primary evidence for every "real captured example").
- This project's `docs/DESIGN.md`, sections 2 and 2.4 (separately
hardware-verified facts, treated as authoritative wherever they conflict
with source-only reading).
- netifyd 4.4.7 source at the exact shipped tag:
`gitlab.com/netify.ai/public/netify-agent` (tag `v4.4.7`), in particular
`src/netifyd.cpp`, `src/nd-detection.cpp`, `src/nd-json.cpp`,
`src/nd-socket.cpp`, `include/nd-flow.h`, `include/netifyd.h`,
`deploy/netifyd.conf.in`, `deploy/netify-categories.json`,
`deploy/netify-apps.conf`, `doc/netifyd.conf.5.in`,
`doc/README-JSON-socket-example.md`, `doc/json-socket-filter.jq`.
- OpenWrt packaging: `github.com/openwrt/packages`, `net/netifyd/`
(Makefile, `files/netifyd.init`, `files/netifyd.config`).
luci-app-appflow/docs/overview.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 138 KiB

luci-app-appflow/docs/statistics.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 174 KiB