small-packages/luci-theme-footstrap/docs/ci.md

# CI and packaging

How footstrap is turned into packages and shipped. The theme is noarch
(`LUCI_PKGARCH:=all`) and supports **OpenWrt 24.10** (`.ipk`/opkg) and **25.12+** (`.apk`/apk).

What the Makefile and the install scripts do: [package.md](package.md). The release runbook:
[releasing.md](releasing.md).

## The job graph

`.github/workflows/build.yml`. Triggers: push to `main`, a `v*` tag, any pull request, or a manual
dispatch.

```
check ─┐          ┌─→ verify ─────┐
       ├─→ build ─┼─→ live ───────┼─→ release ─→ playground-asset ─→ pages   (release and after: tags only)
lint ──┘          └─→ playground ─┴─→ pages-manual                           (workflow_dispatch, publish-pages=true)
```

| Job | What it is |
|---|---|
| `check` | static gates that need no node |
| `lint` | the npm gates: eslint, stylelint, axe-core, the ratchets |
| `build` | both package formats, via owfeed |
| `verify` | installs this very build on real 25.12 and 24.10 userlands and renders its pages |
| `live` | opens every page of the menu on those userlands and measures it — layout, navigation parity, and the assumptions this theme makes about luci-base |
| `playground` | records a real router's pages into a static, replayable site and proves it offline |
| `release` | signs, generates the notes, attaches the assets |
| `playground-asset` | uploads `playground.tar.gz` to the tag's release, once one exists |
| `pages` | refreshes the GitHub Pages portal and the release mirror, from the latest release |
| `pages-manual` | same refresh, off a manual dispatch, from THIS run's own playground instead of a release — see `pages`, below |

`permissions: contents: read` at workflow level; only `release` and `playground-asset` declare
write, both gated to a `v*` tag. It used to be workflow-wide, which handed it to every
`pull_request` run — including `npm ci` in lint, and therefore to the lifecycle scripts of every
dev dependency.

**Every job has a `timeout-minutes`, and every network install runs under `tools/ci-retry.sh`.**
The default job timeout is six hours, and the two things these jobs fetch from somebody else's
server — apt and the Playwright CDN — stall rather than fail: on the 0.13.2 tag one apt step and two
playwright steps sat for 68, 68 and 17 minutes with GitHub reporting every system operational, and
nothing retried them because nothing had failed. Each attempt now gets its own deadline (six minutes
for apt, seven for playwright, three attempts), so a stall is a retry rather than a blocked release,
and the job timeout is the backstop behind that rather than the only clock in the job. Each attempt runs in a session of its own and is killed by process GROUP, because `timeout` signals only the child it started: the first version of the helper killed `playwright install --with-deps` and left the `apt-get` beneath it holding the lists lock, after which both retries died in seconds on that lock.

And apt is asked for as little as possible, because retrying a stalled apt is still waiting for a
stalled apt: a run spent its whole 20-minute budget doing exactly that. `gettext` is installed only
when `msgfmt`/`msgmerge`/`xgettext` are missing from the image, and `tools/ci-playwright.sh` fetches
Chromium from Playwright's CDN and then LAUNCHES it — the claim the gates actually need — going to
apt for the system libraries only if that launch fails. `--with-deps` went through apt every time,
and apt is the half that stalls.

## `check` — gates without node

Needs only `sh`, `awk`, `python3` and `perl`. Seconds to run, and it cannot break the OpenWrt
buildbot, where node does not exist and never will.

1. **`sh -n`** over `luci-theme-footstrap/*.sh`, `install.sh` and `tools/*.sh` — the scripts that
   never reach a router. `tools/` is in the glob because `release-notes.sh` runs only in the
   `release` job, so a syntax error there used to surface at the most expensive possible moment.
   The *payload* scripts are parsed elsewhere: `owfeed doctor` (OWF213) parses everything under
   `files:` in the `build` job, which covers `/etc/uci-defaults/*` once `tools/stage.sh` has staged
   it.
2. **The scan marker.** `include/scan.mk` finds packages by grepping for `call BuildPackage`, which
   this Makefile only reaches through `luci.mk`'s include — so the literal in its trailing comment
   is what makes the SDK see the package at all. It has been deleted as boilerplate once.
3. **The ACL is valid JSON, and grants something.** rpcd skips an unreadable file in `acl.d` and
   says nothing: a stray comma means the grant is issued to nobody, and nothing else notices. A
   document that parses but is a list, or an entry with neither `read` nor `write`, is the same
   silent outcome by another route, so `tools/check-acl.sh` checks the shape too.
4. **`build-css.sh`** into a temp file — the script brace-balances its own output and refuses to
   write a suspiciously short file. That is a broken-build floor (80 KB), a correctness gate,
   not a size budget. There is no upper CSS budget any more.
5. **`audit.py --strict`** — undefined `var()`, shadowed declarations, export-tier reads from
   `styles/`, dead base declarations, stray `!important`, colour literals.
6. **i18n**: `update-po.sh --check` fails if the `.pot` is stale or any `msgstr` is empty. A string
   in `_()` with no translation renders in English silently — which is how the whole Footstrap
   tab stayed English on a Russian LuCI.

**Template compilation is not here** — it runs in `verify`, on the router, with the real `ucode`.
It used to clone the interpreter at a pinned commit and build it with cmake, with the router runtime
stubbed out through `-L`; the container has both for free and stubs nothing.

## `lint` — the npm gates

They live in CI only: the buildbot has no node and does not need it. Nothing in `package.json`
ships. Locally it is all one command, `npm run check`; the full table of what each gate holds is in
[conventions.md](conventions.md). The ones worth naming here:

| Step | What it catches |
|---|---|
| `eslint` | including `wrap-regex`, which forbids `return /re/…`, the form jsmin breaks on |
| `stylelint` | correctness and project invariants only — not a formatter |
| `a11y-gallery.mjs` | axe-core, WCAG 2.2 AA over `docs/gallery.html`, {light,dark} × {footstrap,hicontrast,bootstrap,2020,forum} × {untinted,60°,260°} — **30 combinations** |
| `export-tier.mjs` | the `--*-color-*` contract with foreign apps — axe cannot see it, their widgets are not in the gallery. 70 palette × mode × tint combinations, then the ten untinted ones again with `prefers-contrast: more` emulated, since that query re-states the ink tokens and so publishes a second tier |
| `css-metrics.mjs` | ratchet: `!important` ≤ 27, max specificity, empty rules |
| `css-floor.mjs` | the browser floor derived from the built sheet against the one stated in `docs/css.md`, plus the two shapes that break below it: a `:has()` compound sharing a selector list with one that has none, and a CSS feature nobody has classified |
| `fs-orphans.mjs` | dead `fs-*` selectors (safe only inside our namespace) |
| `css-dup.mjs` | identical declaration bodies under different guards — no linter calls this an error |
| `mirror.mjs` | `@mirror`-pinned copies still byte-identical (CSS **and** shell) |
| `axes.mjs` | the pre-paint in `head.ut` agrees with the live appearance appliers |
| `page-modules.mjs` | the page → module map agrees with the modules, and neither is `require`d at eval |
| `table-contract.mjs` | the break points, and the rule that hides an unanswered data table still names every root `fs-select` scans, guarded on `:root[data-fs-fit]` — the selector half of what `table-tick.mjs` then proves on a page |
| `chrome-fence.mjs` | the `[data-fs-chrome]` marker, fence and pin still match the chrome |
| `conffiles.mjs` | every shipped `/etc/config/*` is declared a conffile — else the manager replaces it on upgrade |
| `bang-ok.mjs` | the `!important` allowlist in `audit.py` and `.stylelintrc.json` still say the same thing |
| `changelog.mjs` | the changelog contract: sections, order, RU mirror, bold leads |
| `jsmin-verify.mjs` | the **only** check that catches jsmin's silent corruption (exit 0) |
| `npm test` | the unit suite — no browser, ~100 ms, and first in the job for that reason. It holds the branches a stand cannot enter: a luci-base missing a surface the router calls, an alias loop or a `firstchild` tie in a menu tree nobody ships |
| `size-budget.mjs` | ratchet on the **artefact**: `build-css.sh` + the token mangle for the sheet, terser over a copy of `htdocs/luci-static/resources` for the JS. `--show` prints the per-module table into the log, so a jump is attributable without re-running anything |
| `build-icons.mjs --check` | the committed app-icon rasters still match `logo.svg` — per channel with a tolerance, plus the maskable invariants (declared size, no alpha, nothing outside the safe zone, a mark in the middle). Not byte equality: that tested the renderer, and a Chromium bump reddened it on a commit that never touched the logo |

**About `jsmin-verify`:** jsmin corrupts a file silently and exits 0. The source shape it breaks on,
and why eslint's `wrap-regex` stands beside this gate, are in [conventions.md](conventions.md); the
two minification paths are in [package.md](package.md). What CI adds is the proof: it builds that
same jsmin from `luci-base/src/jsmin.c` at the pinned commit and compares the token stream of the
output against the source (acorn). The list of shipped JS is not written by hand — it is
`find htdocs -name '*.js'`, because luci.mk copies `htdocs/` wholesale, so that *is* the shipped set.

Both gates stay mandatory **for the source**, even though the release build minifies with terser
instead: a build without node still hands the untouched source to jsmin.

**About `size-budget`:** the numeric budgets were dropped once, and the reason they are back is what
they measure now. The old ones were counted on the SOURCE, which is not what anyone downloads — the
sheet is concatenated and token-mangled and the JS goes through terser, all of it inside the package
build, so a developer's edit-and-check loop never sees the artefact at all. That is the shape a
number drifts in: one feature at a time, each too small to argue with, none of them measured. This
gate reproduces the package build's asset half and weighs the result. uhttpd serves `/www` with no
compression, so identity bytes are wire bytes, read off flash by a core that is also routing packets.
Raising a limit is a decision and wants a comment saying what it bought; lowering one after a cleanup
is how the slack stops being spendable in silence.

Most of what the other jobs run lives in `tools/*.sh` rather than inline in the YAML —
`check-shell.sh`, `scan-marker.sh`, `check-acl.sh`, `build-jsmin.sh`,
`check-packages.sh`, `feed-key.sh`, `stage-release.sh`. A step that is a script can be run by hand
when it fails, and its reasoning lives beside the code instead of inside a workflow nobody reads.

## `build` — owfeed, not the SDK

Both formats are built by [owfeed](https://github.com/owfeed/owfeed) from one staged rootfs.
The SDK legs are gone: the theme is noarch (CSS, templates, browser JS, fonts — not one byte of
compiled code), and each leg downloaded, verified and unpacked a cross toolchain in order to run
`cp`. Twice, because 24.10 is opkg and 25.12 is apk.

| Format | Line | noarch spelling | What owfeed does |
|---|---|---|---|
| apk | 25.12+ | `noarch` (`all` is rejected as uninstallable) | `apk mkpkg` + `.conffiles` sidecars |
| ipk | 24.10 | `all` | an `ipkg-build` container + `CONTROL/conffiles` |

Three things the SDK really provided, owfeed does itself: it compiles `.po` → `.lmo` with its own
compiler (**byte-identical to `po2lmo`** — checked on release 0.11.6), emits the same sidecars as
`package-pack.mk`, and wraps the scripts in `default_postinst`/`default_prerm`, without which
`/etc/uci-defaults/*` never runs at all.

**The SDK check did not leave with the SDK.** owfeed extracts the host `apk` from a release SDK
tarball and keeps it on the same chain this workflow used to spell out: an ed25519 signature over
`sha256sums`, the branch key pinned from a *different* host (`github.com/openwrt/keyring`), and the
sha256 read only out of an already-authenticated file.

**owfeed itself is installed by its own action** — `owfeed/owfeed/setup`, pinned by commit — which
downloads the binary and **verifies it against GitHub's build attestation before it reaches PATH**.
That is why it is preferred to `go install …@<sha>`, which compiles the tool in every job and trusts
whatever the module proxy returns. A `SHA256SUMS` next to a release is not a check either — the same
host serves both the binary and the sum (the same argument this project makes about GitHub's asset
digest). An attestation is signed by GitHub's identity, sits in a public transparency log, and the
action pins the workflow that was allowed to issue it. Verified locally on a release binary:
genuine → exit 0; one byte appended → exit 1; against a foreign repository → exit 1.

The steps:

1. **Resolve the version** — from the tag (`v0.3.6` → `0.3.6`) or a fallback `0.<date>.<run>`. Plus
   `SOURCE_DATE_EPOCH` = the commit timestamp: both containers write mtimes, and a package's
   identity is a hash over its payload, so without a fixed epoch byte-identical content rebuilds
   into a package that *claims* to be new.
2. **`./tools/stage.sh`** — the half owfeed deliberately does not do ("packages a directory; does
   not build one"). It is `Build/Prepare` from the Makefile, step for step: build `cascade.css`,
   mangle the private `--fs-*` names, minify the JS with terser, strip comments out of the templates
   and the shell, stamp `FS_VERSION`. It also extracts postinst/postrm — by awk over the
   Makefile's `define`s, not as a second copy: a script that runs on somebody else's router months
   later is the worst possible place for two copies to diverge.
3. **`owfeed build`** — deliberately without `--frozen-lock`. That flag re-derives `owfeed.lock`
   from `downloads.openwrt.org` and fails if it moved, which is right for a repository that
   **publishes a feed**, where the covered architecture set is part of the contract with
   subscribers. Here two noarch packages ride as release assets and expand into no architectures, so
   the flag buys nothing and costs a red release on every OpenWrt point release. The committed lock
   still pins which SDK the host `apk` comes from, so the toolchain does not float.
4. **Assert the catalogues.** A `.lmo` is compiled in and absent from git, and a silently missing
   one means every `_()` renders the English msgid with nobody reporting anything. Each language
   ships as its own `luci-i18n-footstrap-<lang>`, so the assertion is one catalogue named
   `footstrap.<lang>.lmo` per language package, a uci-defaults line registering that language, and
   **none in the theme** — two packages owning one path is an install apk refuses. Counted against
   the languages in `po/`, so adding one and forgetting to declare its package is a red build
   rather than quiet English for some users.
5. **Flatten the packages into `dist/`** (owfeed writes `dist/noarch` and `dist/all`; apk derives
   the filename from name and version, so two architectures cannot share a directory). CI requires
   that the theme resolve to exactly one asset per format under a name-anchored regex
   (`luci-theme-footstrap[-_]…`). That is not hygiene, it is protection for old routers: see issue
   #6 in [conventions.md](conventions.md).

## `verify` — the package works on a real userland

Downloads the artifact and installs this build on real 25.12 and 24.10 containers through the
owlab action, then renders its pages. On non-PR runs it additionally fetches the feed's public key
and asserts the published feed serves a working theme.

That the build "produced an ipk" proves nothing about the ipk; this job is what does.

**When the feed does not answer, the assertion is skipped and says so.** `tools/feed-key.sh` still
reports `reachable=false` rather than failing — the claim is about a channel this repository does not
own — but the run now ends with "Report whether the published feed was measured", which writes
*measured and passed*, *measured and failed* or *NOT MEASURED* into the run's annotations and job
summary. A green `verify` no longer looks the same whether the feed was tested or never asked;
`tools/ci-local.sh verify` prints the same sentence from the same script.

**Five assertions per leg**, and the fifth is the template gate:

```
package luci-theme-footstrap
file /www/luci-static/footstrap/cascade.css
http 200 /cgi-bin/luci/admin/status/overview
http 200 /cgi-bin/luci/admin/system/system
exec for f in …/themes/footstrap/*.ut; do ucode -T -c -o /dev/null "$f" || exit 1; done
```

`ucode -T -c` is LuCI's own trycompile, and it runs here against the installed templates with
the router's real `luci.core` and `uci` — no interpreter of ours, no stubs. It runs on both
legs: the templates are identical but the interpreters are not, and a construct 25.12's ucode
accepts is no proof that 24.10's does. Without it a stray brace in `header.ut` ships green and
silently moves every user's LuCI to another theme, because luci.mk copies `ucode/` verbatim and
nothing parses it on the way.

The same assertions are what `owlab test` runs locally — see [development.md](development.md), and
keep the two in step.

## `live` — the pages behave, not just the package

`verify` proves the package installs and two pages answer 200. That is not the same claim as "the
pages are right", and the difference is where every field report has lived: #11 a column shredded to
one character per line, #22 a clipped submenu title, #12 a doubled scrollbar in one engine, #8/#33/#36
a third-party app laid out wrong, and two pages that came back empty after a client navigation — all
green under every static gate at the time.

The job boots the same owlab containers a developer runs locally (`owfeed/owlab/setup` puts the CLI
on PATH; the binary is checked against its build attestation), installs the artifact `build` just
produced, and runs the live gates.

**It is three jobs, not one.** Once `anchors` moved out, this was the release's critical path, so it
became three slices (`parity`, `audit`, `motion`), each booting its own routers for 157s it does not
share — the wall clock is the longest slice and not the sum. `fail-fast` is off: a parity failure
says nothing about whether the reader stays put. Where two gates share a slice they also share the
page shapes they read, through `FS_SHAPES` — classifying is one load and a 1200ms settle per page of
the menu, and both gates used to do it back to back (measured: live-audit 421s alone, 303s after
spa-parity had already read the same router). **Splitting them onto separate runners gave that
saving back**: `FS_SHAPES` points inside each runner's own temp, so `parity` and `audit` each walk
the whole menu now — ~90s a router, and the price of the split.

Measured on the last run where every slice finished (`7ff9e56`, push, three routers): the gates are
**485s** (spa-parity), **450s** (live-audit) and **754s** (`motion`'s six). `motion` carried a
seventh until task liveslice — chromium's own scroll-anchor sweep, which grew from 145s to **1434s**
when `--full` started crossing its density axis (`e48434c`) and put the slice at 39 of its 45
minutes. That sweep is a shard of `anchors` now, beside firefox and webkit, where the cap is 75
minutes and the one-job-per-engine rule already lives; the cells are the same cells.

**A number here only holds while every page still answers.** `page.evaluate()` has no deadline in
Playwright, so a page that pins the browser's main thread does not cost a page's worth of seconds —
it costs the slice its whole budget. That is what happened for a day: `/admin/system/filemanager`
under `13e9864` (`fs-fit.js`), `parity` and `audit` both cancelled at 45 minutes on every run, with
nothing in either log. Both gates now print a line per page with the clock on it, so the next one
names the page instead. **Read which page a slice last named; never raise the cap.**

Printing a line is only half of it: a slice that names the page and is then killed at 45 minutes
still measured nothing about the pages behind it. Each shape probe in `classify()`
(`tools/lib/page-shapes.mjs`) is capped at `PROBE_DEADLINE_MS` (10s) — about 2000x the 4-5ms a live
page answers in, and wide enough that even a menu where every path froze costs 16 minutes rather
than the whole slice (96 paths, the widest measured, `owrt2512b` with the third-party fixtures). A
page that loaded and then stopped answering is printed as a FINDING naming the router and the path,
the walk continues on a fresh tab — a frozen page stays poisoned, every later `goto` on it burning
its own 20s, while `close()` + `newPage()` costs 0.5s and the login lives on the context — and
`parity` and `audit` exit non-zero. A page that never LOADED still says nothing, on purpose: that is
the runner or the stand, not the theme, and the two must not read alike.

Cheapest-first still holds inside `motion`, which opens with upstream-contract at 7s; it can no
longer come before the other two slices, which are on runners of their own.

1. `tools/upstream-contract.mjs` — the assumptions about luci-base. It runs first because if one of
   them has moved, every finding below is downstream of it.
2. `tools/spa-parity.mjs` — every page opened by a click and by a full load, compared.
3. `tools/live-audit.mjs` — one page per shape at six widths (resized into) plus one ENTERED with a
   load of its own — and the arrival records only what the resize at that width did not already
   say, since a fault both passes see is one fault and a second copy of it is a baseline entry
   that carries no information and only exists where that app is installed. Ratcheted against
   `tools/baselines/live-audit.json`. `--update` UNIONS into that
   file rather than replacing a router's set: the baseline is a union across platforms, and a
   machine that does not install mwan3 must not delete mwan3's findings.
4. `tools/scroll-jank.mjs` — a real wheel, long enough to meet a poll tick, in both layouts: nothing
   may re-decide or jump while the page moves.
5. `tools/table-tick.mjs` — the poll tick performed deliberately, with the layout forced inside the
   window fs-select answers in: a replaced data table may not be laid out before it has an answer.
6. `tools/scroll-anchor.mjs` — content grows above the reader and the page must not move under them,
   with and without the engine's own scroll anchoring. It runs in its OWN job, **one shard per
   engine — chromium, firefox and webkit** (`anchors`), not here: at 52 minutes against this job's 17
   it was the whole release's critical path, and the three engines share nothing, so the shards run
   at once. Chromium stayed in `motion` while it was the cheap engine at 145s and left when `--full`
   took it to 1434s (task liveslice); it is now the shortest of the three shards and costs the
   pipeline no wall clock at all. Its sweep is narrow on a pull request and `--full` on a push — the
   axes it drops were measured not to change its answer, and a push is where being wrong about that
   must still be caught.
7. `tools/install-check.sh` — `install.sh` twice on each router, fresh and over its own result. It
   goes last in its slice because it replaces the build under test with the published release; #16,
   #28 and #30 were all this script, and all on the second run. That replacement is also why it may
   not share a router with a gate still measuring the build — which a slice of its own guarantees.

**Three routers on a push, one on a pull request.** The push set is the OpenWrt lines the theme
supports — 25.12/apk, 24.10/opkg and the snapshot box, which tracks luci-base's master and so fails
on an upstream change before a user reports it. ImmortalWrt is not in it: same luci-base, different
brand and app set, never the leg that caught something first, and no gate measures it locally
either (`tools/lib/stands.mjs`). What each gate holds, and how
to run it by hand, is in [conventions.md](conventions.md) and [development.md](development.md).

**The snapshot box carries no third-party extras (`owlab.yaml`, `owrtsnap`).** openclash and
ssclash pull `kmod-tun`, which resolves through a kmods index keyed to the box's exact kernel git
hash, and the box's baked kernel is never the one `downloads.openwrt.org/snapshots` is currently
publishing kmods for — permanent drift, not a stale image. Measured 2026-09-07: `apk add` 404s the
kmods `packages.adb` for the running `6.18.33` hash and both apps fail; justclash (no kernel module)
installs cleanly on the same box, which is why it was never in the failure list. `owlab`'s own
`extra_packages` merge is additive only (`defaults:` plus what a router adds, no subtraction), so
the three apps are declared once, on `owrt2512`, and aliased onto the other release routers instead
of living in `defaults:` — `owrtsnap` gets none. The snapshot leg still runs every other gate; only
these three apps are untested there.

## `playground` — a real router, recorded and replayed

Turns `admin/status/overview` and seven other pages of an owlab stand into a static site any browser
can open with no router behind it, and proves that site offline. Three scripts, `tools/playground/`:

1. `capture.mjs` — boots nothing itself, talks to the `owrt2512` this job already installed the
   build on: logs in, fetches every page in `pages.json`, and records the server's own document, the
   `/luci-static/**` assets those pages fetched, every distinct ubus call and the ACL-filtered menu.
2. `build.mjs` — offline, deterministic: rewrites the recorded paths under `--base`, scrubs tokens
   and the stand's hostname, splices in `replay.js` (patches `XMLHttpRequest`/`fetch` to answer from
   the recording instead of a router), and tars the result.
3. `verify.mjs` — opens the built site in a real Chromium and fails it on a page error, a replay
   miss, a 404, or an empty `#view`; also proves the one client navigation this theme has and that an
   Appearance change survives a reload.

Runs on a tag, a pull request and `workflow_dispatch` — not only on release, because a regression in
`fs-*.js` against a real ubus answer is a real fault and the cheapest place to catch it is before a
tag exists to publish from. It replaces the 475 KB hand-written snapshot that used to live here,
which imitated `fs-*.js` inline and had already gone 13 releases stale (`CHANGELOG.md`): this build
runs the theme's own JS, so it cannot drift from it the same way.

Local run: `tools/ci-local.sh playground` (needs `--force`, same stand-safety rule as `live`); by
hand, `docs/development.md`.

## `release` — signing and publication

Tags only, and **it is a call into owfeed's own reusable workflow** rather than steps of ours:

```yaml
release:
  needs: [build, verify, live, anchors, lint, playground]
  if: startsWith(github.ref, 'refs/tags/v')
  uses: owfeed/owfeed/.github/workflows/package.yml@v0.5.1
  secrets:    { sign-key: …OWFEED_AUTHOR_KEY, usign-key: …FOOTSTRAP_USIGN_KEY }
  with:       { owfeed-version: v0.5.1, pre-release: sh tools/stage-release.sh,
                sign-also: install.sh, notes-file: release-notes.md, verify-with: release.pub }
```

**This is the only job that holds a key**, and this `needs` list is what stops a tag publishing a
package no router has installed, no page has behaved on, or no recording has proven runs the real
JS — `playground` is in it for the same reason as `live` and `anchors`: a fault it catches is the
theme's, not a docs nit, and it does not ship with one unfound.

`tools/stage-release.sh` is our half — the `pre-release` hook. It writes the release notes where the
workflow reads them and puts the one non-package asset into `dist/`, before the manifest is written,
because whatever is in `dist/` is signed with everything else:

- **the notes**, from `tools/release-notes.sh` — the tag's changelog section, one bold lead per
  bullet, grouped by category. They fill the release page and are **not** an asset: the only reader
  that ever fetched `notes.md` from a release was the self-update package, which is retired and its
  repository archived;
- **the installer**, because `raw.githubusercontent.com` is rate-limited for unauthenticated callers
  — so the user whose address has run out of budget (CGNAT, a shared exit) fails to download the
  installer meant to rescue them (issue #17) — and because a signed copy is the only one that can be
  checked before it runs as root. The README points at `main` and names this one as the fallback.

`sign-also: install.sh` gets it a signature of its own (it is not a package, so owfeed would not see
it otherwise), and `verify-with: release.pub` re-checks every signature against the key that is in
the repository and baked into the installer.

Result: one theme asset per format plus its `.sig`, the manifest plus its `.sig`, and the installer
plus its `.sig`. Each has a reader: the packages are what a router installs, the manifest is what
`install.sh` installs *from* when the feed cannot be read and where it reads the newest version for
its "the feed has not caught up yet" line, and the installer is the mirror the README names.

**About the manifest and readers in the field.** owfeed's format was copied from this project's own
manifest, with two differences, both safe by construction: the first line became
`owfeed-manifest 1` (nobody parses it), and the package architecture was appended to the end of
the `pkg` line. The order of the first six fields is untouchable: `install.sh` reads them positionally,
and a copy already on somebody's router cannot be fixed remotely — a field inserted before them
would make it fetch a URL that 404s. The workflow checks that order on every release rather than
relying on it.

## `playground-asset` — attaching the built playground to the release

Tags only, `needs: [release, playground]`. `dist/*` rides the manifest `release` signs, and
`playground.tar.gz` is a demo site rather than router payload — owfeed's `package.yml` has no input
for a foreign artifact, and putting one there would sign it as if it were one — so this is a plain
`gh release upload` against the tag `release` just published, in a job of its own with
`permissions: contents: write` and nothing else. `pages` waits on this job, not on `release` directly,
so its own fetch of `playground.tar.gz` (below) never races the upload that puts it there.

## `pages`

Publishes the developer portal to GitHub Pages: `docs/devkit.html` (generated by
`tools/devkit-build.mjs`, never committed) and `docs/gallery.html`, with a freshly built
`cascade.css` and `logo.svg` beside them. It also carries a **full mirror of the
latest release** — the manifest, its signature, the installer **and the packages** — on a different
host from github.com, so an outage or a block covering one need not cover the other. It is called by
`release` (by way of `playground-asset`) rather than triggered by `on: release`, because an event
raised by `GITHUB_TOKEN` does not trigger another workflow.

The devkit assembles itself from files that already exist — the real stylesheet, the export tier
parsed out of `02-tokens.css`, the widget markup from `gallery.html` — so nothing is hand-copied and
nothing can drift.

**The playground is not built here at all.** It is not source that lives in this repository; it is
fetched, already built, from `releases/latest/download/playground.tar.gz` — the same
`playground-asset` upload above — and unpacked into `_site/playground/`. Fails OPEN like the release
mirror above it: a repository with no playground-carrying release yet (or a `workflow_dispatch` run
on a branch, `docs/development.md`) publishes the rest of the portal and says so in the log rather
than failing the build. `_site/playground.html` is kept as a redirect to `playground/` for the links
the README and `devkit.src.html` already carry.

**A maintainer can publish Pages from a single run's own recording, without a tag:**

```sh
gh workflow run build.yml --ref <ref> -f publish-pages=true
```

This runs `build.yml`'s `pages-manual` job (`needs: playground`, off by default), which calls
`pages.yml` with `playground-source: artifact` — everything above is unchanged except the playground
fetch, which reads the `playground` artifact this same run uploaded instead of
`releases/latest/download/playground.tar.gz`, and FAILS CLOSED on a miss rather than publishing the
rest of the portal. Two things to know before using it:

- **The `github-pages` environment accepts only `main` or a `v*` tag as a deployment branch**
  (repo Settings -> Environments -> github-pages; checked 2026-09-14 with
  `gh api repos/…/environments/github-pages/deployment-branch-policies`:
  `custom_branch_policies` = `[main, v*]`). `--ref` naming anything else reaches `deploy` — `build`
  and the artifact fetch already ran — and is refused there.
- **The next push to `main` under the paths `pages.yml` watches overwrites a manual publish.** That
  run fires from `pages.yml`'s own push trigger, `playground-source` defaults back to `release`
  there, and Pages goes back to mirroring the latest tag.

## Installation and the trust chain

`install.sh` does one thing: it adds the owfeed-packages feed (key, repository entry, and a
`keep.d` entry so a sysupgrade does not lose the key) and installs the theme from there.
`apk upgrade` / `opkg upgrade` carries it forward afterwards, which is the whole reason to install
from a feed rather than from a file.

It then installs the catalogue for `luci.main.lang`, and **the feed not carrying that package is not
a refusal**: a release reaches owfeed-packages through a pull request, so a NEW package — which
`luci-i18n-footstrap-<lang>` was in 0.14.4 — is missing from the feed until that merges, while the
signed asset for it is already in the release. The fallback takes it from the release through the
chain below, pinned to the tag the installed theme came from, and `apk upgrade` picks the package up
from the feed once it lands there.

```sh
wget -qO- https://github.com/VizzleTF/luci-theme-footstrap/releases/latest/download/install.sh | sh
```

**The repository line goes into the manager's own customfeeds file**, `customfeeds.list` for apk and
`customfeeds.conf` for opkg — not into a file of the theme's own. apk reads every `*.list` under
`repositories.d/`, so a private file installs and upgrades just as well; what it cannot do is be
seen. LuCI's package manager reads exactly three apk paths — `repositories`,
`repositories.d/distfeeds.list`, `repositories.d/customfeeds.list` — in its rpcd ACL *and* hardcoded
in `package-manager.js`, so a feed anywhere else is absent from "Configure APK" and cannot be edited
or removed there. An installer from before this wrote `repositories.d/owfeed-packages.list`, which
is why the apk branch deletes that file after writing its own line — the same repository configured
twice, once where the admin can see it and once where they cannot, is worse than either. Neither
customfeeds file needs a `keep.d` entry: both are conffiles of their manager (`apk-mbedtls`, `opkg`),
sysupgrade backs up every conffile whose checksum has moved, and `build_list_of_backup_overlay_files`
was already dropping the duplicate entry the script used to add.

**The line is MOVED to be the first non-comment line, never merely appended or left wherever it
already was** — measured on real apk: its customfeeds reader stops at the first line it cannot
parse, so an admin's own unrelated line (or a stale one from an older run) sitting above where the
correct line used to land could cut the feed off with no error at all, and a router this project's
own OLDER installer had already configured has exactly that shape (the line appended, after
whatever else was there) — so "is the line present" has to become "is it first", checked and
repaired on every run, not only when the line is missing outright; the closing message distinguishes
the three outcomes ("Adding"/"Feed added" when the line was missing, "Moving … to the top" when it
only had to be repositioned, "already configured" when nothing changed). Every other ACTIVE line
naming the feed's host (case-insensitive) or, on opkg, its src name, is commented out and reported
rather than left to shadow the correct one; a line for any other host is untouched.

Both this and the stale-line cleanup write the result atomically and never `mv` a fresh file over the
customfeeds path directly: they resolve it with `readlink -f`, `cp -p` its mode and owner onto a temp
file in the SAME directory (so the rename stays on one filesystem), write the new content into that
copy, and only then `mv -f` it over the resolved target. A customfeeds file that is itself a symlink
keeps being one — the rename replaces what it points at, never the link — and a write that fails
partway (a full overlay, OOM, a `cp`/`cat`/`mv` a hostile PATH entry shadows) leaves the original
byte-identical, because nothing is truncated until the copy is complete and the rename is what
actually takes effect. **The failure itself is checked, not inferred**: `disable_other_lines`'s
"disabled:" lines wait until the write actually lands before printing, and `ensure_first` reports
its outcome through a real exit status the caller tests with `if`, never through `$(…)`, whose own
exit code `set -e` cannot see — a failure to write the feeds file now exits 1 with one error naming
the file, leaving it unchanged and no temporary files behind, instead of printing "Adding the
feed…"/"disabled: …" over a write that silently never happened (measured on both package managers
with `cp`, `mv`, and — on opkg — `cat` shimmed to fail; on apk a failing `cat` is intercepted
earlier, at reading `/etc/apk/arch`, and reported there instead).

**A snapshot router is served the newest release branch.** The feed publishes one branch per
OpenWrt minor and has no snapshot channel — owfeed-packages lists exactly two release lines and they
*are* the package-format split (apk from 25.12, ipk on 24.10), not a build of the theme per release.
So `SNAPSHOT`, which parses to no branch, gets the newest branch its own package manager can read:
`FALLBACK_BRANCHES_APK` / `_OPKG` in the script, probed newest-first against
`releases/<branch>/<arch>/<index>` rather than assumed, so a branch listed before it is published —
or one that does not carry this router's architecture — falls through instead of writing a
repository entry that 404s on every update. What makes it sound here and not in general: the theme
is noarch and `+luci-base` is its whole dependency list, so nothing in it was compiled against the
branch it comes from. The probe's bytes are discarded; the index it found is still verified below.
When no candidate answers, the script installs the signed release instead (below) rather than
refusing — measured in an `openwrt/rootfs` snapshot container (`apk add` from the 25.12 branch, no
`--allow-untrusted`, theme registered) and with the feed host pointed at a name that does not
resolve.

**Every downloader on the box is tried, not the first one that exists.** `uclient-fetch` needs
libustream-mbedtls to speak https at all, and an image that ships wget-ssl or curl instead carries
the binary without the library — so choosing by existence reported "the feed carries no branch for
this router" when the truth was "this one tool cannot do TLS here". Reproduced with a
`uclient-fetch` stub that always fails: the old script printed exactly the field report, the current
one falls through to `wget` and installs from the feed. None of the three is ever asked to skip
certificate verification; falling through to the next TOOL is not a downgrade.

**What verifies the bytes on the feed path is the package manager**, against the feed key pinned in
the script: apk checks the index against `owfeed-packages.pem`, opkg against usign key
`9040356b214084da`.

**The theme is now also carried by the official openwrt/luci feed, and a bare `apk add` picks that
build instead of owfeed's.** apk resolves a name it is given with no version constraint to the
HIGHEST version across every configured repository, not the one the repository line just added
serves — and luci.mk stamps a LuCI-carried package from git's commit date rather than this
project's own numbering. Measured on owrt2512 (OpenWrt 25.12.4, apk-tools 3.0.5):

```
# apk list luci-theme-footstrap
luci-theme-footstrap-0.14.10-r1            noarch {luci-theme-footstrap}   ← owfeed
luci-theme-footstrap-26.246.70755~4fd72fd  noarch {feeds/luci/…}           ← official feed
```

`install.sh`'s `apk add --upgrade "$PKG"` moved a router onto the second line — reported as
"Upgraded luci-theme-footstrap 0.14.9-r1 -> 26.246.70755~4fd72fd", which is not an upgrade: that
build predates this release (no rpcd-reload fix, no checkbox-tooltip fix) and comes from a
different publisher. The fix is a version constraint the script now applies to both the theme and
its `luci-i18n-footstrap-*` catalogues, `apk add --upgrade "$PKG<26"`, which both picks owfeed's
build over the official one and REPAIRS a router that already has the official one (`--upgrade`
still applies, so it downgrades, measured: "Downgrading luci-theme-footstrap
(26.246.70755~4fd72fd -> 0.14.10-r1)"). `26` is a ceiling, not a version pin: `apk version -t`
confirms `26.246.70755~4fd72fd` and `27.1.1~abc` both compare `>` against `26` — the LuCI stamp
only grows with the calendar — while `0.14.10-r1`, `1.0.0-r1` and `25.99.99-r1` all compare `<`,
so the constraint excludes every LuCI-stamped build for good and leaves this project's own
numbering majors 1 through 25 to grow into. opkg has no `world` file and no version-constraint
syntax on `install`/`upgrade`, so this has no opkg equivalent; left alone because the official
24.10 feed does not carry the theme today (checked on owrt2410, `opkg info luci-theme-footstrap`
names only this project's own `0.14.9-r1`) — if that changes, the fallback is the signed-release
install path this script already has (`install_from_release`), which bypasses feed resolution
entirely.

**There is no pinned-tag install, and the release-asset fallback is automatic but never the first
choice.** A router that cannot read the feed index at all — an architecture owfeed does not publish,
a resolver that does not answer, a network that intercepts the host — is installed from the release
instead of being sent away with a URL, and told plainly that `apk upgrade` will not carry the theme
forward until the feed works.

That path picks the artifact **from the signed manifest**, never by guessing an asset's name: issue
#6 was a self-update script, shipped at the time and retired since, that resolved the theme by
name and took `head -1`, which on a release
carrying per-language packages installed a catalogue. `manifest.txt` names exactly one file per
format with its size and digest, and the chain fails closed in this order — verified TLS, then
`usign -V` against the release key pinned in the script (the same key as `release.pub`), then the
manifest's own sha256 over the downloaded artifact. A missing `usign`, a signature that does not
verify, or a digest that does not match is a refusal, and the script then prints the by-hand URL as
before. `apk`'s `--allow-untrusted` on this path says only that the `.apk` carries no APK signature
of its own; what is trusted is the usign signature over the manifest, checked before the file is
handed to the manager. Measured on 25.12/apk and 24.10/opkg containers, including the refusal: a
one-character change to the pinned key leaves nothing installed.

A router that wants an OLDER version still does it by hand — the feed carries one version per
branch — with the **raw file from a release**, never the zip artifact from Actions:

```sh
apk add --allow-untrusted luci-theme-footstrap-*.apk   # 25.12+
opkg install luci-theme-footstrap_*.ipk                # 24.10
```

`--allow-untrusted` means the package manager holds no key of ours. The release publishes a signed
`manifest.txt` and a detached `.sig` per asset, so a by-hand install can be checked with
`usign -V -m <pkg> -x <pkg>.sig -p release.pub` — which is what the installer's own fallback does
for you.

## Package formats, and the usual mistake

- **apk** (25.12+) — the Alpine apk-tools format, which OpenWrt adopted in 25.12.
- **ipk** (24.10) — gzip-tar: `./debian-binary` (2.0) + `./control.tar.gz` + `./data.tar.gz` with
  ustar headers. Identical to native OpenWrt ipks.

**"Malformed package file" from opkg** is almost always the wrong file: a GitHub artifact's zip
wrapper, or an `.apk` where an `.ipk` belongs.

An `.ipk` can be checked with a real opkg by hand — there is no such run in CI, and there was no
real proof before either, since "an SDK 24.10 produced it" was the same non-proof:

```sh
docker run --rm -v /path/pkg.ipk:/tmp/p.ipk openwrt/rootfs:x86-64-24.10.4 \
  sh -c 'mkdir -p /var/lock; opkg install --nodeps /tmp/p.ipk'
```

**All four lifecycle scripts are present in both containers.** The first version of owfeed emitted
only `postinst` and `prerm` into the ipk — the pair `package-pack.mk` generates, and easy to mistake
for the whole set — so `postrm` (unregistering the theme, putting `mediaurlbase` back) reached apk
and not opkg. Fixed before the first release built with it, and checked against the 0.11.6 SDK build:
all four agree in meaning, with only cosmetic differences.