Wave operating model

This document carries only what is specific to opnsense2otel. The campaign model itself — run modes, the routing contract, authority and the thread pool, child lane briefs, external-contract freezing, the unattended blocker contract, the goal-file template, the pre-flight checklist — lives in the Agent fan-out protocol (canonical) doc and is not repeated here. If something below could be pasted into another repo unchanged, it is in the wrong document.

Run-end against this tracker

The goal file does not enumerate the work. The queue is:

backlog task list --plain -s "To Do"

Task state is the record, not a report file:

• landed work is Done with the commit SHA in its final summary;
• blocked work is Parked with a concrete resume boundary — the next action, not “blocked on X”;
• untouched work is self-evidently still To Do;
• discovered work is a new task labelled needs-triage.

The run’s closing message goes to the terminal as a covering note: what did this run learn that no single task captures. Nothing durable may live only there. Writing it is the last unit of work, not a reply to a request.

Acceptance criteria live on the task, so a lane reads its own contract with backlog task view <id> --plain and the goal file stops restating them.

Gates

backlog/config.yml’s definition_of_done seeds every task with two gates:

just check
just gen        # only if a generated artifact changed; commit the diff

just check is the whole bare-toolchain gate and it is not the old five-command list. It runs fmt-check lint test metric-lint fuzz-smoke check-public-ips testbed-test canary-test grafana-test gen-check vuln, so the two conditional additions a lane used to have to remember — grafana-test when touching grafana/, testbed-test / canary-test when touching scripts/testbed/ or scripts/canary/ — are already inside it. There is nothing conditional left for a lane to add.

Every make target this repo once had is gone (OPN-0006). make lint, make test, make check-public-ips, make docs-check, make grafana-check and make grafana-test do not exist; a lane that reaches for one gets command not found, not a gate. Discover the surface with just --list, never from memory.

The heavy CI-only gates — race-detector matrix, docker build, helm-in-kind, deployment contracts, cross-compilation — are not a lane’s job. just ci adds the subset that needs Docker or cross-compilation locally. Do not skip a gate by claiming it is CI-only; say the change is untested against it and let CI answer.

auto-rc classifies only the head commit of a push. Its shipped-change job diffs HEAD^..HEAD, so a push whose last commit touches only scripts/, docs/ or backlog/ cuts no release candidate even when the commits under it changed shipped Go. Push the shipped-code commit last, or on its own, when a lane needs a candidate build (the FreeBSD receiver can only take a binary from a release URL).

A major release needs the module path moved first. TestModulePathMatchesReleaseVersion turns the release-please PR red the moment the manifest passes the module’s major; just bump-module-major major=<N> is the repo’s own rewrite and is a [confirm] recipe, so ask before running it. It also rewrites backlog/ task notes that quote old import paths; restore those, they are history.

CodeRabbit: review the source, not the generated tree

A whole-wave review does not fail, it disconnects. Past about 100 files the CLI drops the socket during the connect phase and emits {"type":"error","errorType":"connection","message":"Connection failed: WebSocket closed"} with no complete event. It never reaches analysis, so there is nothing to triage and nothing to retry usefully — a second attempt fails identically. This is not a service outage and not an auth problem: coderabbit doctor passes all nine checks while it happens, and a one-file review against the same service, account and repository completes normally in the same minute.

Wave 2 read two of those as a hard gate failure and parked seven finished, gated, reviewed lanes. The whole loss was avoidable.

Split the review by what the review policy already covers. Generated artifacts are outside the gate anyway — grafana/dashboard*.json, grafana/sentinel-contract.json, grafana/runbooks.md, grafana/tabs/AUTHORING.md, grafana/alerts/grafana-managed/, opnsense/testdata/schemas/, docs/, README.md. On a wave-sized change those are most of the bytes and none of the branching: in wave 2 they were 4,957 of 8,180 changed lines, dashboard.json alone 4,642. Reviewing the 48 source files completed and found six findings, two of them major.

Do it in a scratch worktree so the real checkout is never disturbed:

git worktree add -f --detach /tmp/cr-src origin/main
cd /tmp/cr-src && git apply <the wave patch>
# restore generated paths to base: checkout the tracked ones, rm the new ones
git add -A && coderabbit review --agent --base main

If the source slice is itself too large, split it again by package. Every changed source file must appear in some completed review; record which review covered which files.

A slice has one cost, and it is predictable: a finding raised against a file you excluded is a false positive, because the reviewer is reading a tree where that file was never updated. Wave 2’s log-shipping.md source-table finding was exactly this. Check any docs- or generated-file finding against the real tree before acting on it.

Exclusive resources — serialise these, never fan out across them

The testbed is a single physical resource, powered on demand. Six guests on oli, off by default. Nothing raises the lab on a schedule and nothing dispatches live-canary.yml on a schedule: opnsense-testbed-up.timer and opnsense-testbed-canary.timer are installed on oli but deliberately disabled. Do not re-enable them. When a change needs validating against a live API surface, the main session raises the lab, dispatches the canary and takes the lab down again. opnsense-testbed-down.timer stays enabled as a backstop: at 08:30 UTC daily it powers the lab off unless a hold is live.

Consequences for a wave:

• Raising the lab is a main-thread action. A lane never powers the testbed; it reports that it needs live evidence and stops.
• At most one consumer may use the testbed at a time. There is no locking; two runs will interleave API writes against the same firewall and produce results neither can trust.
• The box is down unless someone raised it. A pre-flight probe failure on a run nobody raised the lab for is not drift, not a box fault and not a regression. Do not open a task for it.

Raising the lab. Authorised, including for an unattended run. oli is reachable over the tailnet as root and carries the power scheduler:

ssh oli '/usr/local/bin/opnsense-testbed-power.sh status'      # hold state + every guest
ssh oli '/usr/local/bin/opnsense-testbed-power.sh up 7200'     # start in dependency order, 2h hold
ssh oli '/usr/local/bin/opnsense-testbed-power.sh release'     # clear the hold
ssh oli '/usr/local/bin/opnsense-testbed-power.sh down'        # refuses while a hold is live

Always pass up a hold sized to the work: without a live hold the 08:30 UTC backstop powers the lab off mid-run. up blocks until both firewalls serve :443, so its return is the readiness signal; never poll for it yourself. The session that raised the lab owns release then down when it is finished; a forgotten hold lapses on its own and the next 08:30 backstop takes the lab down.

Never call qm or pct on oli directly. The scheduler’s hardcoded allowlist (102, 106, 105, 110, 111, 112) is the only thing standing between a typo and powering off home automation (100), the CI runners (101), unifi-os (103), winsrv (104) or postgres (107). Drive power through the script, always, and never derive an id from qm list.

The testbed firewalls’ API credentials are not on any laptop — they live only in the repository’s tailnet GitHub environment (DEVBOX_API_KEY/DEVBOX2_API_KEY, and the DEVBOX2 pair for the release box). So local just capture and just run cannot reach the testbed, and the only route to live evidence is to raise the lab and dispatch the canary, which holds those secrets:

gh workflow run live-canary.yml --ref main
gh run list --workflow live-canary.yml --limit 1 --json databaseId,status,conclusion

That gives a structure verdict, not a payload — cmd/apidrift writes its captures to the runner’s temp and nothing uploads them. The consequence is a two-pass loop for any new endpoint: land the struct derived from upstream source, push, dispatch the canary, and read its verdict against the golden schema. A lane that needs to see a real payload cannot get one this way, and should say so rather than inventing one.

There is no production-firewall canary any more. OPN-0106 retired it on 2026-09-20: the camden timer, its service and the prod probe profile are all gone, and Rob’s production firewall is no longer probed by anything in this repo. It had reported byte-identical output on all 27 runs and ran the same OPNsense build as the release lab box. A lane that finds a reference to it in an older task, an exemption note or the CHANGELOG is reading history.

Grafana Cloud is live. just grafana-check regenerates and diffs local artifacts and touches nothing remote. Pushing dashboards or rules to a stack is a main-thread action, not a lane’s.

Loki cannot show you a historical record for up to two hours. The querier does not send a query to ingesters beyond query_ingesters_within (3h), so a log entry stamped older than that reads only from the store, and it reaches the store when its ingester chunk flushes. A configchange diff replayed from a seeded cursor carries the retained revision’s timestamp, hours or days old; three proof runs shipped one each, queried within a minute, and saw nothing, while the records were sitting unflushed. A proof that asserts its own historical record within the run can never pass. Assert on the previous instance’s records instead, or report visibility-pending rather than absent, and read logs_shipped_total, logs_dropped_total{reason} and partialSuccess off the exporter for the drop-or-not question. Entries older than the tenant’s reject_old_samples_max_age are rejected outright and now surface as logs_dropped_total{reason="rejected"}.

Guest access is through the power script’s exec and put subcommands (since wave 9), never qm or pct directly. put is container-only because qm has no guest file-write; a VM fetches a release archive itself with exec <id> -- fetch -o <tmp> <release-url> and verifies the sha256 in-guest against checksums.txt. Both testbed firewalls run the QEMU guest agent, and 105 and 112 carry python3 and curl. The exporter starts on a guest with no real credential: --opnsense.api-key is a presence check only, and with --exporter.instance-label set startup makes no API call, so --opnsense.address 127.0.0.1:1 plus a dummy key and secret gives a receiver whose collectors merely fail. Bind receivers on the LAN-side segment (105 eth1 with 112, 102 vtnet2 with 105 eth0); the WAN side is behind pf and would measure the firewall, not the receiver.

v4.2.0 cannot start with the syslog receiver on a stock FreeBSD kern.ipc.maxsockbuf. The kernel refuses a 4 MiB SO_RCVBUF with ENOBUFS instead of clamping (OPN-0101, fixed at a86cbb65). Any FreeBSD trial of the receiver needs a binary at or after that fix.

Ownership — the append-only registries

Adding a collector or an endpoint touches a fixed set of shared files that every lane wants to append to. One owner per file, or a single wiring pass at the end. The registries, in the order the collector checklist in AGENTS.md hits them:

Counts pinned in tests move with these (TestNewClient_EndpointCount, the golden POST count, the docs count pins, the rule-count pins). A lane that appends without bumping its count leaves the build red for everyone.

Escape hatch. A lane that needs a change outside its boundary states the exact edit and stops — it does not make it, and it does not silently work around it. A boundary with no escape hatch is a stop condition wearing a safety label.

Recurring defects in this codebase

Five classes, each with instances. These are what a review pass should actually look for.

1. Modelling a payload shape upstream cannot produce

The single most expensive class here. A struct is written against one observed response and the branch it invents is never taken.

• metadata.subsystems was modelled as “the 26.1.11 shape”; upstream never populated it on any release. Cost two fabricated fixtures and a permanently dead branch (#284).
• unbound query-class counters decoded as a fixed IN field, when it is a map.
• nginx overCounts modelled as one union struct, when it is per zone kind — and as a number, when it is an object.
• smartctl wear percentages decoded as numbers, when they are objects.
• endurance_used given a threshold_percent it does not have.

The rule that prevents it: verify against the OPNsense controller/script that builds the payload, and check whether the key is conditional, before writing the struct. The canary triage verdicts in AGENTS.md are the decision procedure; the trap is reaching for drop/chase before asking whether upstream changed at all. box-state was the right answer 7/7 in #271 and 7/7 in #243.

2. Flow correlation: orientation, pairing and double-counting

internal/flow/ has produced more corrections than any other package. Every one is a case where two records that describe the same connection were treated as independent, or one record’s direction was inferred from something that is not evidence.

• orientation decided by arrival order rather than by evidence;
• merge endpoints paired by position rather than by address;
• NAT’d conversations counted twice, then paired by conversation only when the exact window cannot close;
• a merged record’s two halves not split into Tx and Rx;
• a window partial compared against a whole connection without stating the byte basis;
• repair markers not unioned across a conversation’s fragments.

Assume any new flow logic double-counts or mis-orients until a test says otherwise. This is the one area where test-first is not negotiable regardless of change size.

3. Alert rules that fire on a single sample

Rules authored against one observed spike, with no sustain and no hysteresis.

• OPNsenseDHCP6AllocationFailures fired on a single event;
• OPNsenseNetmapRingFull meant one burst, not a sustained condition;
• the gateway flapping rule had no hysteresis;
• OPNsenseFlowSourceDivergence was deleted outright — its threshold sat below the metric’s floor, so it could never mean anything.

Before adding a rule, state what the metric’s floor and normal range actually are. A threshold that cannot be crossed, or that any single sample crosses, is worse than no rule.

4. Decoded and dropped

A per-row payload is unmarshalled and its identifying dimension quietly discarded (#544). This has a mechanical detector: just fieldaudit reports struct fields in package opnsense that are unmarshalled and never read, and the same analysis runs as a unit test, so just test already gates it. If you add an exemption, write the reason.

5. Public metadata leaking into a public repo

This repository is public and backlog/ is now committed with it.

• just check-public-ips rejects any globally routable IP literal in source, tests, fixtures or docs without a justified entry in scripts/public-ip-allowlist.json (#565). RFC1918, loopback, link-local, CGNAT, multicast and the RFC 5737 / RFC 3849 documentation ranges are never flagged.
• Golden schemas are structure-only — key paths and JSON types, never response values.
• Credentials have twice reached places they should not: request headers into a capture, and a bearer token surviving a redirect. Redact at the boundary, not at the sink.

This now extends to the tracker, and just check-public-ips scans backlog/ like any other committed content — verified 2026-08-14 by writing a task that quoted the offending literals from OPN-0002 and watching the gate flag 18 new violations in the task file. A task about a leaked address must describe it, never quote it: give file:line and say what kind of address it is, and let the reader open the line. The same holds for docs.

Tasks and docs must not carry real account identifiers, hostnames beyond the ones already public in this repo, device names, addresses, or data-archive cell values. Write the shape, not the instance. Sweep before committing:

grep -rniE "rob-knight\.net|@gmail|@rob-knight|github_pat_|[0-9]{1,3}(\.[0-9]{1,3}){3}" backlog/ \
  && echo "REVIEW THESE"

Tailnet host names (camden, oli) are already all over this repo’s scripts and issue history, so they are not the thing to hunt for. Tokens, IPs and account identifiers are.

Lane conventions

• Never two lanes on one task. The v1.50.x fix covers the task edit funnel but not reorder, draft saves, the TUI path, doc update or decision updates.
• --append-notes and --append-plan only. Bare --notes / --plan silently replace the whole section and destroy another session’s writes with exit code 0. A PreToolUse hook on Rob’s machines denies the bare form, so being blocked there is the guard working. It does not fire in CI or on a machine without that config, which is why the rule is written down as well.
• Finalize in one call so an interrupted lane cannot leave finished work looking unfinished: backlog task edit OPN-0007 --check-ac 1 --check-ac 2 -s Done.
• Commits come from the campaign root only. Lanes do not commit. auto_commit is off.
• The index is shared. A lane that stages files can put them into the root’s next commit. Wave 8 committed a half-finished console because the console lane ran git add while the root was staging an unrelated batch and then committed bare. Lanes never stage; the root commits with an explicit pathspec (git commit -- <paths>), and reviews git diff --cached --stat against the list of files it meant to commit before every commit.
• Generated artifacts are regenerated, never hand-edited — anything between docgen markers, grafana/dashboard*.json, the grafana-managed manifests, opnsense/testdata/schemas/. A lane that hand-edits one produces a diff that just docs-check or just grafana-check reverts on the next regeneration, silently losing the work.

Issue numbers below #656

They are GitHub issues, not tasks. See the Pre-backlog issue numbers doc.