Description
The three coordination rules shipped 2026-09-04 as advisory and non-paging on purpose: nothing had watched them evaluate. The owner chose a cadence rather than a prediction. On or after 2026-09-11, after seven days of live evaluation, bring him the per-rule firing history and he decides which of CoordinationNoLeader, CoordinationSplitBrain and CoordinationNoStandby acquire paging labels.
A cadence cannot be wrong about the future the way a guess can, which is why this is a dated task rather than a threshold written into the rules now.
Bring, per rule: how many times it fired, how long each firing lasted, and whether each firing corresponded to something real. CoordinationNoStandby in particular is expected to fire permanently against any single-replica coordinated deployment, so separate that from genuine firings before presenting the history.
Acceptance Criteria
- #1 Per-rule firing history since 2026-09-04 is gathered from the stack and presented with each firing classified as real or spurious
- #2 The owner’s per-rule paging decision is recorded, and any rule he promotes gets its label change shipped and proved by a real gcx resources push
- #3 If a rule proved untrustworthy it is fixed or withdrawn rather than left evaluating and ignored
Definition of Done
- #1 just check passes (the full gate; it is what CI enforces)
- #2 just gen leaves no diff (only if a generated artifact’s inputs changed)
- #3 just –fmt –check passes and every new recipe has a # doc comment and a [group(…)]
Implementation Notes
Finding 2026-09-06: the date was not a real dependency. Nothing on the lab runs coordinated mode (one replica, no coordination block), and the only coordination series in the last 30 days came from the temporary Wave 11 sibling, since deleted. The four rules have been evaluating NoData and would still be on 2026-09-11. Owner decisions 2026-09-06: enable coordination on the lab for real, chart first (TSO-0143 per-replica persistence), then flip the lab; this task re-gates on seven days of evaluation after that flip. The paging labels are the owner’s IRM overlay, but he still holds PR #585 until this task lands.
Wave 16 (goal codex/goal-2026-09-06-wave16-*.md) flips the lab to two coordinated replicas after TSO-0144 ships. The seven-day clock starts at the timestamp the wave records here when both pods are Ready and the coordination series are present on the stack.
Soak started 2026-09-06T14:31:44Z: the lab runs two coordinated replicas (chart 0.35.0, image 5.0.0-rc.36) with identities tailscale2otel-0 and tailscale2otel-1; tailscale2otel_coordination_leader_ratio is present for both and the four coordination rules now have data. Earliest review: 2026-09-13. Observation from the flip day: an ordinary leader pod deletion returned the same identity within the Lease and reacquired it, so no handover was counted and the standby never took over; a real survivor takeover was proven only on Kind (about 19 s to the sole Ready endpoint at the default timings).
Owner guidance 2026-09-06: no fixed review date; leave the soak running and revisit when convenient, since a paging alert would be noticed anyway. Takeovers may be simulated at will (cordon the leader node, delete the leader pod, uncordon). First real takeover simulated 2026-09-06T14:48:03Z: the standby held the Lease, carried the label and was the sole Ready endpoint 21 s after the deletion at the default 15s/10s/2s timings; the old leader returned Ready as a standby without the label once its node was uncordoned.
Finding from the first simulated takeover (2026-09-06 14:48Z): the summed leader gauge read 2 from 14:49:30Z to 14:53:00Z because the killed process last coordination_state=leader sample lingers for the five-minute Prometheus lookback while the restarted pod reports a fresh standby series under the same identity. CoordinationSplitBrain (sum > 1 for 5m) sat one evaluation short of a false advisory alert; a rolling restart does not double count (same identity, same state series) so only a real takeover triggers it. CoordinationNoStandby is protected by its 10m for-window; CoordinationNoLeader cannot fire on a total outage because noDataState is Ok, only on live processes reporting no leader. Candidate fix for the review: wrap the leader gauge in last_over_time(…[2m]) in build_rules.py so a dead process series drops out after two export intervals instead of five minutes.
Correction 2026-09-06: promtool shows plain sum cannot meet the 5m for-window at any evaluation alignment because staleness is also exactly 5m, so no false CoordinationSplitBrain alert is reproducible offline; the live window today was about four minutes of Pending. Owner chose to fix anyway: the three leader-gauge rules now read last_over_time(…[3m]), which drops a dead process two export intervals before the for-window and restores a real margin against evaluator jitter. The fixture “split brain ignores a dead leader stale sample after a takeover” documents the shape but passes on both expressions.