Description
The active-passive Kubernetes Lease feature shipped across Waves 5 through 9 with zero alert rules. deploy/alerts/gen/build_rules.py contains no coordination rule and no file under deploy/alerts/grafana-managed/ mentions coordination, so nothing alerts on the HA path failing.
The signal-coverage gate did not catch this because tailscale2otel.coordination.leader is on the health dashboard, which satisfies the visualized disposition. alertable is a separate value and being false is permitted, so a signal can be fully panelled and still have nobody watching it out of hours.
The states worth alerting on, at least:
- No leader. Nothing holds the Lease, so nothing is collecting. This is the outage case and it is detectable from the shipped signal today.
- Split brain. More than one replica reports leadership at once, which means duplicate collection and double-counted logs. Also detectable today.
- Leadership flapping. Repeated handovers in a short window, which usually means the renew deadline is too tight for the API server’s latency rather than a real failure.
- No standby available. Only detectable once TSO-0119 makes standby state observable on the deployment’s delivery path, so treat it as conditional on that task landing.
Alert authoring in this repository has a hard rule: gcx resources validate does NOT validate the spec, and neither does the offline validator or promtool. Only a real gcx resources push proves a rule is deployable, and pushing is pre-authorized. A previous change had 19 rules agree with each other offline and fail at push time on execErrState casing.
Acceptance Criteria
- #1 A no-leader rule and a split-brain rule are generated by build_rules.py, shipped as Grafana-managed manifests, and carry the repository’s label taxonomy
- #2 Every new rule ships non-paging: advisory severity, no page label, so it cannot wake anyone before a human has watched it evaluate for a while
- #3 A leadership-flapping rule is either shipped or explicitly rejected with its reason recorded, since a too-tight renew deadline is the likeliest real cause
- #4 The no-standby rule is shipped only if TSO-0119 landed first; if it did not, that is recorded as deferred rather than silently dropped
- #5 A real gcx resources push succeeded and its result was read; just verify-deploy reports in sync
- #6 The Prometheus rule artifact regenerates and promtool test rules executes the new expressions against fixtures
Definition of Done
- #1 just check passes (the full gate; it is what CI enforces)
- #2 just gen leaves no diff (only if a generated artifact’s inputs changed)
- #3 just –fmt –check passes and every new recipe has a # doc comment and a [group(…)]
Implementation Plan
- Add generated advisory, non-paging no-leader and split-brain coordination rules using the repository label taxonomy.
- Decide leadership flapping explicitly and either generate it with executable fixtures or record the reason for rejection.
- Add no-standby only after TSO-0119 has landed; otherwise record an explicit defer and resume boundary.
- Regenerate Grafana-managed and Prometheus artifacts plus alert-profile docs, validate offline, and execute promtool fixtures. Root alone updates shared catalog artifacts, pushes rules with gcx, verifies deployment, commits, pushes, checks CI, and finalizes.
Implementation Notes
Lane implementation evidence: generated enabled advisory, non-paging CoordinationNoLeader and CoordinationSplitBrain rules with the observability label taxonomy and per-Lease/namespace aggregation. TDD boundary test failed before catalogue entries, then just gen-dashboards, just gen-promrules, 129 Python generator tests, just rules-check (126 rules plus fixtures), and git diff --check passed. Leadership flapping was rejected for this task: the only available signal is a synchronous last-value gauge whose changes() reflects individual-series/retention churn, not completed leadership handovers; a truthful rule needs a handover counter or event stream. No-standby was deferred in this first pass because TSO-0119 had not yet landed, as AC4 requires. Offline checks do not prove Grafana deployability; root still owns the real push and verify-deploy.
After TSO-0119 landed as 15e4777d, the conditional no-standby rule was added. CoordinationNoStandby sums all state-labelled samples per identity before counting zero-only contenders, so a promoted leader ’s retained zero standby series does not count as a live standby. Its 10m for tolerates stale backend samples after replica loss. Promtool fixtures prove a lone leader fires after the window while a leader plus standby in another namespace stays quiet. Final focused evidence: 130 lane tests passed before the follow-up repair; just rules-check validated 127 rules and all fixtures.
Integrated as 74a8c924 after TSO-0119 landed. Root pushed with gcx –context m7kni resources push -p deploy/alerts/grafana-managed: 129 resources pushed, 0 errors. just verify-deploy m7kni then read back 128 shipped and 128 deployed rules, 81 paused on each side, with zero missing, orphaned, or drifted. The cumulative full gate passed at 0e212ab5; integrated generation left no diff and formatting passed. Exact-head CI run 33844779329 attempt 1 succeeded for 630b1d75 before the later fallback fix.
Final Summary
Shipped enabled advisory non-paging no-leader, split-brain, and no-standby coordination alerts with executable Prometheus fixtures. Rejected flapping because retained last-value gauge series cannot truthfully represent completed handovers. Real Grafana push and explicit-context verification succeeded in sync.