Description
Read-only loop28 observation found both exporter replicas in CrashLoopBackOff. Previous startup logs reject ingress_wal.max_permanent_rejections as an unknown configuration key, while current source recognizes that key. This indicates a deployed image/configuration compatibility mismatch, not a dashboard query failure. Current telemetry and process-status prerequisite attestation are unavailable until availability is restored. No cluster or tailnet mutation was authorized in loop28; this follow-up is not admitted into its frozen envelope. Keep deployment identities and raw captures in ignored evidence only.
Acceptance Criteria
- #1 Identify the exact deployed image version and configuration revision and prove the selected image recognizes every configured key.
- #2 With explicit deployment authority, restore both replicas without weakening unknown-key validation or discarding retained ingress state.
- #3 Verify readiness, fresh exporter heartbeat/scrape telemetry, and actual configured collectors on the restored process; retain exact image digest and observation identities in ignored evidence.
Definition of Done
- #1 just check passes (the full gate; it is what CI enforces)
- #2 just gen leaves no diff (only if a generated artifact’s inputs changed)
- #3 just –fmt –check passes and every new recipe has a # doc comment and a [group(…)]
Implementation Notes
2026-10-11 main thread: cause was chart/image skew in the lab GitOps deploy. The Argo source rendered this chart from main while the image tag stayed on an older 5.2.0 rc, and the deploy values’ Renovate had stopped advancing it. Main’s chart writes ingress_wal.max_permanent_rejections, which that binary rejects. Proof: the older image’s -validate reproduces the unknown-key error on the rendered lab config; 5.2.0-rc.35 exits 0 (placeholder env for every credentials-Secret key name). Fix in the deploy repo: chart targetRevision and image.tag pinned to the same release and moved by one Renovate dependency. Validation stays strict and ingress state was preserved (WAL drain pending_entries=0). Readback: Argo Synced/Healthy, both replicas Ready with 0 restarts, leader lease acquired, streaming/admin Services select the leader. AC #3 is partial: exporter-side families are fresh (238 vs 280 a day earlier), but the streaming-ingest families (stream records, ingest, flows, config audit, logstream) had not resumed within the first five minutes. Retained evidence is local, in ignored codex/loop29/restore-evidence.md.
loop30: zero implementation attempts; read-only AC #3 verified from contemporaneous retained Kubernetes/Grafana responses. Both replicas Ready; fresh heartbeat and all 16 scheduled collector results; audit and flow streaming resumed. Historical OOM restart remains a separate reliability risk. Source gate, regeneration and CodeRabbit not applicable: no code or deployment change in this loop. Private evidence retained in ignored /Users/rob/repos/tailscale2otel/codex/loop30/tso0190/.
Final Summary
loop30 completed availability verification only: Ready replicas, fresh collector telemetry and advancing configured streams observed. Exact release, digest and config identities retained locally; no cluster/tailnet mutations. No claim of restart-free or per-target scrape reliability.