Description
Prove the Coverage dashboard against a configured non-production deployment before any production rollout. Compare rendered figures with their source data, preserve the read-only and frozen-scope boundaries, and record unsupported product surfaces as environment limitations rather than manufacturing values.
Acceptance Criteria
- #1 A full Coverage dashboard snapshot is rendered and visually inspected for layout, empty states, table legibility and datasource errors
- #2 Sample service rows reconcile to explicitly-windowed Mimir, Loki, Tempo and Pyroscope label results, including successful empty signals and absent failed-stack rows
- #3 The Mimir SLO matcher and retained service_name association are checked against live generated SLO series
- #4 Sample seven-component observability-completeness scores are independently recomputed with both default and deployment-configured weights
- #5 OnCall acknowledgement, p50, tail and ownership figures reconcile to the timing-reporting population where the environment supplies them
- #6 Grafana-published billing components and unit economics are reconciled in a billing-capable environment; an environment without those series is recorded as unsupported and never rendered as zero
- #7 The feasibility of exporting the existing usage datasource results to S3 is investigated without widening any reader or access-policy scope
Definition of Done
- #1 python3 -m pytest tests -q
- #2 tofu fmt -check -recursive terraform; tofu init -backend=false and tofu validate pass for terraform/ and terraform/examples/standalone/
- #3 customer-identifier and shipped-text gates from .github/workflows/ci.yml return clean
Implementation Notes
Development validation started against the authorised non-production deployment after v0.2.0. The first serial T2 run safely refused publication because one stack Pyroscope LabelValues call returned HTTP 200 with an empty JSON object. Live comparison showed the other stacks returned populated names arrays; auth, endpoint and explicit 24-hour window were correct. Connect/Protobuf omits an empty repeated field, so this is the measured-empty wire form and exposes an absent-versus-zero parser defect to fix before validation can continue.
Added a contract test reproducing Pyroscope HTTP 200 with an empty Connect object; it failed before the parser change and passes after treating only an entirely empty object as an empty names list. Non-empty malformed objects remain invalid. CodeRabbit reviewed the staged change with zero findings. Full product gate: 1,389 tests passed, 2 skipped, 6,614 subtests; module and standalone OpenTofu init/validate passed with isolated data dirs; generated BUDGET.md is unchanged.
Completed authorised development validation. Full and panel snapshots rendered without datasource errors; the wide service completeness register remained legible at 2560px. Retained Mimir, Loki, Tempo and Pyroscope service rows reconciled after documented case normalisation, including the successful empty Pyroscope response and absent failed-stack behavior. Live generated SLO series retained valid service associations. Seven-component scores recomputed exactly for equal weights and for an independently exercised non-default weighting. OnCall population, acknowledgement, p50, tail and ownership outputs matched direct usage-datasource queries. Billing series were absent in the staff environment and rendered as no data, never zero; a billing-capable deployment subsequently returned finite currency values from saved queries whose spend and billable-usage terms come directly from grafanacloud-usage. Exporting that datasource to S3 is not feasible within the frozen reader scope because it would require Grafana datasource-query access not granted to the collector; dashboard-direct remains the accepted path without widening or minting credentials.
Final Summary
Validated Pillar K end to end against the authorised development deployment and then confirmed the billing-capable query path during promotion. Signal inventories, SLO matching, weighted completeness scoring, OnCall outcome measures, empty/absent semantics, layout and datasource provenance all matched their direct sources. The full public gate passed: 1,389 tests with 2 skips and 6,614 subtests, isolated module and standalone OpenTofu validation, customer-identifier and shipped-text checks, and unchanged generated budget.