Unscore components that cannot be evaluated, before improving any matcher

Description

The coverage score divides by 7 unconditionally. Measured across a 270-stack estate, three of those seven components are structurally unsatisfiable on almost every stack, so the score reports absent product adoption as failed coverage. THIS is the dominant defect, not the dashboard matcher.

Measured effect, and why matcher work must come second

A perfect matcher is worth less than a twentieth of what honest unscoring is worth. Do the unscoring first.

The three structural zeros

• profiles: 269 of 270 stacks returned zero service names from a SUCCESSFUL Pyroscope read. The product is not in use; that is not a coverage gap.
• slo: 267 of 270 stacks own zero SLOs. Only four stacks in the estate own any.
• alert: 150 of 270 stacks have an available alert inventory reporting zero rules.

Scoring these as no is the same defect as the dashboard tag, three times over, and it is exactly what the sibling maturity pillar already refuses to do with its explicit unscored reasons.

Unscoring rules

Metrics, logs and traces are the observation itself. A service present in one signal and absent from another is a real finding and must keep scoring no.

dashboard is NOT unscorable merely for a stack having no dashboards: zero of 270 stacks have no tenant-authored dashboards. Where a stack has dashboards and the service has none, no is honest.

Row-level unscoring: one class only

A register row whose identity is machine-generated gets unscored_reason = ephemeral_identity, is excluded from aggregates, and STAYS VISIBLE in the view. Detected from the name alone, no config, no maintained list:

\.(scope|service|slice|socket|mount|timer|target|device)$
^session-\d+
^user-\d+
\d{8,}
[0-9a-f]{8}-[0-9a-f]{4}-

Measured: 425 of 3,566 published rows (11.9%), and 12,459 of 18,376 in the full canonical set (67.8%). The \d{8,} rule is a heuristic and will catch a legitimately numbered service, so publish the count as its own metric and keep the rows visible with the reason attached rather than dropping them.

Denominator and disclosure

• Denominator = count of APPLICABLE components. Never divide by 7 when a component was withheld.
• Mirror MIN_WEIGHT_COVERED: withhold the row percentage entirely below four applicable components.
• Emit coverage_unscored{component, reason} - both bounded enums, budget-safe. Declare it in CATALOGUE or tests/test_budget.py fails.
• A mean-completeness stat panel MUST sit beside a mean-denominator panel. A score rising because the denominator shrank and a score rising because coverage improved are indistinguishable otherwise, and this pillar would publish the second reading of the first event.
• The estate summary needs a plain sentence naming the top unscored reason and its count. “N stacks own no SLOs, so slo is unscored for M services” is the finding the org needs; it is currently rendered as a column of “no”.
• Bump Score version. The number changes meaning, and a trend crossing a silent definition change is worse than a gap.

Acceptance Criteria

• #1 profiles, slo and alert unscore on measured product absence with a bounded reason enum
• #2 metrics, logs and traces never unscore
• #3 denominator is the applicable-component count, and the row percentage is withheld below four
• #4 machine-generated identities are row-unscored, excluded from aggregates, still visible in the view
• #5 coverage_unscored{component,reason} declared in CATALOGUE and rendered
• #6 mean completeness is never rendered without mean denominator beside it
• #7 Score version bumped

Definition of Done

• #1 python3 -m pytest tests -q
• #2 tofu fmt -check -recursive terraform; tofu init -backend=false and tofu validate pass for terraform/ and terraform/examples/standalone/
• #3 customer-identifier and shipped-text gates from .github/workflows/ci.yml return clean

Implementation Plan

1. Add decision tests for per-component applicability, the four-component score floor, and ephemeral row visibility/exclusion.
2. Make the score calculator version 2 and compute weighted scores only across applicable components while retaining numerator/maximum evidence.
3. Derive profile, SLO, alert, dashboard, and row-level unscored reasons from the existing left-joined inputs; emit bounded unscored and mean score/denominator metrics and expose reasons in the service/summary views.
4. Rebuild the Coverage dashboard around a paired mean-completeness/mean-denominator disclosure and render the unscored reason counts.
5. Regenerate BUDGET.md, run targeted tests, the full item gate, CodeRabbit, review the diff, then finalize, commit to main, and push.

Implementation Notes

VERIFIED against independent sources after the product owner challenged the profiles and SLO figures. Both hold, and both mean the products are barely used - which is the point of unscoring them rather than scoring them as failures.

Profiles, three independent measurements agreeing:

• our collector: 1 service carries a profiles identity across the whole estate
• billing datasource, non-zero profiles usage over 24h: 1 stack
• billing datasource, Pyroscope instance PROVISIONED: 274 stacks

So 269 of 270 measured stacks have zero profiling data while every stack has the instance provisioned. That gap is itself a finding of the same shape the estate pillar already publishes for other provisioned-but-unused capabilities, and it belongs on the surface alongside the unscored reason.

SLOs, two independent measurements agreeing:

• the estate metric-name sweep: 4 of 270 stacks carry any grafana_slo_* metric
• the SLO API census: 4 stacks own any SLO

So 266 of 270 stacks own no SLO. grafanacloud_slo_* does not exist as a metric family; do not look for it.

For scale, over the same 24h window: 269 stacks ingest logs and 232 ingest traces. Profiles at 1 and SLOs at 4 are the outliers by three orders of magnitude, not a measurement artefact.

Wording rule for whoever implements this: state these as “N stacks have NO profiles” or “M stacks own zero SLOs”. A bare “269/270” next to the word profiles reads as the opposite of what it means and was misread once already.

Implemented score version 2 with explicit applicability, a four-component percentage floor, bounded component/reason counts, paired mean score and denominator metrics, and visible-but-aggregate-excluded ephemeral identities. Targeted coverage, dashboard, budget, guard, compose, and hydration checks pass; the dependency derivation remained unchanged.

Final verification: 1398 passed, 2 skipped, 6614 subtests; OpenTofu formatting and both module validations passed; customer-identifier working tree and history scans passed; shipped-text check passed; CodeRabbit re-review completed with zero findings.

Final Summary

Made coverage score version 2 defensible by scoring only applicable components, withholding thin percentages, excluding visible ephemeral identities from aggregates, and publishing bounded unscored reasons with a population-matched mean denominator. Verified through the full offline suite, dashboard/view coverage gates, generated budget, Terraform validation, privacy/text gates, and a zero-finding CodeRabbit re-review.