Reader CAP scopes - propagation delay, no re-mint needed

Description

BLOCKER for every phase that reads Loki, Tempo or Pyroscope. Operator action on a live customer deployment, not a code change, and not for an agent to perform.

Verified 2026-08-24: the reader access policy reads back carrying logs:read, traces:read and profiles:read, and the existing token still answers 401 {"status":"error","error":"authentication error: invalid scope requested"} on all three. The same endpoints and basic-auth usernames answer 200 under a broader policy, which is the control case proving the endpoint and username are right.

The scope set is fixed when a token is minted. Widening the policy does not widen an existing token.

metrics:read already works and covers the whole Prometheus query API, not just cardinality - so Phase 1, the registry design and the Mimir half of the label inventory are all unblocked today.

Procedure is in RUNBOOK.md: POST /v1/tokens?region= with accessPolicyId in the body (the plausible /v1/accesspolicies//tokens returns 404), rewrite the WHOLE secret JSON preserving every other key, run one tier by hand, then revoke the old token. A running task does not pick up a new secret value; secrets resolve at task start.

Also note for the generic repo: the live policy carries 16 scopes while collector/config.py READER_SCOPES declares 9. alerts:read, rules:read, adaptive-metrics-config:read and adaptive-metrics-exemptions:read are present live and undeclared. Reconcile the declaration or record why it differs - do not silently widen the declared tuple to match.

Acceptance Criteria

• #1 A replacement token is minted on the existing policy and swapped into the secret with all other keys preserved
• #2 All four signals answer 200 for a sample stack under the new token
• #3 The old token is revoked
• #4 READER_SCOPES declaration versus live policy scopes is reconciled or the difference is recorded

Definition of Done

• #1 python3 -m pytest tests -q
• #2 tofu fmt -check -recursive terraform; tofu init -backend=false and tofu validate pass for terraform/ and terraform/examples/standalone/
• #3 customer-identifier and shipped-text gates from .github/workflows/ci.yml return clean

Implementation Notes

RESOLVED with no action required. The 401 was propagation lag, not a token or policy problem.

Sequence, measured on a live deployment: the org-realm reader access policy was updated at 14:53:41Z to add logs:read, traces:read and profiles:read. Data-plane calls with the ALREADY-MINTED reader token returned 401 invalid scope on all three signals inside the first half hour. The same token, the same stack and the same per-signal basic-auth usernames returned 200 on all three 46 minutes after the policy update. Exactly one token exists on that policy and it is the correct one.

An existing token DOES pick up a scope added after it was minted. The earlier conclusion that the scope set is fixed at mint time was WRONG. No re-mint, no secret rewrite, no revocation, no scope widening.

Cross-region verified with the same token: eight distinct Mimir regions, four signals each, all 200 under a single region-scoped org policy. Tier 2 is fully unblocked for the whole estate, so GCI-0008.04 carries no credential dependency.

The trap worth keeping, now recorded in docs/traps.md under Signal label inventory: the access-policy API reflects a scope change immediately, so a successful policy read-back proves only that the edit landed - never that the signal databases will honour it yet. During the lag the 401 is indistinguishable from a permanently wrong scope. Wait and retry; never re-mint and never widen a scope further to chase it.

Still open, carried into GCI-0008.04: collector/config.py READER_SCOPES declares nine scopes and the observed live policy carried sixteen. alerts:read, rules:read, adaptive-metrics-config:read and adaptive-metrics-exemptions:read were present live and undeclared. Reconcile the declaration or record why it differs - do not silently widen the declared tuple to match one deployment.

RECONCILED. collector/config.py READER_SCOPES now declares all sixteen scopes and matches the live policy exactly, with no drift in either direction. Added: logs:read, traces:read, profiles:read, rules:read, alerts:read, adaptive-metrics-config:read, adaptive-metrics-exemptions:read.

Each new scope carries its verified route in the declaration and in CAPABILITIES.md, probed live rather than inferred:

• rules:read -> {prom}/api/prom/api/v1/rules and /alerts (200); Loki /prometheus/api/v1/rules (200). Mimir config/v1/rules 404s on Grafana Cloud.
• alerts:read -> {amInstanceUrl}/alertmanager/api/v2/{status,alerts,silences} and /api/v1/alerts (200), basic-auth user amInstanceId.
• adaptive-metrics-config:read -> /aggregations/recommendations/config (200). /aggregations/config 404s.
• adaptive-metrics-exemptions:read -> NO route found. Eight candidates under /aggregations all 404. Declared to match the deployed policy; reserved, not a capability, until a route is verified.

Two of these reach beyond inventory and both are now flagged in the declaration, in CAPABILITIES.md and in docs/traps.md:

• /alertmanager/api/v2/status returns the stack RAW Alertmanager configuration in config.original including http_config. Where contact points live in Alertmanager rather than Grafana that YAML can carry webhook URLs and tokens. Nothing derived from it may be stored, logged or emitted beyond bounded counts - same handling as accessToken on a public dashboard.
• /api/prom/api/v1/alerts returns firing instances with full customer label sets. Unbounded and identity-bearing; count only.