Labelling best-practice research and rulebook

Researched 2026-10-06 for the labelling best-practice pillar. Owner decisions D-LBL1 to D-LBL11 and the task plan are in Part 1. Quotes marked as obtained via page summaries must be re-verified (record URL and date) before any rule ships with provenance published.

Part 1: Assessment, decisions and plan

• Context: /private/tmp/claude-503/-Users-rob-repos-grafana-cloud-org-insights/bbe8de11-8e49-4ace-92ce-67dc9d2c5e84/scratchpad/labelling-synthesis.md
• Lane briefs: research-traces.md and research-psguides.md, same directory. Task A copies them into a backlog doc because the scratchpad is temporary.
• v1 failed adversarial review. Every finding is addressed below.
• No customer name appears anywhere in this repository: check-identifiers --history gates it, and doc-0002:14 places customer identifiers in the deployment repo.

Owner decisions (2026-10-06)

• D-LBL1 Retention.

  • label_inventory may persist the following in S3 views and the private label_inventory hydration input only:
    • label and attribute NAMES;
    • distinct-value counts (exact or at_least);
    • closed non-PII value-shape class counts (uuid, hex_id, epoch, url_with_id, long_value);
    • series and stream counts.
  • These items never go to Loki, finding events, metric labels, stdout, --out or errors.
  • A name over 512 bytes, or one matching a pii key or value class, persists only as a class and a count.
  • Raw values are transient and never written. The GCI-0018 raw-value exception is not widened.

• D-LBL2 Deterministic only. No LLM in the scanner or operator tooling. Judgement rules are catalogued and excluded from applicable weight.

• D-LBL3 Layered, tunable thresholds.

  • Published limits are hard rules.
  • The Professional Services bands are the defaults: metrics warn at 100, high at 1,000, critical at 10,000; Loki dynamic labels warn at 100+.
  • Everything else is policy.
  • Each threshold carries a provenance tag (published, ps or policy) and a source URL, and is overridable through the tunables.

• D-LBL4 Presentation.

  • Per stack and signal: findings by severity, rules evaluated vs passed, a 0-100 score, and the coverage figure.
  • The score is computed only at coverage of 0.8 or more; otherwise it is absent.

• D-LBL5 Maturity. The labelling score replaces cardinality_discipline. This waits for the dev proof.

• D-LBL6 Routes. Staff witnesses are approved for:

  • Mimir cardinality/label_values and label_names (limit/selector);
  • Loki /series, index/stats and index/volume;
  • Loki applied limits/OTLP config;
  • Tempo intrinsic name values and Tempo overrides.

  Each route is implemented only after its witness. A parked route’s rules are excluded from applicable weight.

• D-LBL7 Customer grant. The customer deployment may enable label-inventory once it ships in a release and the task K dev proof is recorded. No further owner decision is needed. Routes without a passing witness are not covered. Every reader-role, route, privacy and publication rule is preserved.

• D-LBL8 Fairness.

  • Cardinality bands apply only to metrics and streams above a size floor (tunable, same pattern as CARDINALITY_MIN_SERIES).
  • A static-infrastructure allowlist (host, cluster, namespace, node and similar) has its own higher band.

• D-LBL9 PII shapes. Email, ip, phone, jwt and card shapes stay on label_risk’s retention-governed path. The labelling register refers to risk_label_hygiene and does not duplicate them.

• D-LBL10 Witness stacks. Witnesses may query all five staff stacks and choose or combine evidence. Each recorded witness names the stack slug.

• D-LBL11 Pre-approved read scopes. When a witness shows the org reader lacks a READ scope for an approved route, the witness task may add that read-only scope to this project’s own access policy.

  • The addition is recorded with its object ID, and CAPABILITIES.md is updated.
  • Write or admin scopes, other policies and the per-stack reader role are never covered.
  • A new scope can 401 for about 46 minutes; wait it out, never re-mint.

Hard-rule amendments (land in task A, before B, C and D start)

• (a) Pyroscope POST exception (AGENTS.md, GCI-0018 section), reworded so that:

  • label_risk.py and label_inventory.py may POST only the two exact Pyroscope read RPCs;
  • they do so only through label_risk.profile_read, which keeps the exact path set, the inventory HTTPS host, no redirects and the bounded body and response;
  • no other module, path or helper is granted, and collector/httpclient.py stays GET-only.

• (b) Identity-bearing detail bullet. Append the D-LBL1 text.

• (c) doc-0002:45. “calls label endpoints only” becomes:

  calls Loki label-name and label-value, /series, index/stats, index/volume and limits/config endpoints only, never query, query_range, tail or any route returning log lines; each requires a recorded staff witness.

• (d) New AGENTS.md section “Labelling inventory (D-LBL1..11)”. It lists each route by exact path with its witness-first rule, the D-LBL7 grant and the D-LBL11 scope rule.

• Not amended:

  • MAX_PER_STACK_FANOUT and the one-extra-label rule;
  • gap-is-absent;
  • the limited-run guard;
  • own-input hydration;
  • derived VIEW_INPUTS.

Frozen contracts (task B)

• Result enum: pass | fail | not_evaluated | not_applicable. The reason is a closed enum: missing_input, route_parked, truncated, deadline, no_signal_data, precondition_false.
• Unit: one result per (rule, stack, signal). The evidence holds closed enums and numbers only: count of offending labels, worst band, at_least flag. Names go only in register rows.
• Truncation:
  • at_least N at or above a band is a fail at that band.
  • at_least N below the band is not_evaluated(truncated).
  • A values-read byte-cap overflow is fail(overflow) at no less than the warn band.
  • A truncated name list makes every absence rule (“no X present”) not_evaluated(truncated).
• Applicability:
  • not_applicable covers a signal with no data (a measured-empty 200) and a false precondition.
  • Cross-signal rules need data on both signals.
  • Applicable weight excludes judgement rules and parked or unapproved routes. It includes not_evaluated results caused by missing_input, truncated and deadline.
• Severity: findings are low | medium | high, with weights 1 / 3 / 5. Band mapping: the PS warn band is medium; the high and critical bands are high, with critical recorded as the worst band in evidence. That makes the metric’s severity enum 3.
• Score: 100 * passed_weight / evaluated_weight, published only when evaluated_weight / applicable_weight >= 0.8.
• Catalogue version: catalogue_version is stamped in every view and in an estate-level gauge. Scores are comparable only within one version. Each later route-extension (E) task bumps it.
• Rule-input registry: extensible by E tasks without reopening the seam.

Metrics (budget-shape compliant)

• Signals are the 4 data signals; cross-signal results are views only.
• That is 15 series per stack, about 4,065 at STACK=271, plus the provenance series. INPUT goes from 29 to 30, as re-derived by the test.
• A per-severity count is emitted only when that severity’s rules reached the coverage floor. Otherwise the series is absent.

Tasks (parent + subtasks; IDs assigned at creation; no customer name in any text)

P - Labelling best-practice pillar (parent). Links the decisions, the research doc and the order. AC: all subtasks Done, or Parked with a reason.

A - Record the labelling decisions and amendments. First; blocks everything else.

• AC1: amendments (a) to (d) are landed in AGENTS.md and doc-0002, word for word.

• AC2: D-LBL1..11 are recorded in doc-0002 standing decisions, with D-LBL7 and D-LBL11 as conditional grants.

• AC3: a backlog doc “Labelling best-practice research and rulebook” holds:

  • the merged rulebook;
  • the deprecated-semconv table pinned to v1.44.0;
  • the Professional Services label-analyser bands;
  • every source URL.

  Quotes obtained through a summary are marked for re-verification, with a verification date column.

• AC4: GCI-0101 gets an appended note: D-LBL6 covers its route for labelling purposes, and its top-producers dimension stays open.

• AC5: just check passes, check-identifiers included.

B - Rule catalogue and evaluator seam. After A.

• collector/label_rules.json (versioned) plus a loader with schema validation.
• A pure evaluate() implementing the frozen contracts above.
• AC: tests for each truncation, overflow, not_applicable and coverage-floor case, plus a test that a rule with missing input can never pass. The schema test also rejects a non-enum evidence string.

C1-C4 - Staff route witnesses. After A; run in parallel, each read-only against all five staff stacks (D-LBL10).

• C1 Mimir cardinality label_names?limit=500 with a selector, and label_values with label_names[]. Record the head-only window and the limit behaviour.
• C2 Loki /series over a bounded window, plus index/stats and index/volume. Record shapes and reachability. Check that no log lines come back and that /series stays within its bounds.
• C3 Loki applied limits: compare JSON config/limits/applied with the existing /config/tenant/v1/limits, and choose the one that needs no new YAML features. Fields: otlp_config, discover_service_name, discover_log_levels, allow_structured_metadata, volume_enabled, max_label_names_per_series.
• C4 Tempo intrinsic name values (truncation at the 1 MB cap), and whether the Tempo overrides are readable.
• AC each:
  • witness recorded in docs/traps.md, naming the stack slug or slugs;
  • route added to the AGENTS.md labelling section;
  • GET only;
  • a missing read scope is handled under D-LBL11 (record and update CAPABILITIES) or parked.

D - label_inventory source. After B; does not wait for C.

• Runs in tier T2, iterating gcom.fetch_inventory. It is default-off: its enable flag and its config.py entry belong to this task.
• An isolated ReadOnlyClient with its own deadline slice, at a stated number of seconds and a stated position in run_t2. Running out of budget gives partial(deadline).
• Metrics: per-label distinct counts come from the proven cardinality/label_names route (one call per stack; head-only window recorded).
• Value reads per signal are capped like Bounds.keys. Priority is fixed: name-pattern hits first, then cardinality order.
• Loki and Tempo use the label-values routes. Pyroscope goes through label_risk.profile_read under amendment (a).
• Name minimisation follows D-LBL1. Non-PII shape classes only, per D-LBL9. Register rows per stack and signal are bounded.
• Coverage state per stack and signal: complete | partial | unavailable, with a reason enum.
• AC:
  • contract tests over recorded shapes;
  • zero calls when disabled;
  • a canary test: unique sentinel values through a fake transport are absent from the scan envelope, every view, Loki events, metrics, stdout, --out and error strings;
  • the limited-run guard holds;
  • the INPUT_OWNER entry is added;
  • each input records its window (head vs 24h).

E1-E4 - Route extensions. Each after its C witness and after D, run one at a time because they share label_inventory.py, label_rules.json, budget.py and hydrate.py.

• E1 Mimir series per value and per metric.
• E2 Loki labels per stream and the values-to-streams ratio.
• E3 Loki OTLP index-label set and Drilldown prerequisites.
• E4 Tempo span-name cardinality and override checks.
• Each bumps catalogue_version.

F - Labelling pillar. After B and D.

• Views:
  • labelling_findings, deliberately NOT a FindingSpec, so no label names reach Loki;
  • labelling_stack_summary;
  • labelling_cross_signal, which reuses signal_inventory service and cluster sets and publishes counts and set sizes only;
  • labelling_label_register, bounded or top-N.
• The metrics table above.
• AC:
  • a test that no label_inventory name appears in Loki events;
  • VIEW_INPUTS re-derived;
  • empty-view schemas;
  • gaps are absent series;
  • carry-forward drops departed stacks;
  • BUDGET.md regenerated;
  • each view carries catalogue_version.

G - labelling dashboard. After F.

• Registered in DASHBOARDS, BUILDERS and PILLAR_OF.
• Panels:
  • estate overview: worst severity, score with coverage, trend;
  • per-stack scorecard;
  • cross-signal consistency;
  • per-signal rule tables;
  • label register;
  • not-evaluated and coverage.
• The banner states the data is deterministic and comes from bounded samples.
• AC: coverage gates pass.

I - Config, Terraform and docs. After D.

• Threshold tunables, size floor, static allowlist, coverage floor, budget seconds.
• Updates to RUNBOOK, CAPABILITIES, SPEC and traps.
• AC: just tf-validate; the docs state what is and is not inspected.

K - Dev live proof. After D, F, G and I, plus whichever E tasks landed.

• Full staff-estate run.
• p95 per-stack runtime and total wall time, measured against the 3600 s T2 tier.
• Peak memory, against the 232.7 MiB recorded under GCI-0018.01.
• Raw-value absence: known staff label values are read separately and grepped across S3 objects and Loki.
• Score stability across two consecutive days.
• AC: evidence recorded; release cut.

H - Maturity replaces cardinality_discipline. After K.

• Bump RUBRIC_VERSION.
• Aggregation: the per-stack dimension is the weight-averaged score over signals with a published score. It is withheld if none has one.
• Re-derive the maturity view inputs.
• CHANGELOG note.
• AC: tests updated, with the reason named.

L - Enable on the customer deployment under D-LBL7. After K and the release.

• Done in the consumer deployment repo (doc-0003); the customer name lives only there.
• AC: first run verified; parked routes listed as not covered.

M - Reconcile overlapping findings. Owner-decision label; after K. Decide whether labelling supersedes or keeps:

• risk_label_cardinality (GCI-0018.01);
• the cost_cardinality_outliers worst-label finding;
• maturity cardinality_discipline as already replaced by H.

Both outputs stay unchanged until then.

J - Share one names fetch with label_risk. Parked until after K. label_risk’s privacy behaviour changes only under its own decision. K’s budget accounts for the double fetch in the meantime.

Order

A first. Then B and C1-C4 in parallel. D follows B; I follows D. E1-E4 follow their witness and D, in series. F follows B and D; G follows F. K follows D, F, G and I. H, L and M follow K. J stays parked.

Part 2: Synthesis

Lane briefs on disk: research-traces.md, research-psguides.md (same directory). Metrics and logs briefs are summarised here. Caveat: the metrics lane quoted via WebFetch summaries, so quotes are near-verbatim only. Re-verify any quote before it reaches customer-facing panel text.

1. Current coverage (verified against code)

• collector/sources/label_risk.py (T2 daily, GCI-0018) already reads label NAMES on all four signals (Mimir/Loki/Tempo GET, Pyroscope via the two named POSTs).
  • It samples up to 256 values for 64 keys per signal (Bounds, :36). Keys are selected PII-first, then lexically (:230), so most keys on a large stack are never value-sampled.
  • Names are transient. Only classified PII matches persist (S3 risk_label_hygiene), plus a keys_returned count.
• dataplane.cardinality (T3) keeps the Mimir top-20 label names with value counts. label_cardinality.py does an exact-name denylist over those 20 only.
• signal_inventory (T2) keeps service, cluster and metric-name lists. Adaptive Metrics recommendations are collected as totals plus a top-10 sample.
• loki_config fetches /config/tenant/v1/limits but keeps only retention_stream.
• Nothing scores labelling practice. The nearest thing is the maturity cardinality_discipline dimension (weight 0.10, values/series ratio).

Verdict: the user’s instinct is right. We touch the right APIs but throw the data away. Most of the platform-operator value comes from persisting names and counts we already fetch, plus a few new bounded GET routes.

2. What best practice gives us (merged rulebook, deterministic subset)

Grafana publishes few numeric thresholds. Every threshold carries a provenance tag: published (Grafana/OTel limit or doc), ps (Professional Services guide) or policy (ours, tunable).

Cross-signal (highest value for an org of many stacks)

• X1 Service identity present: Loki service_name; Tempo resource.service.name; Pyroscope service_name; metrics service_name or job. [published]
• X2 unknown_service* (traces/logs) or unspecified (profiles) share of service values. [published fallback; share threshold is policy]
• X3 Environment attribute: deployment.environment.name preferred. Legacy-only is a migration finding. Mixing the two splits Knowledge Graph environments. [published]
• X4 Same service set across signals. The set difference is the correlation gap. [published principle]
• X5 Same k8s identity labels across signals (cluster, namespace). [published]
• X6 Label-name synonym drift on one stack (env/environment/deployment_environment, namespace/k8s_namespace_name, case variants). [alias table is policy]
• X7 Cost-attribution label coverage, only if the configured attribution labels are readable. [published 2 labels / 1,000 combinations]

Metrics

• M1 Name syntax and reserved __ prefix. Dotted/UTF-8 names reported separately as an OTLP signal. [published]
• M2 Unbounded-identifier names (user/session/request/trace id, uuid, ip, email, timestamp, url/path), confirmed by value count. [published list + PS-25]
• M3 Value-shape classes on sampled values (uuid, ip, epoch, hex id, email, url-with-id). Only shape counts persist. [policy regexes]
• M4 Per-label distinct values: warn at 100+, high at 1,000+, critical at 10,000+. [ps bands; Prometheus under 10 / over 100]
• M5 Metrics with 1,000+ series; the top-N share of the series total. [ps]
• M6 Labels per series near 30/40. [published]
• M7 Value length near the 2,048 truncation point. [published]
• M8 Churny labels promoted by OTLP defaults (k8s_pod_name, service_instance_id) on high-series metrics. [inference]
• M9 target_info bloat and resource_to_telemetry_conversion symptoms (process_*, telemetry_sdk_*, os_* on datapoints). [ps + OTel]
• M10 Adaptive Metrics addressable waste, already collected; reuse it. [published]
• M11 Naming hygiene: base units, _total, no values embedded in names. Low severity. [published Prometheus]

Logs

• L1 Labels per stream: 15 is a hard limit; warn above 10. [published]
• L2 Index labels with unbounded names (pod, trace_id, request_id, user_id, ip, timestamp). [published]
• L3 Distinct values per label: dynamic labels “tens”; warn at 100+, high at 1,000+; static-infrastructure allowlist (host, cluster, namespace). [published qualitative + policy]
• L4 Values-to-streams ratio, from Grafana’s own logcli --analyze-labels method. [published method]
• L5 Active streams against the volume-scaled guidance and the stack limit. [published]
• L6 level as an index label: a soft finding, because Grafana’s docs disagree with themselves. [published, conflicting]
• L7 OTLP defaults still indexing k8s_pod_name / service_instance_id. Grafana says these are “no longer recommended”. [published]
• L8 Legacy LokiExporter format (exporter="OTLP"). [published, deprecated July 2024]
• L9 Drilldown prerequisites in applied limits (discover_service_name non-empty, discover_log_levels, allow_structured_metadata, volume_enabled). [published]
• L10 Structured metadata keys and size (128 entries / 64KB). Needs detected_fields. [published]

Traces

• T1 Required resource attributes: service.name, service.namespace, deployment.environment(.name), service.instance.id, service.version. If any k8s attribute is set, the k8s cluster/namespace/pod triple. [published, App O11y quality report]
• T2 / in service.name or service.namespace values. [published]
• T3 Span-name cardinality and ID-shaped span names (intrinsic name values; a truncated list is itself evidence). [published principle, no threshold]
• T4 http.route values are templates. [published MUST]
• T5 Deprecated semconv names present (lookup table pinned to semconv v1.44.0). Low/medium severity, because Grafana’s own defaults lag. [published]
• T6 Sensitive attribute names, and SQL with literals in db.query.text. Overlaps GCI-0018; route to the existing privacy source. [published]
• T7 Attribute key syntax (lowercase dotted snake_case; no custom keys under otel.* or semconv namespaces). [published]
• T8 Large-payload attribute names (bodies, headers, full SQL). [published]
• T9 Span-metrics/service-graph dimension risk. Needs the Tempo overrides, which may not be readable on Cloud. [published]

Profiles

• P1 service_name present and not unspecified. [published]
• P2 No dotted names; name/value length limits. [published]
• P3 Distinct values per label (ID-like). No threshold published. [policy]

Judgement-only rules (no deterministic answer)

• “Used in 9 of 10 queries” needs query logs or dashboard selectors.
• Span-name genericity beyond regex.
• Unusual synonym detection.
• The downstream-dependency check before advising a demote.

These are the only candidates for an LLM.

3. Proposed architecture (follows the golden rule)

• New T2 daily source: label_inventory.
  • Iterates the live inventory: per stack, per signal, it reads label names (Tempo scope-qualified, intrinsic name separately) and a bounded distinct-value count per label.
  • Values are fetched to count and shape-classify, then discarded. Persisted per label: name, scope, value count (or at_least N when truncated), shape-class counts, and the series/stream count where the API gives it.
  • Stays separate from label_risk so the privacy-reviewed source is untouched. Deduplicating the fetch is a later task.
• Rule catalogue: collector/label_rules.json, versioned like label_patterns.json. Each rule carries id, signal, severity, threshold, provenance, source URL and input requirement. A rule whose input is missing yields not_evaluated, never pass.
• Pillar: collector/pillars/labelling.py composes views:
  • labelling_findings: stack, signal, rule, severity, label, evidence.
  • labelling_stack_summary: per stack and signal, rules evaluated, passed and failed by severity, plus coverage state.
  • labelling_cross_signal: the service-set and env/k8s consistency matrix.
  • labelling_label_register: per stack, signal, label name, count, shape and series.
• Metrics (time series justified by the trend):
  • gcinsight_labelling_findings{stack,signal,severity}: 4 signals x 3 severities per stack.
  • gcinsight_labelling_rules_evaluated{stack,signal}.
  • Rule ids stay in views, not labels.
  • Both declared in budget.py; VIEW_INPUTS re-derived.
• Dashboard: a new labelling dashboard (DASHBOARDS + BUILDERS + PILLAR_OF):
  • estate overview: stacks by worst severity, trend;
  • per-stack scorecard;
  • cross-signal consistency;
  • per-signal rule tables;
  • label register drill-down.
  • The maturity dimension cardinality_discipline is fed by, or replaced with, a labelling dimension.
• Hydration: the input owner is t2; T1/T3 hydrate. Carry-forward uses the live stack set.
• Customer: the new reader is default-off. Customer enablement is pre-authorised by D-LBL7 once the dev proof (task K) is recorded.

4. New routes needing staff-stack witnesses before build

• Mimir GET /api/v1/cardinality/label_values (per-value series counts) and label_names with a higher limit (max 500) or a selector.
• Loki GET /loki/api/v1/series (bounded window) for labels-per-stream and the values-to-streams ratio, or index/stats / index/volume. That folds in GCI-0101 (top log producers per stack read, an open owner decision).
• Loki GET /loki/api/v1/config/limits/applied (Cloud self-serve: OTLP and Drilldown limits). Reachability with the org CAP is unverified. Alternative: retain more keys from the existing /config/tenant/v1/limits.
• Loki GET detected_fields (structured-metadata keys). Unverified on Cloud.
• Tempo intrinsic name values (an existing route family, new tag). Tempo overrides (span-metrics dimensions, dedicated columns) are likely not readable with a read token. Probe first, and park if not.

5. Open risks

• Large-estate scale vs the T2 budget. label_risk already takes up to 900s. The new source needs its own budget slice and a coverage state per stack.
• The value-count truncation semantics differ per signal (Loki and Pyroscope ignore limit). Use at_least semantics like the IRM counts.
• Grafana guidance conflicts (level label; the 15 vs 30 OTLP promotion cap; deprecated names in Grafana’s own defaults). Severity stays low where Grafana contradicts itself.
• Label names are usually not personal data but can be (tenant names in keys, dynamic keys). Persistence is an owner decision.

Part 3: Metrics and logs research

Metrics lane (researched 2026-10-06; quotes via WebFetch summaries, re-verify before customer-facing use)

Published thresholds

• Labels per series: 30 recommended, 40 max. Per-stack limit “varies by configuration”.
• Label name: 1,024 characters max.
• Label value: 2,048 characters, truncated with an appended hash.
• Adaptive Metrics does not recommend aggregating a metric with fewer than 100 series.
• Unused metric: no query in 30 days and no dashboard, recording-rule or alert use.
• Cost attribution: at most 2 labels and 1,000 combinations.
• Learning Hub guidance is “tens of values, not thousands”.
• Prometheus guidance is “below 10 typical, over 100 investigate”.

Rules (M-xx)

OTLP rules (O-xx)

• O-01 Grafana Cloud promotes 19 resource attributes to labels by default:

  • service.instance.id, service.name, service.namespace, service.version;
  • cloud.availability_zone, cloud.region, container.name;
  • deployment.environment, deployment.environment.name;
  • k8s.cluster.name, k8s.container.name, k8s.cronjob.name, k8s.daemonset.name, k8s.deployment.name, k8s.job.name, k8s.namespace.name, k8s.pod.name, k8s.replicaset.name, k8s.statefulset.name.

  Changing the list is “contact Support”; a MimirOTLPConfiguration reference is unverified. Source: grafana.com/docs/grafana-cloud/send-data/otlp/otlp-format-considerations

• O-02 Translation: job is service.namespace/service.name, and instance is service.instance.id. job=unknown_service* is a coverage finding (inference). Source: mimir/latest/configure/configure-otel-collector

• O-03 Other resource attributes go to target_info. Flag bloat or churny attributes on it (inference).

• O-04 resource_to_telemetry_conversion symptoms on datapoints: process_*, host_name, os_*, telemetry_sdk_* (OTel UX research, 2025).

• O-05 Suffix translation conflicts (_total_total; foo_seconds vs foo_seconds_total).

• O-06 UTF-8: Prometheus 3 allows it, while Grafana Cloud docs describe underscore escaping. This is unresolved.

• O-07 A ; in values signals a key collision (OTel spec concatenation rule).

• O-08 Always define deployment.environment.name and service.namespace. Adaptive Metrics rules aggregating instance need checking (blog 2025-05-20).

Mimir API notes

• cardinality/label_names and cardinality/label_values:
  • parameters selector, count_method=inmemory|active and limit (default 20, max 500);
  • head-only (about 2h).
• cardinality/active_series may be disabled; verify per stack.
• Grafana advises using the plain /labels and /label/<n>/values routes unless counts are needed.

Logs lane (researched 2026-10-06; Loki v3.7.x docs)

Published thresholds

• 15 label names per stream, hard in Cloud (“aim for 10-15 at a maximum”).
• Label name 1,024 bytes; value 2,048 bytes.
• Structured metadata: 128 entries and 64KB per line.
• Dynamic labels: “tens of values” (“single digits, or maybe 10’s”).
• Active streams: under 100,000 active and under 1M per 24h for tenants above 10 TB/day, scaled down proportionally for smaller tenants.
• Grafana Cloud stream limit starts at 5,000 per tenant, scales to 80,000, with a hard ceiling around 200,000.

Rules (L-xx)

OTLP default index labels (Loki latest, 17 entries)

• cloud.availability_zone, cloud.region, container.name, deployment.environment.name;
• k8s.cluster.name, k8s.container.name, k8s.cronjob.name, k8s.daemonset.name, k8s.deployment.name, k8s.job.name, k8s.namespace.name, k8s.pod.name, k8s.replicaset.name, k8s.statefulset.name;
• service.instance.id, service.name, service.namespace.

limits_config also lists the legacy deployment.environment. Grafana Cloud additionally promotes the Faro app_id, app_key and kind. Everything else goes to structured metadata. Source: loki/latest/send-data/otel

service_name fallback order

service, app, application, app_name, name, app_kubernetes_io_name, container, container_name, k8s_container_name, component, workload, job, k8s_job_name, then unknown_service.

Conflicts

• The OTLP promotion cap is 15 on the format page and 30 on the self-serve page.
• The default list is 17 in one place and 18 in another.
• The level label: guidance says avoid it, yet Grafana’s examples keep it.
• The stream guidance and the Cloud ceiling differ.
• “15 index labels” is per stream, not tenant-wide.

Part 4: Traces, OTel attributes and profiles research

Method: raw .md of grafana.com docs (latest/current as of 2026-10-06) via curl plus WebFetch; OTel semantic-conventions v1.44.0 (released 2026-08-04) cloned and read from source (model/.yaml, docs/.md, CHANGELOG, GitHub release notes). Fetched content treated as data. “INFERENCE” = my deduction, not published text. Doc quotes are short verbatim excerpts.

1. TL;DR

1. Grafana publishes a short hard list for traces: resource service.name, service.namespace, deployment.environment.name (or legacy deployment.environment), k8s.cluster.name + k8s.namespace.name + k8s.pod.name (+ one k8s workload attr), a host id (k8s.node.name | host.id | grafana.host.id), no slashes in service name/namespace, low-cardinality span names. These come from the App Observability “Instrumentation quality” report and the Knowledge Graph prerequisites page.
2. Semconv is v1.44.0. deployment.environment was renamed to deployment.environment.name in v1.27.0 and the new name became STABLE in v1.41.0 (2026-04-28). HTTP is stable (since v1.23.0), DB is partly stable (db.system.name, db.namespace, db.query.text, db.collection.name, db.operation.name stable), RPC is release_candidate, messaging is still development. Many old names still circulate in Grafana’s own docs and Tempo defaults (http.method, peer.service, db.system), so deprecated-name checks need severity “low/medium”, not “high”.
3. Cardinality is the cost lever: span_name is the largest driver of Tempo span-metrics series; each extra dimension multiplies series by its distinct values; Grafana publishes no numeric “too many values” threshold. Scorable proxies: distinct counts of name and http.route values, ID-like regexes, tag-values truncation at the 1 MB default query limit.
4. Tempo limits to check against: attributes truncated at 2048 bytes (values) / 1024 bytes (names) in Cloud; dedicated columns per scope are 10 string (vParquet4) or 20 string + 5 int (vParquet5, default from Tempo 3.1); Cloud max trace size is documented inconsistently (3 MB vs 5 MB).
5. Pyroscope: service_name is “required and must always be present” (fallback value unspecified); label names must match [a-zA-Z_][a-zA-Z0-9_] (no dots); defaults are 30 label names per series, 2048 byte values; no published per-label value-count threshold. Cross-signal correlation requires identical service names (service.name = service_name).

2. Checkable rules

Severity: H high, M medium, L low, I info. Det = deterministic; Judge = needs judgement. “Tags” = /api/v2/search/tags by scope; “Values” = /api/v2/search/tag/<scope.tag>/values (sampled, subject to max_bytes_per_tag_values_query, default 1,000,000 bytes, so a long list may be truncated: treat truncation as “unknown”, and high-cardinality evidence).

Traces (T)

Cross-signal (X)

Profiles (P)

3. Deprecated -> current semantic-convention names (lookup table; semconv v1.44.0, extracted from model/*/deprecated yaml, versions from CHANGELOG/release notes)

Stability of replacements in v1.44.0: HTTP and server.*/url.*/error.type/network.peer.* stable; db.system.name, db.namespace, db.collection.name, db.operation.name, db.query.text, db.query.summary, db.response.status_code stable; deployment.environment.name, service.name|namespace|version|instance.id, k8s.cluster|namespace|pod|node|container|deployment|statefulset|daemonset|job|cronjob|replicaset.name stable (k8s promoted v1.42.0); rpc.* release_candidate; messaging.* and cloud.*, host.*, service.peer.name development.

Tempo-side observations tied to renames (from the Tempo/Grafana docs read): span-metrics example dimensions still list http.method, http.status_code; service-graph peer_attributes default peer.service, db.name, db.system, db.system.name; Tempo best-practices page examples use http.url, k8s.namespace, k8s.cluster, k8s.container_name (stale, undated).

4. Gaps (nothing published, or only partial)

• No Grafana-published numeric thresholds for: distinct values per attribute, distinct span names per service, distinct tag names per scope, label values per Pyroscope label, active series per dimension. Only qualitative (“high”, “low”, “handful”) and the tag-values 1 MB query cap. Any threshold chosen is INFERENCE.
• No Grafana/Tempo statement of which service.name values count as “unknown” for traces or App O11y (only the semconv SDK fallback and Loki’s unknown_service label default).
• Traces Drilldown: only “Tempo 2.6+ with TraceQL metrics configured”; no required attribute list.
• Adaptive Traces: no statement whether span metrics are computed before or after sampling; no rule about required attributes beyond the diversity fingerprint list.
• Cloud Traces: no published per-stack list of dedicated columns, no self-serve procedure found for Cloud (INFERENCE: via support/overrides), no published tag-count limits. “Maximum number of resource attributes / span attributes: Limits are on the size of the traces.”
• No Grafana doc stating the migration deadline for deployment.environment -> .name; App O11y accepts both.
• Pyroscope: no published cardinality threshold or tag-count guidance beyond limits in the reference config; Cloud’s profile limits are not in the usage-limits page (only metrics/logs/traces tables).
• PII/sensitive-attribute guidance from Grafana is thin (Tempo OOM page and reduce-trace-size page); semconv only flags enduser.id, url.full, db.query.text, exception.message as sensitive. Name/regex lists in T-20/T-21 are INFERENCE.
• Messaging semconv is still “development” in v1.44.0, so no stable compliance target exists; RPC is release_candidate.
• No published way to read per-attribute block size via the HTTP API (dedicated-column candidate selection needs tempo-cli analyse blocks).

5. Conflicts and version notes

• Max trace size in Cloud: usage-limits page lists max_bytes_per_trace 3,000,000 (https://grafana.com/docs/grafana-cloud/billing-and-usage/usage-limits); OTLP format page says “5 megabytes (MB) per trace, ingest rate of 15 MB/s and bursts of 20 MB/s”; Tempo OSS default 5,000,000; ingestion rate in usage-limits table is 500,000 B/s. Treat the per-stack live limit (grafanacloud_traces_instance_limits) as authoritative.
• max_attribute_bytes: distributor config default 2048 and Cloud docs say truncation at 2 KB; the per-tenant ingestion override block in the same configuration reference says default = 0 (unlimited). Do not infer truncation is off from the override alone.
• Dedicated columns: vParquet4 10 string per scope (Tempo 2.x/older docs, e.g. v2.8) vs vParquet5 20 string + 5 int (default in Tempo 3.1). Block version per stack decides the limit.
• Environment attribute: older Grafana pages list only deployment.environment; newer quality/KG pages prefer .name. Semconv: deprecated v1.27.0, new name stable v1.41.0 (2026-04-28).
• Tempo/Grafana docs lag semconv: peer.service (deprecated v1.39.0, 2026-01-12) remains a Tempo default; Adaptive Traces diversity list has rpc.grpc.status_code (deprecated v1.39.0).
• Loki OSS default index labels list only deployment.environment.name; Cloud Mimir promotion list has both names.
• Grafana “metrics labels under 30 per series” vs Loki “15 index labels” vs Pyroscope default 30 label names: different limits per signal.

6. Source index

• App O11y instrumentation quality: https://grafana.com/docs/grafana-cloud/observe-and-act/monitor-applications/application-observability/setup/instrumentation-quality/
• App O11y resource attributes: https://grafana.com/docs/grafana-cloud/observe-and-act/monitor-applications/application-observability/setup/resource-attributes/
• KG instrumentation (required metrics/labels): https://grafana.com/docs/grafana-cloud/observe-and-act/monitor-applications/application-observability-kg/setup/instrumentation.md
• KG prerequisites (environment): https://grafana.com/docs/grafana-cloud/platform/knowledge-graph/get-started/prerequisites/
• Cloud OTLP format considerations (promoted attrs, trace attr limits): https://grafana.com/docs/grafana-cloud/send-data/otlp/otlp-format-considerations/
• Tempo span metrics: https://grafana.com/docs/tempo/latest/metrics-from-traces/span-metrics/span-metrics-metrics-generator/
• Tempo service graphs: https://grafana.com/docs/tempo/latest/metrics-from-traces/service_graphs/
• Tempo cardinality: https://grafana.com/docs/tempo/latest/metrics-from-traces/metrics-generator/cardinality/
• Tempo metrics-generator troubleshooting: https://grafana.com/docs/tempo/latest/troubleshooting/metrics-generator/
• Tempo dedicated columns: https://grafana.com/docs/tempo/latest/operations/dedicated_columns/ ; schema: https://grafana.com/docs/tempo/latest/operations/schema/
• Tempo configuration reference: https://grafana.com/docs/tempo/latest/configuration/ ; API: https://grafana.com/docs/tempo/latest/api_docs/
• Tempo OOM / large attributes: https://grafana.com/docs/tempo/latest/troubleshooting/out-of-memory-errors/
• Tempo best practices: https://grafana.com/docs/tempo/latest/getting-started/best-practices/
• Cloud Traces metrics-generator: https://grafana.com/docs/grafana-cloud/send-data/traces/configure/metrics-generator/ ; reduce trace size: https://grafana.com/docs/grafana-cloud/observe-and-act/send-data/traces/configure/reduce-trace-size/
• Adaptive Traces diversity/volumetric: https://grafana.com/docs/grafana-cloud/observe-and-act/adaptive-telemetry/adaptive-traces/manage-recommendations/diversity-policy/
• Signals together / key concepts: https://grafana.com/docs/grafana-cloud/telemetry-signals/use-signals-together/key-concepts/
• Loki labels and OTel: https://grafana.com/docs/loki/latest/get-started/labels/ , https://grafana.com/docs/loki/latest/send-data/otel/
• Pyroscope: https://grafana.com/docs/pyroscope/latest/configure-client/ , …/configure-client/grafana-alloy/java/ , …/configure-client/trace-span-profiles/ , …/configure-server/reference-configuration-parameters/ , …/reference-server-api/
• Traces Drilldown access: https://grafana.com/docs/grafana/latest/visualizations/simplified-exploration/traces/access/
• OTel semconv v1.44.0 (https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.44.0): docs/general/naming.md, docs/http/http-spans.md, docs/db/database-spans.md, docs/rpc/rpc-spans.md, model/service/registry.yaml, model//deprecated/.yaml; migration guides https://opentelemetry.io/docs/specs/semconv/non-normative/http-migration/ and …/db-migration/; span-name guidance https://opentelemetry.io/docs/specs/otel/trace/api/

Part 5: Professional Services best-practice-guides mining

Repo: /Users/rob/repos/best-practice-guides (read-only mining; nothing edited). Root for all paths below: /Users/rob/repos/best-practice-guides/guides

1. TL;DR

• The repo is ~1,100 files, 2 trees: public/ (customer-facing PDFs) and internal/ (PS-only). Only ONE labelling guide is internal: internal/loki/label-analyzer-skill/. Everything else relevant is public.
• Git history is a single squashed commit (2026-10-01) so git log -1 --format=%cs is 2026-10-01 for every file and tells you nothing about freshness. Use content staleness cues instead (e.g. “Grafana Agent Flow”, “only Grafana Agent supports structured metadata”).
• The label-analyzer-skill guide is NOT the methodology itself. It is a how-to-run-it wrapper. The methodology lives upstream in the public grafana/skills repo (skills/grafana-cloud/loki-label-analyzer/SKILL.md + 4 reference files, last commit 2026-07-24), which I read read-only with gh api. It is a prompt/skill (no scripts, no numeric scoring, no code). Scoring is a qualitative banded table (cardinality band -> keep/evaluate/demote/never).
• Upstream also has prometheus-label-strategy, prometheus-cardinality-troubleshooter, adaptive-metrics, cost-management, dpm-finder skills (only first ~260 lines of the first two read). prometheus-label-strategy is the metrics twin and CONTRADICTS the PS cost-optimization guide on label dropping (see section 3 “Conflicts”).
• PS guidance is strongest and most checkable for Loki (labels, structured metadata, stream/chunk diagnostics) and for Prometheus cardinality (series maths, thresholds 100/1,000 values). It is thin or absent for profiles (Pyroscope dir is empty), Tempo attribute design, OTel semantic-convention compliance (one line each), Adaptive Logs/Traces label guidance, and any numeric cardinality thresholds for traces.
• Numbers that exist and are scanner-usable: label value counts 100+ (investigate) / 1,000+ (likely problematic); focus on metrics with 1,000+ series / top 10 by series; dynamic Loki label values “single digits or tens”; Loki chunk-size diagnostic total_bytes / cache_chunk_req (KB not MB = over-split); histogram multiplier 10-20x (PS) or bucket+3 (upstream); attribution tracker defaults max_cardinality 10000 (Tempo, Mimir) and 2000 (Loki example), max 2 Mimir attribution labels.

2. Inventory

All dates are 2026-10-01 (single-commit repo) unless stated.

Internal

Public: Loki

Public: metrics / cost / alerting

Public: traces / OTel / collectors / correlation

3. Checkable rules

Signals: L=logs, M=metrics, T=traces, P=profiles, O=OTel generic, A=attribution/cost. Short quotes are verbatim.

Logs (Loki)

Metrics

Traces / OTel / attributes

Correlation

Attribution (cost)

Conflicts, contradictions and stale items

1. labeldrop of distinguishing labels. PS …/cost-optimization/06 Strategy 2 and Scenario 1 recommend labeldrop of pod|pod_name (with a duplicate-series warning). Upstream prometheus-label-strategy and prometheus-cardinality-troubleshooter skills say NEVER drop a label that makes a series unique (counter-reset merge, inflated DPM); aggregate with Adaptive Metrics instead. Upstream is stricter. Treat PS-28/PS-29 as superseded by the stricter rule for any scanner recommendation.
2. pod for logs vs metrics. Loki guide says pod is a bad index label (PS-05). Metrics skill (upstream) says keep pod for k8s metrics. k8s meta-monitoring labelsToKeep includes pod for logs (…/k8s-meta-monitoring-overview/03-k8s-configuration.md). Rule must be signal-specific.
3. filename: PS-06 says drop/normalize; PS-15 default allowlist includes it.
4. Stale: “only the Grafana Agent” supports structured metadata (04-log-metadata.md); Grafana Agent/Flow/Promtail naming throughout; “Loki … structured metadata 2.9+”.
5. …/public/grafana-cloud/app-o11y/03-dotnet.md sets OTEL_RESOURCE_ATTRIBUTES=service_name:webmvc,service_namespace:eshop,deployment_environment:dev, wrong OTel format (needs dotted keys and =). Java example is correct.
6. …/apm-migration/06-troubleshooting.md “normalize” snippet uses merge_maps(attributes, {"user.id": "{user_id}"}, "upsert"), which writes a literal constant; works only as a dimension collapse, not real normalisation.
7. …/cost-optimization/06 “Actual: 50,000 active series due to Prometheus limits” is unsupported; …/09-traces-optimization.md example math ($2.15k -> $750 but “Savings: $350/month”). Do not copy these numbers.
8. …/ec2-tag-label-mapping/02 example promotes the AWS Name tag to instance_name, which conflicts with AWS 07’s “avoid full names”.
9. Alerts on cardinality: Grafana Cloud Cardinality dashboards cannot be alerted on (PS …/alerting-on-high-cardinality/01-introduction.md); must use queries/API.
10. I did not compare against Grafana product docs. No PS text explicitly says it is stricter than or differs from Grafana docs; the only explicit disagreements found are internal to PS/upstream skills (items 1-3).

4. label-analyzer-skill methodology (detailed)

What the internal guide is

/Users/rob/repos/best-practice-guides/guides/internal/loki/label-analyzer-skill/ is a 7-chapter SA runbook for operating a published skill (loki-label-analyzer in grafana/skills marketplace, plugin grafana-cloud@grafana-skills). It contains NO scripts and no scoring code. It references only: logcli, Loki HTTP API, helm template, Claude Code plugin commands, Grafana Assistant, Logs Drilldown, grafanacloud-usage datasource, three PS Ops dashboards on ops.grafana-ops.net (Customer Dashboard / Datasource Managed (Ruler) / Grafana Managed Alerts LogQL Query Insights). Use cases: health check, “why is Loki slow”, pre-migration design (Datadog/Splunk/ELK), cost or stream blowup, post-incident.

Inputs (02-prep-context.md)

Required: (1) label set per log source (logcli labels or GET /loki/api/v1/labels); (2) cardinality per label (logcli labels <name>, logcli series '{sel}', Logs Drilldown table); (3) 3-5 sample log lines per source (redacted); (4) access patterns (which labels dashboards/alerts filter on; derive from query logs or the PS Ops dashboards). High value: full Alloy pipeline (render with helm template ... --show-only templates/alloy-logs/configmap.yaml), a slow query plus its metrics.go line, limits_config (allow_structured_metadata, max_global_streams_per_user, cardinality_limit), Loki version (>=2.9 for SM), business goals (cost, speed, LBAC). Env: LOKI_ADDR, LOKI_USERNAME (numeric tenant), LOKI_PASSWORD (CAP token with logs:read). Optional Grafana MCP with a Viewer service account in --disable-write. Guardrail: do not evaluate without cardinality data; do not paste PII.

Heuristics (public SKILL.md, read via gh api; guide 04 summarises as “Good / Acceptable / Avoid / Never”)

Source: https://github.com/grafana/skills/blob/main/skills/grafana-cloud/loki-label-analyzer/SKILL.md (last commit 2026-07-24).

• Core: stream = unique label set. “Dual impact”: ingest path (more streams, bigger index) and query path (a high-card label absent from the selector forces a scan of all matching streams).
• Key question: “Will this label be used in 9 out of 10 queries?” If no, not a label, EXCEPT protected correlation labels.
• Cardinality scoring table:
  • service_name / deployment_environment / job: any cardinality -> keep key, remediate values
  • env 2-5 values: good; level 3-6: good; namespace tens: acceptable
  • instance/hostname hundreds-thousands: evaluate access patterns
  • pod thousands + transient: demote to structured metadata (migrate selectors first)
  • user_id, request_id: unbounded, never a label
• Access-pattern checklist: on protected allowlist? used in most queries? segments data as users think? would demotion break alerts/dashboards/LBAC/correlation? force scanning more data?
• Static vs dynamic: static values cost nothing relative to scope; dynamic values must be single digits to low tens.
• Consistency: case-sensitive names, normalised values, one naming convention (snake_case or camelCase).
• Protected labels (references/protected-labels.md): service_name (OTel service.name), deployment_environment (deployment.environment), job. Never drop or omit from label_keep; fix values (UUID/ephemeral service_name -> stable service identity). app/service are aliases: align to service_name, do not delete without migration plan. Downstream-dependency guardrail required before any demote/rename.
• Labels-to-avoid table: pod, user_id, request_id/trace_id, raw k8s filename, unnormalised level, UUID service_name values, any dynamically-named label key.
• Performance diagnosis from metrics.go: 4 stages: queue (queue_time), index (chunk_refs_fetch_time), storage (store_chunks_download_time), execution (duration - chunk_refs_fetch_time - store_chunks_download_time); ideal is mostly execution time. Avg chunk size = total_bytes / cache_chunk_req (KB = over-split). post_filter_lines << total_lines = low selectivity.
• 80/20 rule: demote pod; add level and always query it (can cut 94%+ scanned); normalise label values; normalise/demote k8s filename.
• Quick wins also: add container/workload, |= over |~.
• Recommended common labels: service_name, deployment_environment, job, app/service (legacy), env, cluster, region, level, team/squad, source, classification; k8s: service_name, namespace, container, workload; host: instance, filename (normalised); journal: instance, unit.
• Enforcement: stage.label_keep as final stage always including protected labels; soft enforcement via stage.template injecting unknown.
• Sequencing (internal guide 05): normalise level -> normalise filename -> add workload -> remove pod (last, after dashboards/alerts updated) -> enforce label_keep (lock schema, roll out behind flagged config). Non-prod first, watch a full day for value drift.

Scoring

There is no numeric score. Output is a per-label verdict (Keep / Evaluate / Demote / Never; emoji tiers) with Used-in-Queries and Action columns. The Assistant paste-prompt names the four grades “Good / Acceptable / Avoid / Never”.

Output format (fixed)

## Loki Label Strategy Audit with sections: Disclaimer (mandatory verbatim text from references/disclaimer.md, must contain “Confidential Information of Raintank, Inc.”), Summary, Downstream dependency check, Label Analysis table (Label | Cardinality | Used in Queries? | Verdict | Action), Estimated Impact (stream reduction, query perf, storage, correlation impact), Cost Impact Analysis, Recommended Label Set (must include protected labels), Migration Notes (Alloy stages, dual-write). Internal guide 05 adds a 3-doc handoff: label-audit.md, label-audit-summary.md, label-migration-plan.md.

Cost impact (references/cost-impact.md)

Label hygiene alone gives $0 direct ingest savings (billing is compressed bytes); it reduces streams/index/query cost. Scenarios: A labels only (streams -65 to -90%), B + approved debug/trace drop (~15-30% volume, customer-approved, env-scoped), C + log-line compaction (additional ~5-15%; Istio example 38% ceiling). Monthly GB = bytes/s x 86400 x 30 / 1e9. Measurements (Grafana Cloud usage datasource): grafanacloud_logs_instance_active_streams, grafanacloud_logs_instance_billable_bytes_received_per_second, grafanacloud_org_logs_overage{monetary="true"}, sum by (<attr_label>) (grafanacloud_logs_instance_attributed_bytes_received_per_second) (unattributed share shows as __missing__; Cloud attribution uses up to two customer-configured labels). Verification metrics in internal guide 05: loki_ingester_streams_created_total rate down, loki_distributor_lines_received_total unchanged, chunk size up, p95 latency, grep dashboard JSON for removed labels. Most loki_* metrics are not visible to Cloud customers: use grafanacloud-usage and PS Ops dashboards. Log-line optimisation (references/log-line-optimization.md): strip inline timestamps (~30-34 bytes, ~6%), stage.decolorize, duplicate level, JSON nulls/placeholders/empties.

Reusable by an automated estate-wide scanner

• Deterministic checks that map directly from the skill/guides, all read-only API:
  • Loki GET /loki/api/v1/labels + /label/<n>/values (or logcli series --analyze-labels '{}' equivalent via /loki/api/v1/series): per-label unique value count and stream count; compare to bands (<=6 good, tens acceptable, hundreds-thousands evaluate, thousands/unbounded flag).
  • Flag-by-name: pod, filename, user_id, request_id, trace_id, case-variant duplicates (Level vs level), level value set not in {info,warn,error,debug,…}, UUID-shaped service_name values, absence of service_name/deployment_environment/job.
  • Total active streams vs limit: grafanacloud_logs_instance_active_streams against grafanacloud_logs_instance_limits.
  • Unattributed share: __missing__ in attributed bytes metric.
• Metrics twin: per-label unique value counts via Mimir cardinality API /api/v1/cardinality/label_names|label_values and TSDB status; thresholds 100/1,000 values, 1,000+ series per metric; histogram _bucket amplification; churn (created/removed).
• Traces: grafanacloud_instance_active_spanmetrics_series and grafanacloud_traces_instance_metrics_generator_series_limit_percentage_used.
• Judgement-only (not scannable without access patterns): “used in 9 of 10 queries”, downstream dependency (needs query logs/dashboard JSON). A scanner should emit these as “needs access-pattern input” rather than a verdict.
• Do not copy: the Disclaimer text (Raintank confidential) is part of PS report output, not a scanner concern.

Upstream adjacent skills worth reading before building (outside this repo)

grafana/skills skills/grafana-cloud/prometheus-label-strategy/SKILL.md (cardinality bands: env 2-5, job 5-50, cluster/region tens, namespace tens-low hundreds, instance hundreds-low thousands evaluate, pod keep, path only templated, version/image_tag churn -> info metric, user_id/request_id/trace_id/error_message never; histogram multiplier bucket+3, 11 default buckets -> 14x; “The One Rule” never labeldrop distinguishing label), prometheus-cardinality-troubleshooter (tsdb status endpoint, any label with >10K unique values is almost certainly a bug, growth > a few % per day red flag), adaptive-metrics, cost-management, dpm-finder. Only partially read.

5. Gaps

• Profiles: /Users/rob/repos/best-practice-guides/guides/public/pyroscope/ is empty. Only a .NET env example PYROSCOPE_LABELS=namespace:eshop,environment:dev in …/grafana-cloud/app-o11y/03-dotnet.md. No profile label guidance.
• Tempo/trace attributes: no guidance on span attribute cardinality, dedicated columns, generic_dimensions, or which attributes to index; only span-metrics dimension examples and TraceQL scoping.
• OTel semantic conventions: one-line exhortations only; no registry/Weaver, no attribute-placement rules (resource vs span vs metric), no compliance checks. OTLP-to-Loki index label promotion (Grafana Cloud default promoted resource attributes) not covered.
• Adaptive Logs / Adaptive Traces: cost-level only (Strategy 1), no label/attribute requirements or exemptions, no guidance on how labels affect Adaptive Logs patterns.
• Mimir/Prometheus: no limits discussion beyond sample alert JSON (no per-metric series limit, max_label_names_per_series, label value length), no churn thresholds, no recording-rule label guidance.
• Structured metadata: no cardinality/size limits, no guidance on SM key count, bytes (only total_bytes_structured_metadata field described).
• K8s Monitoring Helm: only OpenShift and meta-monitoring pages touch labels; no recommended labelsToKeep/podLogs label policy for general use beyond the legacy Agent allowlist.
• No machine-readable assets: no scripts, schemas, CSVs or code for label analysis anywhere in the repo (tools/ is lint/build only).
• No freshness signal (single squashed commit); the internal skill guide’s own claims (e.g. “Alloy rather than legacy Agent”) are consistent with upstream, but several public guides are Grafana-Agent-era.
• Upstream SKILL.md changed after the internal guide was written (protected labels, mandatory disclaimer, cost-impact scenario cards are not described in the guide’s 01-intro). Re-read upstream, not the guide, for current behaviour.

Loop14 catalogue source re-verification (2026-10-07)

Current uncached full-page passages below were successfully retrieved. Firecrawl CLI reported Not authenticated, so usage was keyless rather than attributed to a per-home key. This is an attribution/runtime-wiring caveat, not a failed content retrieval; no credentials were created or changed. Root accepts the observed dated passages, not a claim of authenticated lookup. Unverified rules remain excluded.

The owner-requested inclusive 10,000 critical threshold is policy, not a misquotation of the published strictly-greater-than-10,000 PS band. Population floor, static-infrastructure multiplier and tunable overrides are policy. Successful retrieval does not prove source collector runtime, route scope or raw-value privacy; those remain separate source acceptance checks.