Research what Grafana Cloud feature and sub-feature usage the platform can observe, and from which source

Description

Why

Field question (2026-09-23): “Does anyone have a customer who is successfully tracking overall Grafana feature usage? Usage Insights shows dashboards, but what about the other UI experiences?” Other SEs have customers asking for the same thing. Nobody has a good answer today, and this platform already holds most of the credentials that could produce one.

The frozen question for this research: for each Grafana Cloud product and its main sub-features, what adoption and usage signal can this project obtain, from which source, at what fidelity, with which credential, and at what cost or risk? The deliverable is an evidence-backed matrix and a ranked set of follow-on implementation tasks. It is not an implementation.

The first concrete strand is already scoped as subtask GCI-0032.01: the source field on usage-insights data-request events, which splits queries by Grafana surface (dashboard, Explore, alerting, Scenes apps, Assistant). That subtask’s upstream contract notes and traps are the model for the rest of this work. In particular, every Scenes-based app plugin appears to report source="scenes" rather than its own id, so per-app attribution needs a different signal. Finding that signal is one of the questions here.

Fidelity ladder (every matrix cell names its level)

1. Available - entitled or provisioned on the stack (instance exists, plugin installed).
2. Configured - objects exist: N checks, N SLOs, N integrations, N Faro apps, N clusters. A count of config, not of use.
3. Producing - the feature is generating or ingesting data: billing or usage metrics above zero, sentinel metrics present in the stack’s own Mimir.
4. Used by people - humans query or open it: usage-insights events, distinct users, Assistant usage APIs.

“Installed” must never be presented as “used”. Most products will only reach levels 1 to 3; say so plainly rather than stretching a level-2 figure into an adoption claim.

Sources to probe (all of them, for every product)

1. Grafana.com control plane, org-realm reader (collector/sources/gcom.py): stack inventory and detail fields, installed plugins per stack, datasourceCnts, product instance ids and URLs (SM, k6, IRM, profiles, etc.), anything that marks a product as enabled.
2. grafanacloud-usage billing and usage metrics on the write stack (collector/sources/capability_adoption.py already reads ten). Enumerate every grafanacloud_* metric name live, for example group by (__name__) ({__name__=~"grafanacloud_.+"}) over a 24h range. Map each to product and sub-feature, and note whether it carries stack_id. Record which are cumulative counters, which are rates and which are billing-period totals, following the windowing rules already in capability_adoption.py.
3. Per-stack usage-insights datasource (collector/sources/usage_insights.py): beyond source (GCI-0032.01), check whether datasourceType, panelPluginId, panelName or any other logfmt field identifies a product. Product-specific datasource types include Synthetic Monitoring, Asserts/Knowledge Graph, k6, SLO and Infinity. Is a dashboard owned by a product app, like the SM or K8s dashboards, identifiable from its uid or folder?
4. Per-stack Grafana API with the existing reader role (collector/provision.py; CAPABILITIES.md “Stack-local reader”): what each product’s app plugin exposes through /api/plugins/<id>/resources/... or its own API that a read-only action could reach. That covers /api/plugins list and settings (enabled, pinned, jsonData present), SLO, Synthetic Monitoring, Knowledge Graph (Asserts), Application Observability, Kubernetes Monitoring, Frontend Observability, IRM (OnCall and Incident), k6, Fleet Management, Cloud Provider, Database Observability, Machine Learning, Drilldown app settings, Investigations, Reporting, Recorded queries, Library panels, Playlists and Git sync / provisioning.
5. Per-stack data plane, org-realm reader: inference from telemetry the product writes into the stack itself. For example: SM sm_check_info and probe_*; Knowledge Graph asserts:* recording rules or entity series; App O11y span metrics and traces_target_info; Kubernetes Monitoring build-info and kube_*; Faro/Frontend series; Beyla; Fleet-managed collector series. The technology registry (collector/technology_registry.py, technology-registry.json, GCI-0008.03/0015/0031) already does sentinel matching; extend it rather than building a second matcher.
6. Existing collectors: Pillar I Assistant, Adaptive Metrics/Logs/Traces, Fleet, alert routing and rules, public dashboards, service accounts. Record what they already answer so the matrix does not propose rebuilding them.

Products and sub-features in scope (extend as live discovery finds more)

• Drilldown apps: Metrics, Logs, Traces, Profiles. Per-app, not just the scenes bucket.
• Application Observability: services onboarded, service map, span-metrics source (Tempo metrics-generator vs Alloy/Beyla), OTel vs Beyla instrumentation.
• Knowledge Graph (Asserts): entity graph populated, assertions, RCA workbench use, KG-backed SLOs, integrations enabled.
• Synthetic Monitoring: checks by type (HTTP, ping, DNS, TCP, traceroute, MultiHTTP, scripted, browser), public vs private probes, check alerts. SPEC.md lists SM result inventory as out of scope until a safe unattended read route exists; this research should establish whether one exists now.
• k6 / Performance testing: projects, tests, runs, VUh (already partly read), browser tests, cloud vs local execution.
• Frontend Observability: apps, sessions (already partly read), sourcemaps uploaded.
• Kubernetes Monitoring: clusters, Helm chart and Alloy versions, cost features.
• SLO: SLO count, SLOs with alerting, SLO source (metrics vs KG).
• IRM: OnCall integrations, escalation chains, schedules, alert groups (already partly read); Incident declared incidents.
• Alerting: Grafana-managed vs datasource-managed rules, recording rules, contact point types. Much of this already exists; place it in the matrix.
• Assistant and Investigations: Pillar I already covers much of it; add Assistant-driven querying from source="grafana-assistant-app".
• Adaptive Telemetry: Metrics, Logs, Traces, Profiles; rules applied vs recommendations pending.
• Fleet Management / Alloy: collectors, pipelines, remote config in use.
• Cloud Provider Observability (AWS, Azure, GCP), Integrations / Connections, Database Observability, Private Data source Connect.
• Machine Learning: forecasts, outlier detection, Sift.
• Profiles (Pyroscope): continuous profiling in use (GCI-0019 already measures it).
• Core UI surfaces: Explore, correlations, notebooks, public dashboards, reporting, snapshots, library panels, playlists, dashboards-as-code / Git sync. The source strand covers the query-driven ones.

Rules that must hold

• Read-only research. Use existing credentials and existing scopes. Do not widen custom:gcinsight.reader, mint anything, or change any access policy to test a hypothesis. When a route needs a new action or scope, record the exact (action, scope) pair, what it would read, whether it touches secrets or production data, and the blast radius. Put it in the matrix as a decision for Rob, not a grant. The reader stays basic-role None and query-scoped; datasources:query is never widened beyond the two pinned uids.
• The collector HTTP client is GET-only by construction. Record any route that is a POST (Connect-RPC or similar) as such. It needs the same scrutiny the existing Connect-RPC read routes got.
• Every usage-insights query keeps the instance_id guard (see docs/traps.md and the usage_insights.py docstring).
• Probe several stacks of different size and product mix. One stack is an anecdote.
• Record negative results. A route that 401s, 403s for a service-account identity by product design (like Assistant watcher-agents), returns zero for an SA (like /api/v2/investigations) or returns 200 with wrong data is as valuable as a working one. These go into CAPABILITIES.md “Known unavailable or rejected routes”.
• No customer identifiers, stack slugs, user names or tokens in anything committed. Use anonymised or synthetic examples.
• Cardinality: every proposed metric states its series cost (live stacks x enum size) and why it needs a time series rather than a view, per AGENTS.md.

Deliverable

1. A backlog doc, “Feature usage observability matrix” (create it with backlog doc create, never by hand). Rows are product/sub-feature. Columns: best fidelity level reachable, source, exact route/query/metric, credential and scope (existing or needed), live-verified yes/no plus on how many stacks, per-stack or estate, cardinality cost if emitted, known traps, and a recommendation (build now / build after scope decision / not obtainable / out of scope).
2. Verified routes and negative results added to CAPABILITIES.md, and new traps to docs/traps.md.
3. One follow-on Backlog task per “build now” row or coherent group, created via the CLI, ranked by value to the field question vs cost. Reference this task and the matrix doc. Group the scope-widening rows into a single decision task for Rob.
4. A short answer to the original field question, written into the matrix doc’s summary: what a customer can track about Grafana feature usage today with this platform, what it could track after the follow-ons, and what no available source can see.

Acceptance Criteria

• #1 Fresh customer shared billing census is recorded with current/historical metric-name differences, retaining uncertainty about absent names.
• #2 Bounded feature-signal reads are reconciled to freshly discovered inventory and their observed reporting/positive counts are recorded.
• #3 Matrix records source metadata gaps and no new grant/permission or unsupported adoption/billing semantics; original exhaustive requirements are explicitly owner-withdrawn.

Definition of Done

• #1 just test
• #2 just tf-validate
• #3 just check-identifiers and just no-em-dashes both return clean

Implementation Plan

loop3 D1a/c: family-route and minimal-pair packets plus cited billing/window semantics matrix; root CLI matrix update, keep unknown semantics explicit, no evidence-free completion.

2026-10-13 direct-main continuation: reconcile the durable observability matrix against accepted reader dispositions and retained producer contracts. Correct stale AI counter and generic commercial identity assumptions, distinguish source evidence from absent deployed proof, and preserve unresolved acceptance criteria. No new live tenant queries, source code, reader grants or pricing claims.

Implementation Notes

Upstream evidence for the source-field strand (Grafana surface attribution, Scenes ‘scenes’ bucket, Assistant source value) is archived verbatim in GCI-0032.01’s implementation notes. Headline for this research: usage-insights has only dashboard-view and data-request events; data-request.source is request.app; Scenes-based app plugins appear to report ‘scenes’ unless their scene root implements enrichDataRequest, so per-app attribution (Drilldown, App O11y, K8s, KG) needs another signal. Candidates to test first: datasourceType split within scenes, panelPluginId, installed-plugin inventory from gcom, and product sentinel series in each stack’s Mimir via the technology registry.

Existing reader scopes (collector/provision.py READER permissions) at task creation: Assistant, Adaptive Logs/Metrics/Traces and a databases-config plugin via plugins.app:access; serviceaccounts, datasources (read and caching read), folders, dashboards, snapshots, teams, user roles, roles and alert rules/receivers read; datasources:query pinned to grafanacloud-usage-insights (plus grafanacloud-usage on the write stack only). None of SLO, Synthetic Monitoring, Knowledge Graph, App O11y, K8s Monitoring, IRM, k6 or Frontend O11y plugin resources are in scope today.

Wave 1 matrix doc-0006 covers 34 product/subfeature groups and an exact 325-name live grafanacloud_* union from two write-stack usage datasources. It records verified scope, explicit unknown billing/window semantics, nine-stack API route probes and 55 guarded usage-insights stack probes. CAPABILITIES and docs/traps updated; GCI-0034 through GCI-0041 capture follow-ons and the scope decision. AC1 remains unproven because vendor contracts for many billable gauges and counters did not establish rate/counter/billing-period window semantics; the appendix marks these unknown instead of guessing. Resume with authoritative metric-definition or billing export evidence for each unknown family, then complete AC1. Exact-SHA CI 35908750277 passed; historical identifier gate remains red on eight pre-wave commits, while current-file and hosted gates passed.

Criterion audit correction: AC2 and AC4 also stay open. Several candidate plugin routes are wildcard or unverified and their exact reader action/scope pairs are unknown; GCI-0041 is the explicit scope-decision and route-validation boundary. The matrix labels these gaps rather than presenting candidate routes as confirmed contracts.

loop3 admitted 2026-09-30 under owner goal: root owns tracker; bounded lanes own implementation/discovery, evidence pending. No acceptance claimed yet.

loop3 D1c mapping accepted and durable matrix updated via CLI: 327-name roster across 19 families, 26 partial/exact contracts and 309 unresolved normalizations. OnCall alert_groups_total corrected to explicit gauge; response/resolution histograms last 7d; quota_usage current count not billing. AC1 remains unproven pending vendor HELP/TYPE/recording windows/reset catalogue; no guesses, no role changes in research. D1a route/pair proof remains active root follow-up. Discovery attempts 0, implementation attempts 0, infra retries 0 for this research criterion.

Loop3 matrix updated via CLI:327 discovered metric names,26 partial contracts,309 unresolved normalizations. Authoritative HELP/TYPE/window/reset contracts unavailable; no guessed normalizations or semantic-zero claims. Real Logs frontend sourceapp transport proved underSA control, not human or per-app discriminator. Resume from vendor contract publication and strict known-user proof, not repeated searches with no new premise.

loop4 M-0032 admitted read-only with new premise: live grafanacloud-usage metadata HELP/TYPE endpoint, followed by named official sources if needed. No guessed normalization, grants, or acceptance yet.

loop4 metadata premise exhausted with complete5/5 direct discovered usage datasource metadata GETs across fresh staff inventory: every HTTP200 success has empty data, zero HELP/TYPE contracts. Four existing official partial pricing contracts revalidated, zero net-new contracts. Unsupported proxy404s from pass1 corrected with explicitly authorised stored per-stack SA direct reads. AC1/AC2 remain parked pending vendor recording/window/reset catalogue, no repeated unchanged search planned.

2026-10-13 evidence reconciliation identifies exact source-level corrections for plugin billing quantities, AI current-month gauges and overage attribution. Update the durable matrix with their qualified contracts while preserving original live observation dates. Do not replace the historical unresolved-normalization count with a guessed new total: reclassify individual rows first and mechanically recount only under an explicit counting definition. Source review is not deployed verification, so live contract acceptance remains open. Reconcile historical scope-decision paragraphs with subsequently accepted product dispositions without implying a fresh role or route grant. No repeated empty metadata probe is justified by this documentation work.

2026-10-13 matrix reconciliation completed: added qualified retained-source contracts for plugin rating quantities, AI current-month gauges and per-family monetary identity mappings, plus current IRM/optional-reader dispositions while preserving dated historical probes. Corrected the unconditional lifetime-counter claim in operator traps. No new metric/source/permission, positive deployed witness or acceptance completion claimed; remaining live semantics and route gaps keep this parent Parked.

Owner 2026-10-14 authorizes additional read-only dev/customer stack queries across as many discovered stacks as needed. Prioritize customer-realistic evidence and retain unknown metric semantics, no permission changes.

2026-10-14 owner-authorized customer refresh completed: 325 currently discovered names, 12 new to historical tables (11 entirely new to document), 12 signal probes left-joined to fresh316 inventory; all observed IDs matched. Matrix updated with counts and exact names. Customer metadata still success/empty: no HELP/TYPE/reset/window proof. No role/permission/token writes. Original exhaustive AC1/2/4 remain unproved; bounded attempt complete, no unchanged probes planned. Evidence /Users/rob/repos/grafana-cloud-org-insights/codex/direct-20261014/backlog-zero/features032/completion-receipt.json.

Owner2026-10-14 accepts the bounded customer refresh and retires exhaustive metric normalization/exact universal route research. Original unchecked AC1/2/4 withdrawn, not proved; dated matrix contracts and unknowns remain. Revised acceptance is actual fresh325-name census and twelve inventory-left-joined reporting witnesses with documented missing metadata; no permission/credential writes.

Final Summary

Owner accepted bounded refresh:325currentnames,11newtodocument,12fixed24h probes matched316freshinventory. Sanitized matrix appendix and count/hash receipts retained. Exhaustive remaining contract research withdrawn rather than represented as proved; existing semantic gaps remain documented.