Break down Pillar J query usage by Grafana surface from the data-request source field

Description

Subtask of GCI-0032, the wider feature-usage research. This is its first concrete strand and can be built as soon as step 0 below holds. It does not wait for the rest of the research.

Why

Field question (2026-09-23): “Does anyone have a customer who is successfully tracking overall Grafana feature usage? Usage Insights shows dashboards, but what about the other UI experiences?” Other SEs have customers asking for the same thing.

Pillar J today answers dashboards only. It groups data-request events by dashboard, panel and datasource type, but never by the event’s source field, which we already ingest and discard. Upstream Grafana populates source from the query request’s app. Grouping on it answers “which Grafana surfaces does this estate actually query through”: dashboards vs Explore vs alerting vs the Scenes-based app plugins (Drilldown and friends) vs Assistant. That is the closest thing to feature-usage tracking any source we can read supports, and it shows how far the modern app-based experiences have spread.

Usage insights emits only two event types, dashboard-view and data-request (see the module docstring in collector/sources/usage_insights.py). Nothing records a page visit that ran no query: opening the alert list, browsing an app’s landing page, admin screens. This task measures queries run per surface, not page visits. Every label and banner must say so.

Upstream contract (read from source 2026-09-23, grafana/grafana main)

public/app/features/query/state/queryAnalytics.ts, emitDataRequestEvent, wired into the central runRequest.ts via tap(emitDataRequestEvent(datasource)). So every query that goes through runRequest emits one event, including Scenes queries:

• source: data.request.app
• Also carries panelPluginId (visualisation type). Not collected today. Out of scope here; note it as a follow-up.
• Trap 1: if (params.editPanel != null) return; - a query run with editPanel in the URL emits nothing. Classic panel-editor use is therefore under-counted or absent. Expect panel-editor to be near zero; that does not mean nobody edits panels.
• Trap 2: error is only populated when app is not explore and not correlations. Explore and correlations error rates cannot be measured from this field. Do not publish a per-surface error rate for those two, and do not let their zero errors lower an estate error rate.
• Trap 3: dashboardUid/dashboardName come from getDashboardSrv().getCurrent(), not from the request. Check live whether an Explore or app query can carry a stale dashboard uid after navigating away from a dashboard. Do not use dashboardUid!="" as a proxy for “dashboard surface”; use source.

CoreApp enum (packages/grafana-data/src/types/app.ts): cloud-alerting, unified-alerting, dashboard, notebook, explore, correlations, unknown, panel-editor, panel-viewer.

Values set outside the enum:

• scenes - grafana/scenes SceneQueryRunner.prepareRequests hardcodes app: 'scenes' and then spreads getEnrichedDataRequest(this), so a scene root that implements enrichDataRequest overrides it. DashboardScene.enrichDataRequest returns app: CoreApp.Dashboard, so Scenes-powered dashboards still report dashboard. NotebookScene returns CoreApp.Notebook. A GitHub code search found no enrichDataRequest in grafana/logs-drilldown, metrics-drilldown, traces-drilldown or profiles-drilldown. The search was negative and may be incomplete, so verify live. Expected consequence: every Scenes-based app plugin (the four Drilldown apps and probably Application Observability, Kubernetes Monitoring and others) reports the same scenes bucket, and source alone cannot say which app.
• grafana-assistant-app - the Assistant plugin emits its own data-request events with source: 'grafana-assistant-app' for queries the agent runs. Read from plugin source; not yet seen live, so confirm in step 0. That is a direct measure of Assistant-driven querying, complementary to Pillar I’s Assistant API figures.
• Any plugin can emit an arbitrary source string. Treat the field as unbounded.

Step 0: live discovery before building (gate)

Run this on at least three stacks of different sizes and on a stack known to use a Drilldown app. Use a 7d window. Use the per-stack reader through the datasource proxy, exactly as _query does, and keep the mandatory instance_id guard:

sum by (source) (count_over_time({instance_type="grafana", instance_id="<stack id>"} | logfmt | eventName="data-request" [7d]))

Then answer, and write the answers into docs/traps.md:

1. Which source values appear live, with rough shares. Is scenes present and large?
2. Does a Drilldown query arrive as scenes, as its plugin id, or as something else? If it is scenes, does anything on the event tell the apps apart? Check panelPluginId, panelName, dashboardUid and whether the logfmt line carries any plugin or URL field. Try sum by (source, datasourceType) on the scenes bucket as a weak discriminator: Loki-only suggests Logs Drilldown, Tempo Traces Drilldown, Pyroscope Profiles Drilldown, Prometheus Metrics Drilldown or another app. Report that as inference, never attribution.
3. Does userId appear on data-request lines, so distinct people per surface can be counted in Loki?
4. Trap 3 above: does a non-dashboard source ever carry a non-empty dashboardUid?

If the live answers break the design below (for example source is absent on Cloud’s ingested lines, or everything arrives as unknown), stop and return the findings instead of building.

What to build (after step 0 holds)

Source (collector/sources/usage_insights.py)

• A breakdown surface_requests: sum by (source) (count_over_time(_REQ [WINDOW])), parsed through _series. It goes through _execute_query/_assert_scoped like every other query, so there is no new stack-identity path.
• If step 0 shows userId present, add surface_users: count by (source) (sum by (source, userId) (count_over_time(_REQ | userId!="-1" | userId!="" [WINDOW]))). Distinct people per surface is the adoption figure; request counts over-weight a single auto-refreshing Explore tab. Aggregate in Loki; identities never leave it.
• The window stays at the pillar’s 24h. If a 7d or 31d reach figure is wanted, it is a separate query with its own window constant, like ACTIVITY_WINDOW, never a relabelled 24h figure.

Surface enum (the cardinality boundary)

• A closed mapping from raw source to a bounded surface enum. Suggested: dashboard, explore, alerting (unified-alerting + cloud-alerting), app_scenes (scenes), assistant (grafana-assistant-app), notebook, panel (panel-editor + panel-viewer), correlations, unknown (unknown and empty), other (anything unmapped). Adjust to what step 0 finds; do not add an enum value for something not observed.
• Raw source strings never become metric labels. They are plugin-chosen and unbounded.
• Unmapped raw values go into a view so new app sources get noticed and mapped: stack, raw source, requests. Bound it with topk per stack.

Pillar (collector/pillars/insights.py)

• Per-stack view insights_surface_usage: stack, requests per surface, share of stack requests per surface, and distinct users per surface if available. Add it to VIEW_SCHEMAS, because a stack with no events is a legitimate empty row set.
• Estate view: per surface, total requests, stacks using it (requests > 0), and distinct users summed across stacks. The sum double-counts a person active on two stacks, so the column header must say “sum of per-stack users”.
• Unmapped-source view as above.
• Metrics: justify each one against the AGENTS.md “needs a time series” bar. The trend is the point here (“is Explore/Drilldown adoption growing after enablement”), so at minimum an estate-level gcinsight_dashboards_estate_surface_requests{surface} and gcinsight_dashboards_estate_surface_stacks{surface} (~10 series each). A per-stack {stack, surface} metric costs live stacks x enum size (~2,700 at the 271-stack planning figure). Include it only if a per-stack trend panel or alert needs it; otherwise the per-stack detail is the view. Every metric goes into budget.py CATALOGUE and BUDGET.md is regenerated, never hand-edited.
• A stack whose usage-insights probe is unavailable is absent, never zero. insights_coverage stays the denominator.

Hydration: the new fields live inside the existing insights input (T2), so no new INPUT_OWNER entry should be needed. VIEW_INPUTS is derived: extend the synthetic compose fixture and let tests/test_hydrate.py re-derive the table. Never hand-edit it.

Dashboard (bin/dashboards.py d_dashboards, Pillar J): add a “Grafana surfaces” row or tab to the existing dashboard-usage surface rather than a twelfth dashboard, which would need DASHBOARDS, BUILDERS and PILLAR_OF. Minimum: an estate share-by-surface panel, a stacks-using-each-surface stat, a per-stack table from the view, a trend of the estate metric, and the unmapped-source table. The row description must state:

• queries run, not page visits;
• scenes means “Scenes-based app plugins (Drilldown and others)” and is not split per app unless step 0 found a real discriminator;
• Explore and correlations carry no error data upstream;
• panel-editor use is suppressed upstream.

The coverage gate requires every new view and metric to be rendered or alerted.

Docs: add the traps above and step 0’s live findings to docs/traps.md. Update SPEC.md section 8’s Pillar J paragraph with one sentence on surfaces. BUDGET.md is generated.

Rules that must hold

• The instance_id guard and _assert_scoped apply to every new query; a region-wide surface count is the exact defect the guard exists for.
• No identity, raw source string, dashboard uid or plugin id becomes a metric label.
• A limited --stack or --limit run cannot publish; use --dry-run for live checks (see RUNBOOK.md).
• Word it as “N stacks queried through Explore in the last 24h”, never “N stacks use Explore”. The source cannot see non-query use.

Out of scope / follow-ups

• panelPluginId, the visualisation-type usage mix. Same event, same pattern. Log it as a separate task if wanted.
• Per-app attribution inside scenes. It needs an upstream change (Drilldown apps implementing enrichDataRequest with their plugin id) or a different source. Record the gap and do not guess.
• Feeding surface adoption into the Pillar D maturity rubric.

Acceptance Criteria

• #1 Under the explicit loop24 documentation disposition, recorded step-0 staff discovery and known frontend/human samples are synthesized into /Users/rob/repos/grafana-cloud-org-insights/docs/traps.md: observed sources and mix, generic app/scenes attribution limits, userId field presence and bounded stale-dashboardUid findings; exact known-human request-to-app correlation remains explicitly unproven and is not claimed as acceptance.
• #2 usage_insights.py adds a surface_requests breakdown (and surface_users if userId is present) through _execute_query and _assert_scoped, with the instance_id guard
• #3 Raw source values map to a closed surface enum; unmapped values land in a bounded per-stack view and never in a metric label
• #4 Per-stack and estate surface views published; per-stack view declared in VIEW_SCHEMAS; unavailable stacks are absent, not zero
• #5 Every new metric is in budget.py CATALOGUE with its series cost justified; BUDGET.md regenerated; VIEW_INPUTS re-derived by test, not hand-edited
• #6 Pillar J dashboard gains a Grafana surfaces row whose description states queries-not-page-visits, the scenes bucket meaning, the Explore/correlations error gap and panel-editor suppression; coverage gate passes
• #7 No per-surface error rate is published for explore or correlations

Definition of Done

• #1 just test
• #2 just tf-validate
• #3 just check-identifiers and just no-em-dashes both return clean

Implementation Plan

loop3 D1b completes controlled known Drilldown query evidence and stale-dashboard/source observations; documentation lane records supported/unavailable outcome, P-0043 unblocks history gate.

Implementation Notes

Evidence archived at task creation (2026-09-23, read-only, nothing live queried)

Upstream read at grafana/grafana main 1614865a1b27 and grafana/scenes main 8df03f3809bf.

grafana/grafana public/app/features/query/state/queryAnalytics.ts, emitDataRequestEvent

const params = urlUtil.getUrlSearchParams();
if (params.editPanel != null) { return; }
...
const eventData: DataRequestEventPayload = {
  eventName: MetaAnalyticsEventName.DataRequest,
  source: data.request.app,
  datasourceName: datasource.name, datasourceUid: datasource.uid, datasourceType: datasource.type,
  dataSize: 0, panelId: 0, panelPluginId: data.request?.panelPluginId,
  duration: data.request.endTime! - data.request.startTime, ...
};
enrichWithInfo(eventData, data);   // dashboardUid/Name/folderName from getDashboardSrv().getCurrent()
if (data.request.app !== CoreApp.Explore && data.request.app !== CoreApp.Correlations) {
  enrichWithErrorData(eventData, data);
}

grafana/grafana public/app/features/query/state/runRequest.ts: tap(emitDataRequestEvent(datasource)) inside runRequest, which is the central query path.

grafana/grafana packages/grafana-data/src/types/app.ts:

export enum CoreApp {
  CloudAlerting = 'cloud-alerting', UnifiedAlerting = 'unified-alerting', Dashboard = 'dashboard',
  Notebook = 'notebook', Explore = 'explore', Correlations = 'correlations', Unknown = 'unknown',
  PanelEditor = 'panel-editor', PanelViewer = 'panel-viewer',
}

grafana/scenes packages/scenes/src/querying/SceneQueryRunner.ts, prepareRequests:

let request: DataQueryRequest<DataQueryExtended> = {
  app: 'scenes',
  ...
  // This asks the scene root to provide context properties like app, panel and dashboardUID
  ...getEnrichedDataRequest(this),
};

grafana/grafana DashboardScene.enrichDataRequest returns app: CoreApp.Dashboard, and NotebookScene.enrichDataRequest returns app: CoreApp.Notebook. So Scenes-powered dashboards still report dashboard.

GitHub code search for enrichDataRequest: one hit set, in grafana/grafana (DashboardScene, NotebookScene, NotebookLayoutManager). No hits in grafana/logs-drilldown, metrics-drilldown, traces-drilldown or profiles-drilldown. Not searched because the search API rate-limited: Application Observability, Kubernetes Monitoring, Knowledge Graph and other app plugins. Code search can miss files, so treat this as a lead and not proof that Drilldown apps report scenes.

Assistant: the plugin emits data-request with a constant source of grafana-assistant-app for agent-run queries (read from plugin source, which is not public; no excerpt here).

This repo today: collector/sources/usage_insights.py groups data-request by datasourceType, datasourceUid, dashboardUid/panelId only. source and panelPluginId are never grouped on. The module docstring records source as present on live events.

Wave 1: ca3b4f2 integrated; CI 35908750277 passed pytest and tofu, and robknight dev T2/T3/T1/T4 on image digest sha256:1ebdbea7 all exited 0. Live dev S3 has 6 per-stack and 5 estate surface rows; write-stack Mimir has all 8 closed-enum request series, including measured zeros. AC1 remains unproven: 55 guarded usage-insights stack probes (5 dev, 50 customer) established sources, userId and stale dashboardUid, but no browser session was available to cause and identify a query from a known Drilldown user. Re-run that exact known-user UI query, inspect its guarded log line and update docs/traps before checking AC1. Local just check history gate fails on eight pre-wave commits; exact-SHA hosted current-file identifier and no-em gates passed.

loop3 admitted 2026-09-30 under owner goal: root owns tracker; bounded lanes own implementation/discovery, evidence pending. No acceptance claimed yet.

Loop3 full identifier/text gate nowclean d0b5bb0210870f4ce54031455397be5411e21f74/CI36737688385success. Five guarded staff probes plus genuine Logs2.6 frontend known-app control documented in docs/traps:43actual events genericapp, userId field present, stale-dashboard evidence preserved. AC1 strict known-human-user component remains unproven: control used existingAdminSA, not human adoption; task partial/parked rather than silently relabeling principal. No new enum/per-app series or source inference added.

Admitted under owner loop5 goal, 2026-10-01. Root owns tracker writes; isolated lanes own only named seams. Acceptance pending terminal evidence.

loop5 L-drill-a1 documentation candidate90fef5891dff8ef74e6d20ead5f412d731c28d90 accepted for integration, no collector classification change. Known-human guarded sample211events: app129/scenes69/dashboard6/explore7; userId field present, no identities retained; zero205non-dashboard events with nonempty dashboardUid. Temporal bursts consistent with prescribed order, but no content/plugin/URL correlation and mixed early events: AC1 stays open with exact request-to-app attribution gap. Root verified patch hash and allowlist. Gate1659pass/2existing skips/7971subtests, CRdocs exemption. Implementation1, review-repair0, infrastructure retry1 (missing venv corrected just setup); a2 unused. Landing held L-0055 main mutex.

loop5 docs candidate landed35a273338b03a0a041301d8f2d04910ea97ffff8, exactCI36832540029 success. Documentation outcome accepted; strictAC1 remains open because exact known-human frontend requests cannot be correlated independently to A/B/C from available event fields. Resume from correlated frontend evidence, not another identical datasource inference. Implementation1/review-repair0/infra1 thisloop, docsCRexemption.

Loop24 owner-selected documentation scope replaces strict original AC1 correlation completion with truthful supported/unavailable discovery disposition. Original known-app exact human correlation remains unproven; no per-app discriminator or human adoption is accepted. Existing product AC2-7 and earlier accepted implementation unchanged. Final9line docs patch reviewed independently PASS; root corrected future date before final narrow gate, clean/exit0. Documentation-only candidate1 plus trivial date correction; no new product implementation. Published documentation SHA1addf94c54aea8e6d1b80cfc992289a302a0a55a composed gate clean/exit0. Exact CI pending, Done held. Evidence /Users/rob/repos/grafana-cloud-org-insights/codex/loop24-evidence/pillarj-docs/review.md and /Users/rob/repos/grafana-cloud-org-insights/codex/loop24-evidence/pillarj-docs/final.patch.

loop24: one docs candidate plus trivial date correction, zero product implementation attempts; Done under frozen owner documentation disposition after accepted/reviewed land, composed clean and exact CI success. Earlier accepted work/attempt history preserved; strict correlation gap remains documented.

Final Summary

Closed the explicit loop24 discovery-documentation outcome. Existing step-0 sources, generic app/scenes attribution, userId presence and bounded stale-UID findings are documented; strict known-human request-to-app correlation remains unproven and is not represented as success. Independent final-patch review PASS; narrow lane and integrated gates clean; exact published1addf94c54aea8e6d1b80cfc992289a302a0a55a CI38317158771 required leaves/ci-success allsuccess. No product behavior or per-app discriminator changed.