Description
Found by a 2026-09-03 emitted-vs-visualised sweep of dashboards/v2/codexlb2otel-full.json against the Go source (assessment saved at codex/assessment-2026-09-03-dashboard-coverage.md, gitignored; regenerate with the scripts described there if missing). Three panels are silently wrong or missing, and the generator cannot detect any of them.
Defects:
- Tools & Agents / “Tool calls per operation” (3 queries, p50/p95/p99) queries codexlb_tool_calls_per_operation_bucket. That name is not on the wire: the instrument is gen_ai.client.tool_calls_per_operation (internal/attr/names.go MetricToolCallsPerOperation), mangled gen_ai_client_tool_calls_per_operation_bucket. The panel is permanently empty.
- Latency & Critical Path / “gen_ai client operation duration (by status)” groups sum by (le, status). The real label is codexlb_status (attr.Status mangled). PromQL folds every series into one unlabeled group, the legend renders {{status}} as empty, and the panel never splits fast errors from slow ones, which is its stated purpose.
- codexlb.rate_limit.secondary_used_percent (MetricRateLimitUsed2) has no panel at all and is absent from generate.py ALL_METRICS/PROM_NAME and from .metrics_from_code.txt.
Why the generator did not catch them: verify() compares two hand-typed dicts in the same file against a hand-regenerated sidecar. The sidecar regen command in the generate.py comment uses [A-Za-z]+ for the constant name, so MetricRateLimitUsed2 (the only Metric* constant with a digit) is invisible even to a fresh regen. Nothing runs the regen (no just recipe, no CI step), and the committed sidecar is dated 2026-08-08 with the pre-rename tool_calls name. Coverage is “the query string contains the declared name”, so a wrong name declared in both places passes. SPAN_NAMES omits invoke_agent (the second toolOperationName value) and critical_path.client_tool_pause / critical_path.other. dashboards/scripts/check_names.py globs dashboards/*.json non-recursively, so it never inspects dashboards/v2/ at all; it does catch the identical stale name in the legacy 05-tool-usage.json (exit 1 today).
Not in scope: adding panels for signals that have none (that is the dashboard-additions task). Not in scope: the legacy 0*.json dashboards beyond leaving check_names.py green or documenting why it is red.
Acceptance Criteria
- #1 The metric sidecar is produced by a just recipe (group gen) whose extraction matches every Metric* constant including ones with digits, and just check fails when the committed sidecar is stale
- #2 generate.py verify() fails when a panel query references a metric wire name or a label that the source does not emit, not only when a declared name has no panel; check_names.py (or its replacement) covers dashboards/v2/
- #3 SPAN_NAMES enumerates every span name the otlptrace sink emits, including invoke_agent and every critical_path.* phase, and verify() fails when one has no panel
- #4 Regenerated dashboards/v2/codexlb2otel-full.json is pushed to m7kni with gcx dashboards update and the live spec matches the committed file
- #5 Contract proof for the secondary-window panel (Rob, 2026-09-05, replaces the live-data criterion): the generated panel queries the real wire name codexlb_rate_limit_secondary_used_percent, and a source-level test proves the instrument is emitted with codexlb_rate_limit_window_minutes for a secondary window; live rendering is not required because no secondary-window traffic has occurred since August
Definition of Done
- #1 just check passes: fmt-check, lint, build, test-short and probe-ci all clean
Implementation Plan
Wave 1 dashboard lane after wiring: repair real wire names and labels, make sidecar and coverage validation honest, regenerate, then root publishes and reads back.
Implementation Notes
Correction 2026-09-03 (live-verified on Mimir, not derived): the instrument declares unit ‘count’, and the translator appends the unit as a suffix, so the real wire name is gen_ai_client_tool_calls_per_operation_count_bucket / _count_sum / _count_count (256 bucket series live), NOT gen_ai_client_tool_calls_per_operation_bucket as the assessment file states. Fix the panel to the count form. Live baseline for reference: ~7,100 active series total for job=codexlb2otel, dominated by gen_ai_client_token_usage_bucket (1,140).
Final integration at 7a54c59 and 334a4db: generated coverage is 63 of 63 metrics, 9 of 9 record types, and 10 of 10 span names; final just check passed. Dashboard generation 12 is live on m7kni and its spec matches the committed resource after normalizing empty custom objects stripped by Grafana. Live PromQL returned 352 tool-call histogram bucket series and 26 completed operation-duration count series grouped by codexlb_status. The secondary rate-limit metric returned no live series, so AC 1 remains unproven for that panel and this task is parked until genuine secondary-window data is emitted.
Known follow-up in the generated resource: the dashboard description still says all 57 metrics even though the verified inventory is 63. It was left unchanged after the final green gate to avoid manufacturing post-gate source. Correct the generator wording, regenerate, and rerun dashboard-check plus just check in the next run.
Corrected the generated v2 dashboard description from 57 to the verified 63 metrics after CodeRabbit identified the stale label; regenerated output and dashboard-check passed with 63/63 metrics, 9/9 record types, 10/10 span names, and 137 panels. The task remains parked because genuine secondary-window live data is still absent.
The final full-branch review also found an empty-ID Loki lookup match and non-atomic generated artifact recipes. The lookup now requires each extracted request, response, or thread ID to be non-empty before equality with the text variable, with a generator regression. Dashboard and sidecar recipes now generate to same-directory temporary files and move only after successful non-empty output. Forced failures returned 1 and 127 respectively while both tracked artifact digests remained unchanged and no temporary files remained.
A further review found the family variable default hard-coded three known values and stale explanatory text. The default All value now uses a RE2-compatible regex that excludes only exact probe while admitting future family values; probe remains manually selectable. Representative regex cases passed. Signal docs now state the explicit non-probe user-traffic contract. The tool-call histogram mapping finding was rejected: unit count is translated into the observed gen_ai_client_tool_calls_per_operation_count_bucket series, and the live Wave 1 query returned 352 such bucket series.
Final v0.4.0 deployment recheck: dashboard generation 13 remains exact-spec identical to the committed resource. The resourceVersion-safe identical update was accepted without changing generation. Primary and per-model rate-limit panels now have live codexlb_rate_limit_window_minutes values at 300 and 10080 minutes, but codexlb_rate_limit_secondary_used_percent still returns no series. AC 1 therefore remains parked on genuine secondary-window traffic; source or synthetic evidence is not substituted for live rendering.
2026-09-05: AC 1 (live secondary-window rendering) replaced by a contract-proof criterion on Rob’s decision. Evidence: dashboards/v2/codexlb2otel-full.json panel expr codexlb_rate_limit_secondary_used_percent{job=“codexlb2otel”} (line 6996) and internal/sink/otlpmetric/sink_test.go:235 asserts attr.MetricRateLimitUsed2 is recorded with codexlb_rate_limit_window_minutes for a 10080-minute secondary window. Live m7kni still returns no series for the metric (checked 2026-09-05 17:10 UTC); that is absence of traffic, not a defect.
Final Summary
Fixed the three v2 dashboard defects, made the sidecar and coverage validation honest, regenerated and published generation 12/13 to m7kni (live spec identical to the committed resource). The secondary-window panel is proven at contract level (real wire name in the panel, sink test for the instrument) because no secondary-window traffic has existed since August; Rob accepted that criterion on 2026-09-05. Verified with just check, live PromQL for the tool-call histogram (352 bucket series) and operation-duration by codexlb_status (26 series).