Make priority/fast-mode value answerable at session level, not just per response

Description

Priority mode’s per-response win is measurable today, but its value across a whole wave session is not. Closing that needs the requested tier on four more signals plus one missing label on the rate-limit gauges.

What was measured (2026-09-04)

Window 2026-08-27 to 2026-08-31, the only period where both tiers ran interleaved on the same days. 28,877 priority turns vs 34,781 default, request_kind=turn.

Per turn, priority wins clearly:

The win holds into the tail on model round-trip seconds, per model:

gpt-5.6-sol    p50 4.70 -> 3.26   p90 13.19 -> 9.52   p99 43.21 -> 29.89
gpt-5.6-terra  p50 5.84 -> 3.93   p90 20.42 -> 14.42  p99 68.57 -> 44.73
gpt-5.6-luna   p50 5.40 -> 3.93   p90 20.29 -> 12.15  p99 65.24 -> 43.50

TBT matters more than TTFT. At ~330 output tokens per response the decode-rate saving is ~1.9s, roughly 4x the 0.5s TTFT saving.

At session level it mostly washes out. 319 sessions of 40+ turns reconstructed offline from Loki turn records (177 default, 142 priority):

A 1.44x per-turn model win compresses to roughly 3-10% on median session throughput. Mechanism: long sessions are entirely fan-out, so wall clock is set by the slowest lane per wave, not the mean, and the tool-pause half of the cycle does not move.

Confounds, not controlled: cohorts are not randomised, model mix differs (terra 14.9% -> 31.1%, sol 68.3% -> 52.2%), and priority sessions ran larger jobs.

Current signal coverage

Carries codexlb_service_tier_requested (deliberate, record.go:255):

• gen_ai_client_time_to_first_token_seconds
• gen_ai_client_operation_duration_seconds
• codexlb_turn_duration_seconds

Narrowed out everywhere else, so unaskable in PromQL: the engine wall / queue / TBT breakdown, codexlb_harness_unblocked_seconds, codexlb_responsesapi_excl_client_tools_seconds, tokens, tool calls, errors, transport events, and every rate-limit and credits gauge.

Loki is the only complete session-level source today. {service_name="codexlb2otel", codexlb_record_type="turn"} carries thread_id, parent_thread_id, session_id, logical_turn_seq, all four timestamps, both tier fields, is_subagent, every *_delta critical-path field, tokens and queue wait. It needs an offline pull: LogQL caps at 5000 entries per query, so the 4-day window above took 48 chunked calls.

Tempo has the session shape but not the tier. Root spans are named turn, one observed span covered 2.2 hours, gen_ai.conversation.id is present, and there are zero codexlb.* span attributes.

Separately: the rate-limit gauges carry neither tier nor a window label, so “does priority burn quota faster” cannot be answered at all. Account <redacted account> was at 98% used when this was written.

Scope

Four changes, smallest first. Each adds at most one extra label value per deployment since only one tier is requested at a time, so the cardinality cost is ~nothing outside a changeover.

1. codexlb.service_tier_requested on the turn span. This is the one that changes the workflow: session-level becomes a TraceQL query instead of a 48-call chunked pull.
2. Same attribute on codexlb_harness_unblocked_seconds and codexlb_responsesapi_excl_client_tools_seconds. Those two split waiting-on-the-model from waiting-on-the-client, which is the ratio that decides whether priority can help at all.
3. Same attribute on codexlb_tokens_total and codexlb_errors_total, for the cost and reliability side.
4. A window label on the rate-limit gauges (codexlb_rate_limit_used_percent, codexlb_rate_limit_model_used_percent, codexlb_rate_limit_reset_after_seconds) so 5h and weekly quotas are separable, and the tier attribute alongside it if the wire carries enough to attribute burn.

The existing Fast Mode tab (CXO-0010) covers requested-vs-served tier and TTFT cohorts. It is not being rebuilt here; it gains panels once the signals above exist.

Do not use served tier as evidence of anything. Per internal/turn/turn.go, every request since the 2026-08-08 cutover asked for priority and no response has ever reported it back.

Acceptance Criteria

• #1 The turn span carries codexlb.service_tier_requested; synthetic span evidence covers priority and absent-label normal turns, and any session query classifies pure-priority, pure-normal, and mixed sessions into disjoint cohorts.
• #2 codexlb_harness_unblocked_seconds preserves codexlb_critical_path_coverage and codexlb_family while adding codexlb_service_tier_requested; codexlb_responsesapi_excl_client_tools_seconds preserves codexlb_family while adding the same tier label.
• #3 codexlb_tokens_total preserves provider, operation, request model, response model, account, request kind, family, reasoning effort, thread source, API key, and gen_ai_token_type while adding codexlb_service_tier_requested; codexlb_errors_total preserves provider, operation, request model, account, error type, error code, and status while adding the tier label.
• #4 Account and per-model rate-limit gauges carry codexlb_rate_limit_window_minutes; primary and secondary windows are emitted distinctly, and requested-tier attribution is recorded as a hard negative unless the wire identifies which request consumed the shared quota.
• #5 The attribute registry maps dotted span and metric key codexlb.service_tier_requested to Prometheus label codexlb_service_tier_requested, registers the bounded window label, updates exact-set and cardinality guards, and records that neither custom key has a GenAI semantic-convention equivalent.

Definition of Done

• #1 just check passes: fmt-check, lint, build, test-short and probe-ci all clean
• #2 Focused tests cover priority, absent normal, both quota windows, and exact attribute preservation without widening unrelated instruments.

Implementation Plan

1. Pin the existing turn-span mapping with a synthetic priority-versus-absent regression. 2. Add requested tier only to the two named wait histograms, token counter, and error counter while exact-set tests preserve every existing attribute. 3. Register a bounded rate-limit window-minutes key, retain both primary and secondary per-model windows, and keep requested tier off shared quota snapshots as a measured hard negative. 4. Determine whether current Tempo can aggregate all thread traces for one session; implement the dashboard only with an authoritative cross-trace source, otherwise park that criterion on the missing session aggregate and traces-disabled deployment. 5. Run focused races, cardinality validation, generated-dashboard checks, just check, and review before terminal task state.

Implementation Notes

Implemented requested-tier propagation on the turn span and the four scoped metric families while preserving their existing selectors. The operation-duration selector already claimed requested tier in its source comment and task baseline but had omitted it in code; restored that pre-existing declared contract and pinned it in the focused test.

Added bounded caller-supplied codexlb.rate_limit.window_minutes, emitted it on account and per-model gauges, retained primary and secondary per-model windows plus both reset intervals, and kept codexlb.service_tier_requested off every shared quota snapshot because the wire does not attribute account-wide quota consumption to the current request.

AC 5 is not implementable honestly from the current trace model: trace IDs are derived from ThreadID, while one session spans a parent thread plus subagent thread traces. Current Tempo panels aggregate spans or individual traces and no authoritative cross-trace distinct-turn session aggregate is available. The frozen deployment also has traces disabled. Do not substitute deployment-level throughput for session throughput.

Verification: focused race tests passed; just check passed in 14.8s, including fmt, vet, golangci-lint, build, corpus-free race tests, dashboard checks, and the intentionally absent-corpus probe path. Per the revised run contract and user direction, the potentially multi-hour just test-corpus gate was not run for this non-decoder change.

CodeRabbit pass 1 completed with one major finding: the dashboard name checker guarded probe exclusion on counters but not latency histograms. Fixed it by covering every family-bearing latency histogram, adding a regression test for an unfiltered turn-duration query, and adding explicit non-probe selectors to the legacy latency dashboard. The post-fix just check passed in 10.3s.

CodeRabbit pass 2 completed with three major findings, all verified and fixed: per-term probe-selector validation with a mixed-expression regression; archive retention documented as independent OR rules; and Postgres documentation aligned with lazy-pool behavior for a syntactically valid but unreachable DSN. Two-pass review ceiling reached; no third pass run. Final just check passed in 5.7s. AC 5 remains unproven and is the sole park.

Final v0.4.0 deployment recheck: rate-limit window labeling is live with distinct 300-minute and 10080-minute per-model windows. Requested-tier labels were absent from the immediate response and turn-duration series, so no live priority cohort is claimed. AC 5 remains parked because sessions span multiple thread-derived traces, traces are disabled, and no authoritative cross-trace distinct-turn session aggregate exists.

2026-09-05: AC 5 (session-level throughput panel) removed on Rob’s decision. It required an authoritative cross-trace distinct-turn session aggregate; trace IDs derive from ThreadID and a fan-out session spans a parent thread plus subagent threads, so neither Tempo nor the traces-disabled deployment can provide it. Recorded here so it is not re-added as a defect.

Final Summary

Delivered requested-tier span and bounded metric attribution, distinct account and per-model quota windows, exact-set and absent-normal regressions, dashboard legend and probe-filter validation, and corrected related operator documentation. Focused race tests and final just check passed; CodeRabbit completed two passes and every major finding was fixed. Parked only the session-throughput panel because a fan-out session spans multiple thread-derived traces and neither current Tempo nor the traces-disabled deployment provides an authoritative cross-trace distinct-turn aggregate.

Delivered requested-tier span and bounded metric attribution, distinct account and per-model quota windows (300 and 10080 minutes live on m7kni), exact-set regressions, dashboard legend and probe-filter validation, and corrected operator docs. Verified with focused race tests, just check, and two CodeRabbit passes with every major finding fixed. The session-throughput panel criterion was dropped on 2026-09-05 because no authoritative cross-trace session aggregate exists.