Description
REST gateway logs are the primary AI Gateway source. Requests sent with log collection off, or dropped by gateway log limits or retention, never appear in REST. aiGatewayRequestsAdaptiveGroups counts them anyway. Emit a coverage signal comparing the GraphQL Groups request count with the REST-derived request count per gateway over windows held back past ingestion lag (loop 3 measured 139-365 s event-to-Groups; hold back at least 10 minutes). It is a completeness check only, never a second primary rate. Decided by Rob 2026-09-26 when closing CFO-0009.
Acceptance Criteria
- #1 A metric declared in internal/semconv reports GraphQL Groups minus REST request count per gateway for each lag-safe window, with a test covering equal, missing-logs and GraphQL-behind cases
- #2 Selections come from settings.availableFields; the collector respects maxDuration and notOlderThan and never double-counts a window on replay
- #3 Live: after a Camden deploy, one window’s GraphQL count, REST count and the emitted value agree on m7kni Mimir
Definition of Done
- #1 just check (fmt-check, lint, vet, test, tidy-check, build, vuln)
- #2 just ci before a change that touches the Dockerfile, goreleaser or the image (adds snapshot + image)
- #3 Every new signal or attribute name declared in internal/semconv and listed in docs/signals.md
Implementation Notes
Loop 8: PROBE-38 (read-only, [16:40Z,16:45Z)): REST logs list with start_date/end_date RFC3339 returned 200 and result_info.total_count 19 = paged rows; Groups aiGatewayRequestsAdaptiveGroups count for the gateway 19; dimensions_gateway advertised. L38 de07aa7 added collector aigateway.coverage (gauge cloudflare.ai_gateway.log_coverage.gap {request} per gateway; Groups count minus distinct REST ids for one closed five-minute window at least 10 minutes old; selection checked against availableFields; maxDuration/notOlderThan enforced; retry recomputes the same window). Root added the aiGatewayRequestsAdaptiveGroups API-contract entry (db2875b). REV-L38 round 1 FAIL (interval over 5m lost windows; jitter false scrape errors; -since/-before skipped the holdback; no dedupe test) -> review-repair round 1 b01ed38 (cadence pinned to 5m, boundary-aligned Lag, holdback refusal, dedupe test) -> REV-L38 round 2 PASS. Landed in 8a56011d3520a647458d221fc5d482715a897ba2; CI 36261997229 success incl ci-success after an infrastructure rerun (a fleet Actions allowlist change briefly blocked nested actions). Off by default; enable only in YAML under collectors: aigateway.coverage with enabled, interval, initial_lookback and max_window (the env form fails for dotted names, CFO-0039). AC3 open: live proof needs the collector enabled on camden, which needs a config.yaml edit this loop may not make.
Loop 8 run-end park: AC1-2 done and released in v0.8.0 (deployed to camden 18:57:55Z, healthy); the collector is off by default and camden’s config does not enable it. AC3 resume: add collectors: aigateway.coverage (enabled true, interval 5m, initial_lookback 30m, max_window 1h) to the camden config.yaml under a goal that authorises that edit, restart, then compare one window’s Groups count, REST count and the emitted gauge on m7kni Mimir. Loop 8: implementation 1/4 (L38-a1), review-repair 1/3 (L38-rr1 after REV-L38 FAIL), infrastructure retries 1 (CodeRabbit WebSocket); grants: PROBE-38 read-only, contract entry by root; reason parked on config authority.
Loop 9: implementation attempts 1/4 carried, review-repair 1/3 carried; infrastructure events: 1 carried CodeRabbit WebSocket, 1 deploy-health watcher deadline overrun this loop. Grant: release v0.8.1 and enable coverage by four literal Camden compose environment entries after the Email Sending checkpoint receipt. DEP-1 on 0.8.1 passed compose validation and reported healthy with coverage checkpoint 21:00Z, matching image digest and no startup errors, but its receipt arrived 17m03s after watcher start, beyond the 15-minute outer deadline. Per ship contract, restored .env and compose backups and recreated 0.8.0; the full rollback watcher was healthy and matched the previous digest and file hashes. LIVE-38 did not start, so AC3 stays unchecked. Resume with a separately authorised deploy whose 10-minute health receipt finishes within the 15-minute outer deadline, then compare a positive committed Groups window, REST count and Mimir gauge exactly.
Loop 10: implementation 1/4 and review-repair 1/3 carried; infrastructure retries 2 health-watcher unobserved windows. One authorised v0.8.1 forward deploy used the four literal compose environment entries. Both sleep-guarded health watcher launches produced no receipt within 15 minutes; neither supplied the required observation count. Restored both pre-loop10 backups and recreated v0.8.0. Readback healthy, image digest and all three file hashes equal prestate. LIVE-38 was not admitted; AC3 remains open. Resume with a reliable sleep-guarded observation path and a fresh authorised deploy, then exact positive-window source/REST/Mimir comparison.
Loop 11: implementation 1/4 and review-repair 1/3 carried; infrastructure retries 0 this loop. Grant: DEP-1 v0.8.1 with four compose environment entries and revert. DEP-1 health receipt 2026-09-27T11:06:57Z to 11:16:59Z: 11 healthy observations, no gaps/errors, correct image digest, no prior checkpoint regression, clean startup. LIVE-38 positive committed window [2026-09-27T00:25:00Z,00:30:00Z): Groups 12, REST 12, Mimir cloudflare_ai_gateway_log_coverage_gap sample 0 at 11:23Z for the same gateway; historical Mimir read at 12:19Z. Source receipt observed checkpoint advance at 11:22:09Z. DoD 1-3 were satisfied by the earlier L38 landing; this loop performed no collector code or signal change.