Description
Loop17 deployed0.16.1 with default shared0.5rps/burst1, process remained healthy with retained checkpoints. HOUR2 at15:05:21Z and15:10:21Z stopped after2reads: collector count39 versus49expected; healthchecks.events attempts/success series missing and last-success timestamp stillzero. This is observed incomplete/never-successful scrape coverage, not yet an attributed limiter defect or infrastructure failure. One unchanged49collector sustained-hour retry is running; no healthy deploy reverted. First0.16.0 hour passedall49. Preserve firstreceipt and diagnose exact missing collectors, queue/context budgets and actual safe errors before source/config changes.
Acceptance Criteria
- #1 Missing or never-successful collectors and their precise cause are attributed with sanitized timestamped evidence
- #2 Any required correction has an authentic registered-collector or public-client reproduction and candidate pass without weakening49collector coverage or timeout/window guarantees
- #3 A deployed sustained-hour proof shows attempts equal successes for affected collectors, or remaining live criteria are parked with both receipts
Definition of Done
- #1 just check (fmt-check, lint, vet, test, tidy-check, build, vuln)
- #2 just ci before a change that touches the Dockerfile, goreleaser or the image (adds snapshot + image)
- #3 Every new signal or attribute name declared in internal/semconv and listed in docs/signals.md
Implementation Plan
Read-only root diagnosis mapping against deployed limiter source and first/retry-hour receipts, safe current Mimir poll/error/age samples and sanitized host logs. No new implementation budget or configuration write granted; affected live criteria park after the one authorized retry if still failing.
Loop18: D1 source/live quota design packet, independent design review, F1 registered-collector red/green correction with security review and CodeRabbit before landing, composed gate and exact-SHA CI, release/deploy ops, all first polls then unchanged 65-minute hour and interval-age closeout.
Loop19: resume preserved packing/fair-limiter draft; owner-authorized exported-client red/green replaces registered scheduler witness. Pre-land security review and CodeRabbit; just check plus repeated race gate; deploy via green main push, confirm build SHA and every first poll, unchanged 65-minute live-hour and final interval ages.
Loop20: owner grants two additional clock-only attempts. Refuse ConfigureProcessRateLimitWithClock outside testing.Testing; preserve packing/fairness semantics. Rebase onto current main, unchanged public-client and FIFO witnesses, delta security and CodeRabbit, candidate gates, landing and exact-SHA CI, then all49 first polls followed by unchanged65minute proof and interval ages.
Implementation Notes
Loop17 D79 read-only mapping attributes hour-witness failure to incomplete completed-poll coverage, not demonstrated error or starvation. Poll counters publish only after RunOnce returns. At15:29:32Z46completion series, all49error serieszero, three names email.routing/firewall.metrics/httpreq.metrics have lastsuccesszero;1064APIrequests compatible with shared0.5rps pacing but no percollector queue/inflight proof. Both HOUR2 and its only retry failed after2reads due count39/49 then46/49, all present attempt/success counts match. Root started before direct proof of all49first cycles, an avoidable precondition miss, not new source evidence. No rate/burst/config/source change or budget reset authorized, no healthy deploy reverted. Resume with percollector execution/API wait or complete-cycle evidence; future separate version hour must verify all49first completions before timing.
Fresh loop17 drift review confirms AC3 explicit parked alternative is satisfied: remaining live-quiet criterion parked with both terminal failed receipts /Users/rob/repos/cf2otel/codex/live-loop17/hour-2.json and /Users/rob/repos/cf2otel/codex/live-loop17/hour-2-retry.json. This checks only parked-with-both-receipts, NOT sustained-hour success. Task remains Parked, AC1/2 open, no third retry/refund or source/rate change. Root prior withholding corrected; startup precondition miss retained in report.
Owner requested High-priority investigation of deployed polling performance. Fresh Grafana0.17.0 read around18:04UTC shows46/49completed collectors and zero scrape errors; email.routing/firewall.metrics/httpreq.metrics still never-success. Mean completed polls: HTTPthreats2373.98s, HTTPevents2198.50s, Logpushfailures2088.44s, firewall1720.37s, certificates1627.81s, DNS1512.22/1461.24s.1277APIrequests at shared0.5rps imply roughly42.6minutes request capacity. Sharedpacing strongly supported bottleneck, exact missingcollector queue/inflight state and starvation notproved. Most configured5mincadences therefore not sustained. Preserve frozenrate/burst until new authorized correction; nextloop should prioritize cause/reproduction, fairboundedpacing and realistic wholepoll budgets without weakening upstreamquota/window guarantees. Numeric safe receipt /Users/rob/repos/cf2otel/codex/live-loop17/poll-performance-1758.json.
Owner close out at18:10UTC. Final0.17.0startupreadback stopped before secondread, not an hour pass; healthy final deployment remains in place. High-priority performance investigation is next-loop priority. At18:04UTC46completed series, three never-success, no errors; durations19-40minutes prove configured5minute cadence not sustained. Source correction and exact in-flight/fairness attribution remain open, parked-live AC3alternative checked, no live-quiet acceptance.
Loop18 historical read at 18:20UTC resolves the three previously missing collectors: email.routing first poll43.14min, firewall.metrics48.84min, httpreq.metrics54.64min. Completion/success counters publish only after RunOnce returns; all later succeeded. Shared0.5rps pacing saturated continuously; exact live percollector request/queue breakdown unavailable. Independent design challenge rejects184 unbatched selections as irreducible quota demand: bounded envelope coalescing already authorized, request frequency distinct from node/query cost. Revised design in progress; no correction/deploy/live-hour pass yet.
Loop18: implementation attempts2; parked authority before unproved code landing. Draft in persistent polling worktree implements packing/fair dual quotas, safe validation and testclock migration; config/cfapi and migrated domain checks pass but allsix realregistered synctest cases deadlock because existing namecache mutex holds across paced REST while siblings wait. This is NOT authentic pacing red. Root inspected manualclock alternative: owned scheduler/admission injection cannot control unowned domain cacheTTL/currentmonth/snapshot/retention clocks, so full frozen clock semantics would be silently lost. Root rescue inspected only, no third change-and-verify attempt charged. No fullgate/race/CodeRabbit/security/baseline-red/candidate-green or F1 commit/push/release/deploy/hour proof. Preserve draft; resume after narrow domainclock ownership or independently validated witness amendment is authorized. AC2 stays open; existing AC3 checked only parked alternative from loop17, not real pass. Three initially missing collectors now precisely attributed as delayed firstpoll completions; percollector live request attribution remains unavailable.
Loop19: total implementation attempts4 (two historical, two resumed). Exported-client identical semantic ledger baseline552 physical requests versus candidate69 passes packing reproduction; REST FIFO/cancellation passes. Pre-land security rejects production-exported caller clock: valid capacities can be accelerated without wall-time pacing, contrary to private-only seam contract. No landing/deploy/live-hour. Final gate initially blocked by lint-lock infrastructure; unchanged-state gate retry in flight. Ceiling exhausted; resume needs owner-raised attempt ceiling to remove public clock bypass, then review/gates/deploy and live proof. AC2 remains unchecked; AC3 remains only prior parked alternative.
Loop19 final unchanged-state gate retry passed just check and go test -race -count=10 ./internal/cfapi/ ./internal/config/, exit0. Tested dirty worktree on base f1a82ed280a1dddc856e8fdca722ac204592a8cb plus candidate patch b57b8595258db4982f2fa255e151d840924c7915359f35c94546f55ed744e248. Security clock-bypass blocker remains; no acceptance, commit, push, deployed proof or composed gate. Four attempts exhausted. Gate green is not landing permission.
Loop20 adopted. Historical attempts4 remain; two owner-granted clock-only additional attempts. Current main already publishes the newer loop19 tracker records. Root owns tracker writes; no pushes during live proof.
Loop20 clock-only additional attempt1 passes at a756b7af7e6524143137731a5ceaaf1d0d2a9c52 on base0f69a0b60321b4aa5ea3233479afe67d6c30eba7. Frozen public-client ledger552 physical requests to69, identical semantic hash; FIFO A,C,newcomers; production normal executable clock rejection red/green. just check and race count10 exit0 after lint-lock infrastructure retry. Independent security accepts clock-only delta; CodeRabbit complete with two owner-rejected unchanged-path majors, no new delta blocker. No push yet; live acceptance and composed gate/CI outstanding.
Loop20: total production implementation attempts5 (historical4 plus owner-granted clock-only attempt1; second additional clock-only attempt unused). Candidate production a756b7af7e6524143137731a5ceaaf1d0d2a9c52 landed through0519970c1257738eb4f4f947a6db33b8161cd8cf; gates, security, CodeRabbit disposition and exact-SHA CI green. Deployed RC0.17.1-rc.6 fails live performance envelope: first email.routing2337.420s, firewall.metrics1748.335s, htt preq.threats2337.419s;48/49 completed after~100minutes, htt preq.metrics no completed attempt; several5minute collector success ages exceed twice interval despite zero error counters. No65minute hour started. Owner correctly challenged longpolls as unresolved issue; root prior incomplete-proof framing corrected. Goal-authorized singlecommit production rollback in progress, no code retry afterdeploy. Private receipts first.json and first-retry.json retained; original first receipt not-observed due repaired1ms helper skew, not exporter failure. AC1/2 remain open; AC3 remains only loop17 parked alternative, not real sustainedhour pass. Resume needs a separately authorised realcollector throughput/catchup diagnosis, then correction budget and new deployed proof.
Loop21 live measurement interrupted by verified unhealthy instrumentation epoch:14collectors23scrapeerrors including8repeated failures versus0baseline. Goal requires rollback inonecommit; no throughput fix. Sanitized partial cumulative request/wait/duration table /Users/rob/repos/cf2otel/codex/live-loop21/attribution.json; no all49firstpollor90minute completed, no exact epoch increase/time-share or per5minute demand claimed. AC1 remains unchecked; cause remains unattributed. Resume recommendation: separately budget authentic runtime-error reproduction and restore measurement before throughput correction.
Correction: originalversion46cbb26 has22errors(not23), newsameSHA0.18.0-rc.1 has0errors/21successes after autoRC37243958078. Singleepoch broken by automaticRCredeploy; codecausation NOT proven. Partial historicalinstant table atalarm:49rows,441cumulative requests, wait/duration sums/counts; no reliable zero baseline, exactfirstpoll shares or5minute rate. Conservative explicit goal rollback chosen; AC1 unchecked. Re-grade release/epoch fence before renewed measurement.
Partial table reduction (not AC1):49collectorrows,441cumulative original-version requests,20688.666103seconds aggregate limiterwait and109.917351seconds API duration across concurrent requestseries. Leaders:requests email.sending33,healthchecks.events29,dns.events/firewall.events28; wait dns.events842.411s,dns.metrics840.896s,httpreq.metrics840.821s; API time tunnels.status10.873s,email.sending6.015s,healthchecks.events5.629s. These totals are neither firstpolltimefractions nor exactfive-minute demand; counterzero/epoch/source fence absent. Cause of22errors across14collectors remains unattributed, sameSHA RCnewversion0errors. Proposed nextscope: freeze/identify finalRCredeploy epoch, capture authenticated sanitized error cause and startupzero metrics, then measure all49or90minutes. Recommend atmost2implementation attempts for authentic registered/runtime error and epoch prerequisite only; no throughput correction until attribution acceptance.
Loop22 L1 observation notstarted, AC1unchecked: final3e85282a exactCI37305970420success and autoRC37307606314fullcompletion12:15:56UTC; RC0.18.0-rc.6scrapes visible12:15:30(26searlier). Identity-awarehistory through12:19 found87series/0decreases, no qualifyingpostcompletionreset. Actualmetricnames/labelkeys discovered, no guessedzeros or measurement accepted. Receipt /Users/rob/repos/cf2otel/codex/live-loop22/L1-final-history-1791202745.json. Implementationattempts parent historical5unchanged; currentchildrenR1total3,M2conservative2; liveprobe consumesnone. Parkowner: authorise sameSHArestartafterfullRCcompletion or amend fence to demonstrablysettledpublication-linked epoch; rootcannotwaive it, SSH/restartnotgranted. No ongoingregression demonstrated/no rollback. Workflowdispatch follows parkedL1, noactiveobservation.
Loop23 adopted: H1 limiter hardening precedes L1 on exact green main SHA; revised publication-linked reset fence and conditional one-shot restart authorized. No tracker pushes during L1; AC1 remains open until complete attribution.
Loop23 L1: no implementation attempt (parent historical5 unchanged). H1 limiter hardening parked at lint infrastructure retry ceiling and did not land; measured current main d936bef5afbfb8fbaec49c237ac46d743a269f29. Exact CI37332966344 ci-success green, autoRC37334986467 complete15:48:55UTC, process reset bracket15:50:01.925-15:51:03.425UTC after RCstart15:42:27UTC; no further observed reset/gap or main change. Original90min epoch15:51:03.425-17:21:03.425UTC closes48/49 completed,47 firstdurations. httpreq.metrics completion absent; tunnels.status completed but firstcount2 makes firstduration unproved, not failure. Nine firstpolls exceed300s; email.routing870.307s vs2337.420s baseline, firewall.metrics795.169s vs1748.335s. httpreq.metrics GraphQL409cumulative/408observed-increase requests,5202.122/5174.378s limiterwait,117.633/117.560s APIduration; wait97.79percent of those two recorded cumulative components, not critical-path/wall-share proof. Other-class aigateway.logs REST28cumulative/26increase followed by unclassified-error request and collector successes; no exact retry/application recovery asserted. No rate_limited series observed/no rate-limit recovery claim. AC1 remains unchecked: precise slow/missing cause incomplete; need separately authorized registered/runtime inflight diagnostic evidence, not an assumed observation retry or fairqueue correction. Sanitized receipt /Users/rob/repos/cf2otel/codex/live-loop23/attribution.json with detailed history and independent partial-evidence review. No restart, rollback, push or source landing. Tracker edits remain local pending publication with next authorized code push; never pushed during L1.
Main-thread diagnosis 2026-10-05 (no new live window). Cause attributed from the loop23 epoch receipt /Users/rob/repos/cf2otel/codex/live-loop23/attribution.json, 15:51:03-17:21:03 UTC: GraphQL requests across all 49 collectors were 4328 in 5400s = 0.8015 req/s against the 0.8 req/s process bucket, i.e. saturated for the whole epoch; REST was 508 (0.094 req/s). Nine per-zone collectors (firewall.events, dns.metrics, httpreq.metrics, email.sending, firewall.metrics, httpreq.events, dns.events, logpush.failures, email.routing) issued 351-414 GraphQL requests each, 79% of the total. Source mapping: each issues one query per zone per poll, httpreq.metrics about three (breakdown batch, Groups, latency), plus settings probes per zone and dataset every 15 min. httpreq.metrics spent 97.8% of its recorded time in the limiter and never returned a poll in 90 min; the slow and missing collectors were all queue-bound on one per-token quota, not erroring. Cloudflare docs: default GraphQL quota is 300 queries per 5 min per token; with X-Rate-Limit-Type: account-based it is 300 per 5 min per zone and per account. Correction 9c3dec1: header sent, per zone/account buckets 0.9 rps, process GraphQL cap 2.5 rps, REST 1.25 rps (general limit 4 rps), resource-scoped budget pauses, per-token fallback clamp to 0.9 rps, settings cache 45-75 min. Public-client fake-upstream test TestAccountBasedGraphQLBudgetIsPerResource is red on base (8 distinct zones took 7.5s on one shared quota) and green on the candidate; scope-pause and clamp assertions mutation-checked. just check green, CodeRabbit complete with 0 findings on 14 files, independent review findings fixed (apidrift floor, false-pass assertion, per-token fallback, REST bucket default). Live proof pending after the deploy.
Live validation 2026-10-05 20:30 UTC on build 6859a75 (0.18.0-rc.10), about 100 min since the auto-RC rollout with no counter reset: 49/49 collectors completed, zero scrape errors, no API retry or envelope-error series. Over the trailing 60 min every collector attempted at least once and attempts equal successes for all 49. Mean poll duration now 1-137 s (was 1500-2400 s); GraphQL 1.13 req/s under the 2.5 cap, REST 0.11 req/s, mean limiter wait 3.2 s. Success ages: 5-min collectors at most 340 s, access.seats 622 s of 900 s, hourly collectors about 2400 s of 3600 s. Owner accepted this retrospective Prometheus read in place of a fresh live-hour.py run. DoD: just check green on the fix; just ci not required (no Dockerfile, goreleaser or image change); no new signal or attribute names.
Final Summary
Slow and missing scrape series were GraphQL quota saturation: nine per-zone collectors shared one 0.8 req/s process bucket and queued for most of each poll. 9c3dec1 sends the account-based rate-limit header, meters GraphQL per zone and account, raises the process caps and stretches the settings cache. Verified by a public-client fake-upstream test red on base and green on the fix, just check, CodeRabbit, and a live read on 0.18.0-rc.10 showing 49/49 collectors with attempts equal to successes over the last hour and poll times down from 25-40 min to under 2.5 min.