Description
On the live account cloudflare_dns_queries_total has exactly 2000 series and an otel_metric_overflow series: the OTel Go SDK default cardinality limit (2000 datapoints per instrument per collect) is hit, driven by zone x colo x query type x response code x cached x protocol. Past the cap every new attribute set folds into one overflow series, so breakdowns are silently wrong while totals survive. internal/telemetry sets no limit. In SDK v1.46 the limit is per instrument but configured globally (WithCardinalityLimit) or per instrument kind (reader CardinalityLimitSelector); there is no per-metric-name override, so per-metric control stays with cf2otel series caps.
Acceptance Criteria
- #1 A reproduction test shows the overflow before the fix and none after
- #2 The DNS metric carries no attribute combination that can exceed its cap at default config; colo stays available on dns.query log events
- #3 A documented config key sets the SDK cardinality limit (default unchanged unless justified), validated at load
- #4 Any instrument reaching its cardinality limit is visible in cf2otel self-observability (metric or warning log naming the instrument), not only in the backend
- #5 After deploy, the live DNS metric has no otel_metric_overflow series over a 21h12m clean read (owner-accepted 2026-10-04, replacing 24h)
Definition of Done
- #1 just check (fmt-check, lint, vet, test, tidy-check, build, vuln)
- #2 just ci before a change that touches the Dockerfile, goreleaser or the image (adds snapshot + image)
- #3 Every new signal or attribute name declared in internal/semconv and listed in docs/signals.md
Implementation Notes
Loop14 exact candidate 6019ce7039e38ef8b5a9eef4926140324103f1e4 passed fresh independent review after the initial default-cardinality guarantee failed review. Registered DNS with real SDK and OTLP export proves the lifetime bound, fallback total preservation and churn behavior; provider limit/default/unlimited and overflow visibility tests pass. Landing d0387dfe728e1893f7a796a2cc80298aaa566528 passed root gate and CI 36826339268. AC5 remains pending deployment plus a full 24-hour window. Dockerfile and packaging unchanged; separate packaging gate not required.
LIVE45 AC5 remains parked, not waived: the healthy DEP2 epoch was 2026-10-01 09:14:17 UTC and a partial read 2.13 hours later reported 131 DNS-family active series, no current SDK-overflow series, and absent cf2otel_metric_cardinality_overflows_total. Those were instantaneous observations, not continuous 24-hour proof. Healthy DEP3 restarted at 2026-10-01 13:37:00 UTC, so a clean post-final-deployment 24-hour window is not yet available. Required final verification: cloudflare_dns_queries_total{service_name=“cf2otel”,otel_metric_overflow=“true”} over the full post-deploy interval must have no series, and cf2otel_metric_cardinality_overflows_total{service_name=“cf2otel”} must be absent or flat. Preserve the exact final deploy epoch and query interval; absence at one instant must not be reported as a numeric zero or a full-window pass.
Loop18 interim owner-requested read over17:21UTC previousday..14:33UTC today (21h12m41s) found no DNS SDK-overflow series and absent selfoverflow counter, but not24h acceptance. Version reduction found two old0.16.1 samples at17:21:00.907, while existing DEP3 operation ended17:21:01.803; the original rounded boundary therefore straddles deployment. Root records factual goal-clock repair: safe postdeploy full24h window17:22UTC Oct3..17:22UTC Oct4, no-deploy condition retained. Original fixed window is not claimed accepted; one-minute shift will be explicitly reported for owner re-grade. No deployment or external write in loop18.
Loop18: implementation attempts0 (read-only proof). Owner steering accepts operationally okay classification from clean21h12m41s evidence rather than holding run for24h. Clock-only wait canceled; literal AC5 remains unchecked, NOT a24h pass. Root recommends formally recording a shorter owner-accepted criterion if task closure is desired, otherwise resume full24h read after safe17:22UTC boundary. DNS overflow absent and selfoverflow counter absent over interim span; no deployment in loop18. Receipt retained privately, no values/identifiers in tracker.
Loop19: owner accepts the 21h12m clean read in /Users/rob/repos/cf2otel/codex/live-loop18/dns-overflow-interim.json, replacing the prior 24h criterion. Implementation attempts0 in this loop; closed by explicit owner acceptance. Tracker change rides the polling fix landing, not a standalone push.