Description
The OTLP-native metrics lane was built to carry exactly what web_service needed: one explicit-bound histogram and one UpDownCounter. Before any further catalog kind can join the lane, the mechanics have to cover the instrument shapes the real world uses.
Known shapes to establish support for, each verified against reality before encoding: monotonic Sums, non-monotonic Sums, Gauges, explicit-bound histograms, and exponential histograms. Temporality is a live question in its own right — cumulative versus delta is a real per-exporter choice, the repository contract mandates cumulative-across-ticks on the promrw path, and whichever is correct here must be evidenced rather than assumed to match. Instrumentation scope attributes are a further open item: signals/otlp-metrics.md records that Grafana Cloud does not add otel_scope_* labels, so what synthkit should put on the wire needs stating explicitly.
Encoding stays hand-rolled in internal/sink/otlp, matching how traces and metrics on that lane already work; the OTel SDK is confined to internal/selfobs by the architecture contract.
Coordinate with SKT-0006.05, which adds the OTLP logs lane to the same sink package. One owner at a time.
Acceptance Criteria
- #1 The lane encodes every instrument shape the SKT-0007.01 in-scope set requires, including exponential histograms
- #2 Temporality behaviour is evidenced against reality and documented, not assumed to match the promrw cumulative rule
- #3 What the lane does or does not put on the wire for instrumentation scope is stated explicitly in signals/otlp-metrics.md
- #4 Encoding remains hand-rolled with no OTel SDK dependency outside internal/selfobs
- #5 Existing web_service OTLP output is unchanged, proven by its live-validated shape still matching
- #6 Tests cover each instrument shape and the temporality decision
Definition of Done
- #1 make gate (build vet test race rw-proto-check spdx-check forbidden-words)
- #2 make blueprint-schema (only if a blueprint field or construct/workload config struct changed)
- #3 DRY_RUN=true go run ./cmd/synthkit -once -dump — inventory diffed against signals/
Implementation Notes
DELIVERED 2026-08-27 (lane L8). internal/sink/otlp/{metric_types.go,metrics.go,metrics_test.go}. internal/workload/webservice was NOT touched — the change is purely additive at the sink.
Instrument shapes now encoded: Gauge (metrics.go:110, start time deliberately omitted), monotonic Sum, non-monotonic Sum (:114), explicit-bound Histogram (:122), and the new ExponentialHistogram (:129 via expHistoPoints at :187). The first four already worked but had nothing pinning them; they do now. MetricExponentialHistogram is APPENDED to the enum so existing constant values do not shift — internal/inventory and the e2e receiver switch on them.
Exponential-histogram encoding notes worth keeping: OTLP carries ONE dense contiguous run of absolute counts per sign range (Offset + BucketCounts, interior empties are explicit zeroes), structurally different from the Prometheus native-histogram form in internal/sink/promrw (sparse spans, delta-encoded counts). Positive/Negative submessages emitted only when non-empty. OTLP Scale is numerically identical to the Prometheus schema (both base = 2^(2^-n)), so state.NativeSchemaSpanMetrics transfers unchanged.
TEMPORALITY (AC #2): cumulative only, and the previous code comment claiming “delta histograms are dropped at the gateway” was REMOVED rather than carried forward — nothing in this repository ever evidenced it. Three independent pieces of evidence replace it: the OTLP exporter spec default is Cumulative (OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE, read 2026-08-27, with OTEP 0131 giving the transit-loss rationale); the reference k8s-monitoring spanmetrics connector leaves aggregation_temporality unset and inherits Alloy CUMULATIVE default; and Mimir lists delta OTLP ingestion as experimental and opt-in. TemporalityDelta stays encodable but unused. The operative safety property is pinned by test: the ZERO VALUE encodes CUMULATIVE for Sum, Histogram and ExponentialHistogram alike, so a caller who never thinks about temporality cannot accidentally ship delta.
SCOPE (AC #3): exactly one ScopeMetrics per resource carrying the real instrumentation library name and version (zero Scope falls back to “synthkit”); scope ATTRIBUTES and schema_url are never emitted, because no modelled producer sets them and inventing them would fabricate telemetry. Recorded in signals/otlp-metrics.md as a new [slug: otlp-scope-on-wire] section by the wiring pass.
AC #5 (web_service unchanged) was proven properly rather than asserted: a characterisation test written BEFORE any production change, hashing the complete decoded wire content of a real k8s_monitoring-mode tick (not a hand-picked assertion subset), with attributes sorted because kvs iterates a Go map. Golden 270c094c… stable over -count=8 pre-change and identical post-change. Independently corroborated at estate level by diffing full -once -dump output against a HEAD restore: identical except the sigil summary line, which varies run-to-run by design (SKT-0004).
cantfind SK-91 filed for the unverified gateway delta-temporality behaviour (the lane proposed SK-84; renumbered by the wiring pass to avoid a collision with SKT-0007.01 allocations).
HAND-OFF ROUTED: the lane flagged a latent defect it could not fix — internal/inventory/synth.go addOTLPMetricResource has no MetricExponentialHistogram case, so such a metric falls through and is recorded as a gauge with resource attrs only, losing every datapoint attribute, while e2e/receiver/receiver.go:534 already handles it correctly as InstrumentHistogram{Native:true}. Nothing emits the shape yet so it blocks nothing today; it blocks the first construct that uses it (SKT-0007.04 and SKT-0008).
Final Summary
The OTLP metrics lane now encodes every instrument shape the in-scope catalog needs — Gauge, monotonic and non-monotonic Sum, explicit-bound Histogram, and the new ExponentialHistogram — with the new kind APPENDED to the enum so existing constant values do not shift for internal/inventory and the e2e receiver.
Temporality (AC #2) is the substantive result. The previous code comment claiming the gateway drops delta histograms was REMOVED rather than carried forward: nothing in this repository ever evidenced it. Three independent sources replace it — the OTLP exporter spec default is Cumulative with OTEP 0131 giving the transit-loss rationale; the reference k8s-monitoring spanmetrics connector leaves aggregation_temporality unset and inherits Alloy cumulative default; and Mimir lists delta OTLP ingestion as experimental and opt-in. Delta stays encodable but unused, and the operative safety property is pinned by test: the ZERO VALUE encodes cumulative for all three shapes, so a caller who never thinks about temporality cannot accidentally ship delta.
Scope (AC #3): exactly one ScopeMetrics per resource with the real instrumentation library name and version; scope ATTRIBUTES and schema_url are never emitted, because no modelled producer sets them and inventing them would fabricate telemetry. Recorded in signals/otlp-metrics.md as [slug: otlp-scope-on-wire].
AC #5 was proven rather than asserted: a characterisation test written BEFORE any production change hashes the complete decoded wire content of a real tick — not a hand-picked assertion subset — with attributes sorted because the encoder iterates a Go map. Digest identical before and after, and corroborated at estate level by diffing full -once -dump output against a HEAD restore.
Follow-up routed: internal/inventory/synth.go had no case for the new kind, so an exponential histogram fell through to the gauge branch and lost every datapoint attribute. Found independently by the pre-commit review and fixed in the same wave, with a test. cantfind SK-91 filed for the unverified gateway delta behaviour. make gate green.