Description
DECIDED 2026-08-24 (Rob): synthkit models BOTH transports. podLogsViaOpenTelemetry is now an officially supported way for Grafana Cloud to emit Kubernetes observability telemetry, so the OTel-native path is a first-class lane, not a divergence to document away. This task is the implementation; it supersedes the earlier framing of the same task as an open decision.
Today synthkit emits pod logs only as Loki-native pushes to /loki/api/v1/push (stream labels + structured metadata). The chart configuration in m7kni/rkps-awsinfra applications/k8s-monitoring/values.yaml sets podLogsViaOpenTelemetry.enabled: true, so on a real cluster the same log content leaves the collector as OTLP logs carrying resource attributes and log-record attributes, and the chart promotes only an explicit labelsToKeep allowlist to Loki labels at the destination. The two transports therefore produce genuinely different observable shapes for identical content, and a user reading a synthkit-fed dashboard must see whichever one their chart config produces.
The reference labelsToKeep allowlist is the promotion contract, and it deliberately mixes dotted OTel attribute names with flat ones: app.kubernetes.io/name, container, instance, job, level, namespace, service.name, service.namespace, deployment.environment, deployment.environment.name, k8s.namespace.name, k8s.deployment.name, k8s.statefulset.name, k8s.daemonset.name, k8s.cronjob.name, k8s.job.name, k8s.node.name. Capture the real shape before encoding it; do not infer it from this list alone.
Seam. The established precedent is core.OTLPMetricWriter / World.OTLPMetrics, the opt-in native-OTLP alternative to the promrw Metrics lane, nil unless the instance declared it. The logs lane mirrors it: a new writer interface plus a World field, wired by the runner from a declared signal class, with hand-encoded OTLP logs in internal/sink/otlp (the architecture contract forbids the OTel SDK outside internal/selfobs; traces and metrics on that lane are already hand-encoded).
Selector scope is a design fork to settle during planning, not now. The chart flag is collector-level and flips every pod log on the cluster, which argues for the selector living on the k8s-monitoring cluster declaration rather than per-workload. Confirm against the captured reality before freezing the blueprint surface.
Out of scope: clusterEvents and nodeLogs (separate chart features on their own collectors) and applicationObservability app-log transforms. Establish whether those also have OTel-native paths worth modelling and record the answer, but do not implement them here.
Acceptance Criteria
- #1 The real captured shape of pod logs is recorded in signals/logs.md for BOTH transports, with provenance, including which attributes are promoted to Loki labels on the OTel path and which remain record attributes
- #2 A native OTLP logs writer lands in internal/sink/otlp, hand-encoded with no OTel SDK dependency, following the OTLPMetricWriter precedent
- #3 core exposes the new logs lane as a World field that is nil unless the instance declared it, and the runner wires it from a declared signal class
- #4 The transport selector is an explicit declared blueprint or config surface, never implicit behaviour, and its scope (cluster-level vs per-workload) matches how the real chart flag behaves
- #5 Emitted OTLP log records match the captured reality for resource attributes, record attributes, severity, and body
- #6 Both transports carry the same log content, so switching transport changes shape but never loses or invents a log line
- #7 Constructs and workloads do not import the OTel SDK, selfobs, or profiling; the architecture isolation test still passes
- #8 A blueprint exercises the OTel-native transport and appears in -once -dump output
- #9 make blueprint-schema is rerun and the generated schema docs are committed if a blueprint field was added
- #10 Whether clusterEvents and nodeLogs have OTel-native paths worth modelling is recorded, with the answer, without implementing them
- #11 The OTLP-logs decode this task depends on is consumed, not reimplemented: SKT-0006.03 owns the capture receiver, and this task compares both transports against the corpus it produces
Definition of Done
- #1 make gate (build vet test race rw-proto-check spdx-check forbidden-words)
- #2 make blueprint-schema (only if a blueprint field or construct/workload config struct changed)
- #3 DRY_RUN=true go run ./cmd/synthkit -once -dump — inventory diffed against signals/
Implementation Notes
DELIVERED 2026-08-27 (lane L7 built the lane; the wiring pass applied the runner half, the signals text, the doc comments and the schema regen).
The defect was real, and the fix reframes it
Before this, emitPodLogs dispatched pod_logs_method: opentelemetry to a LOKI PUSH with flattened k8s_pod_name/log_iostream stream labels. Both selector values shipped Loki-native; only the label spelling differed. A user selecting the OTel transport believed they were modelling podLogsViaOpenTelemetry and got Loki push with otel-ish labels.
The insight that made the fix clean: the old otel-shaped Loki push was not wrong DATA, it was the wrong OBSERVATION POINT. SK-20 live-verified k8s_pod_name/log_iostream — but those were read OUT OF LOKI, after Loki OTLP ingest promoted resource attributes and sanitised dots to underscores. The k3d corpus captured the SAME reality at collector egress, where the names are still dotted. Emitting dotted OTLP now reproduces SK-20 Loki-side observable through the real mechanism instead of hard-coding the post-promotion result onto the wrong wire.
Resource vs record split — ESTABLISHED, not guessed (AC #5)
The corpus flattens both into one list, so this was resolved from the pipeline contract instead: collector-contrib pkg/stanza/docs/operators/container.md gives the container parser output verbatim — attributes {logtag, log.iostream, log.file.path}, resource {k8s.pod.name, k8s.pod.uid, k8s.container.name, k8s.container.restart_count, k8s.namespace.name}. The two structured-metadata keys in the corpus are EXACTLY that parser record attributes. Loki OTLP ingest promotes only RESOURCE attributes, so stream_labels are a subset of resource attrs and structured metadata is record attrs plus non-promoted resource attrs — and since the corpus structured metadata is only those two, no third category exists.
Emitted: 11 resource attributes (dotted/flat mix verbatim as captured), 2 record attributes. log.file.path not emitted (parser can add it; capture has none). k8s.pod.uid not emitted (it is on the chart default-remove list, which explains its absence from the corpus). Absent dimensions omitted — the corpus second entry is exactly the unowned/unscheduled pod, so the asymmetry is real and modelled.
Two deliberate choices with evidence: severity_number/severity_text UNSET (the captured pipeline has no severity parser; Loki derives detected_level from the body; setting INFO would invent a field) — this is the one thing that could NOT be verified against the corpus because inventory.Log has no severity field, so cantfind SK-98 records it. And the instrumentation scope is left EMPTY rather than taking the “synthkit” fallback the traces/metrics sinks use, because Loki emits a scope_name structured-metadata key when the scope name is non-empty and the corpus has none.
AC #6 proved structurally, not incidentally
buildPodLogEntries is the ONLY producer of pod-log content; both projections consume its output and can only re-shape it, so neither can add or drop an entry. Pinned by a test that reduces both transports to a multiset of {namespace, pod, container, body} and asserts equality in BOTH directions. Plus a sink-level test that a malformed trace/span ID costs the correlation field but never the log line — the traces sink drops the whole span, which for logs would violate AC #6.
Selector (AC #4)
No new blueprint field. pod_logs_method already existed and is cluster-level, matching the collector-level chart flag. opentelemetry/“” to OTLP, kubernetes_api/loki to Loki, none/objects to nothing. If World.OTLPLogs is nil the construct emits NOTHING rather than silently falling back to Loki — a silent transport swap is exactly the implicit behaviour AC #4 forbids.
AC #10 — clusterEvents and nodeLogs: recorded, not implemented
Verdict: no OTel-native path worth modelling today. The chart has no clusterEventsViaOpenTelemetry or nodeLogsViaOpenTelemetry feature — pod logs got split into two named features precisely because two genuinely different pipelines exist, and these did not. Their collectors are Loki-ecosystem Alloy components. An OTLP-only operator would get them routed there by the chart generic ecosystem conversion, but that is a mechanical rewrite of the same content, not a distinct captured shape, and modelling it from inference would be invention. No capture exists. If revisited: capture first, then decide — the seam is now free, so each would be a small projection function with no sink work.
Wiring applied by the root
internal/runner/{runner.go,delivery.go,writers.go}: Sinks.OTLPLogs, the queue with shardLogResource (routes by pod identity so one pod records reach the same ordered sender — Loki rejects out-of-order lines within a stream), drainer + depth registration, and stampedOTLPLogs mirroring stampedOTLPMetrics. PodLogsMethod doc comments corrected in internal/fixture and internal/blueprint (the old wording understated the field — it selects a transport, not just a collection mechanism) and make blueprint-schema rerun.
STILL PENDING at the time of writing: the internal/inventory/synth.go projection and the cmd/synthkit sink construction, held because lane L12 owns internal/inventory concurrently. Until those land the lane is inert (Sinks.OTLPLogs stays nil), which builds and tests clean.
Defect found outside scope
The fidelity comparator can never match pod-log entries on EITHER transport: internal/inventory/synth.go keys Loki log entries on stream.Labels[“source”], and pod-log streams deliberately carry no source label (SK-20 reserves source for journal and kubernetes-events), while the corpus records source: k8s_pod_logs. Fixing it means classifying a log entry by SHAPE rather than by an absent label — comparator semantics, SKT-0010.01 territory. Predates this task and affects the Loki transport equally.
Final Summary
Kubernetes pod logs now ship genuinely OTel-native. The defect was sharper than this task described: pod_logs_method: opentelemetry already dispatched to a LOKI PUSH with otel-shaped stream labels, so both selector values were Loki-native and only the label spelling differed — a user selecting the OTel transport believed they were modelling podLogsViaOpenTelemetry and got something else.
The reframe that made the fix clean: the old otel-shaped Loki push was not wrong DATA, it was the wrong OBSERVATION POINT. SK-20 live-verified k8s_pod_name and log_iostream, but read them OUT OF LOKI, after ingest promoted resource attributes and sanitised dots to underscores. The k3d corpus caught the same reality at collector egress, where the names are still dotted. Emitting dotted OTLP now reproduces SK-20 Loki-side observable through the real mechanism instead of hard-coding the post-promotion result onto the wrong wire.
AC #5 could not be answered from the corpus, which flattens resource and record attributes into one list — so the split was established from the collector container-parser contract, whose record attributes are exactly the two structured-metadata keys the capture shows. Two things are deliberately NOT emitted, each with evidence: severity, because the captured pipeline runs no severity parser (cantfind SK-98 records that the corpus schema cannot observe this either way), and the “synthkit” scope fallback the traces and metrics lanes use, because a non-empty scope name would surface as a scope_name key Loki never showed.
AC #6 holds structurally, not incidentally: one function is the only producer of pod-log content and both transports merely re-shape its output, pinned by a test comparing content multisets in both directions. AC #4 needed no new blueprint field — pod_logs_method already existed and is cluster-level, matching the collector-level chart flag; when the lane is nil the construct emits NOTHING rather than silently falling back, since a silent transport swap is the implicit behaviour that AC forbids.
AC #10: clusterEvents and nodeLogs have no OTel-native path worth modelling — the chart has no such feature for either, their collectors are Loki-ecosystem components, and no capture exists. Recorded, not implemented.
Verified: DRY_RUN=true BLUEPRINT_NAMES=k8s-logs-events -once -dump prints [dry-run otlplogs] receipts and the otlp_logs inventory section with exactly the captured attribute set; pod logs have left the Loki lane. make gate and make blueprint-schema green.