Description
Two families synthkit emits in the wrong SHAPE — distinct from the missing families in the sibling emission task, and more serious, because a wrong shape looks like working data until someone queries it.
1. storage_operation_duration_seconds is a counter that should be a histogram. The real kubelet publishes it as a classic histogram (corpus instrument_types: ["histogram"], histogram.classic: true). synthkit publishes a standalone counter storage_operation_duration_seconds_count at internal/construct/k8scluster/kubelet.go:308 via st.Add, so _bucket and _sum do not exist and no quantile is computable against synthkit data. Labels already match (operation_name, status, volume_plugin, migrated, node). Fix is st.Observe with the real bucket bounds — which must be captured, not chosen. This is the same class as SKT-0010.03 and was outside that task remit.
2. coredns_hosts_reload_timestamp_seconds is emitted at 0 for a Corefile with no hosts block. internal/construct/coredns/coredns.go:429-432 pins it at 0 with the comment “hosts plugin not used”. Real CoreDNS registers hosts-plugin metrics only when the plugin LOADS, so the current shape matches neither a plugin-loaded CoreDNS nor a plugin-absent one — it invents a third state. SKT-0010.04 verdicted the sibling coredns_hosts_entries out of scope precisely because synthkit models a Corefile without that plugin, so the consistent fix is to stop emitting this family at all.
Acceptance Criteria
- #1 storage_operation_duration_seconds is emitted as a classic histogram matching the real kubelet
- #2 Its bucket bounds are captured from reality, never chosen
- #3 coredns_hosts_reload_timestamp_seconds is no longer emitted, consistent with modelling a Corefile that does not load the hosts plugin
- #4 signals/k8s.md and signals/k8s-addons.md record both corrections with provenance
Definition of Done
- #1 make gate (build vet test race rw-proto-check spdx-check forbidden-words)
- #2 make blueprint-schema (only if a blueprint field or construct/workload config struct changed)
- #3 DRY_RUN=true go run ./cmd/synthkit -once -dump — inventory diffed against signals/
Implementation Plan
Per the 2026-08-30 goal, Lane F handles only CoreDNS hosts-plugin absence. Root files separate le-elision and histogram-conversion follow-ups without choosing bounds.
Follow-up ownership: SKT-0041 owns the storage_operation_duration_seconds node-label correction and literal _count/base-family pairing together with the evidence-gated shape decision. SKT-0040 owns the prerequisite capture and bucket-evidence policy. This wave does not close or duplicate either storage correction.
2026-08-31 Remains dependency-parked behind SKT-0040 then SKT-0041; close only after the storage shape, node label, comparator pairing, and serialized signals provenance are verified.
Implementation Notes
PARTIALLY DELIVERED 2026-08-27 (lane L23). One half done, one half correctly STOPPED.
DONE — coredns_hosts_reload_timestamp_seconds is no longer emitted. It is now a plain coverage gap in the same category as its sibling coredns_hosts_entries, which SKT-0010.04 already verdicted out of scope. The invented third state is gone, and signals/k8s-addons.md records why.
STOPPED — storage_operation_duration_seconds. The lane refused to convert the counter to a histogram and was right to.
This task premise said the corpus proves reality publishes a classic histogram. It does not. The corpus entry carries classic: true with EMPTY bucket_bounds and NO le key in its label list at all, and that classic flag is inferred from the _bucket|_count|_sum NAME SUFFIX at e2e/receiver/receiver.go:234 rather than from an observed bucket series. So the evidence supports only “reality publishes a _count”, not “reality publishes a histogram”. Converting st.Add to st.Observe would emit _bucket{le=…} that the capture does not carry — trading a shape contradiction for an le one, and inventing a bucket set to do it.
The wiring pass checked the sibling evidence and it sharpens the question rather than settling it: every OTHER kubelet histogram in the same corpus document DOES carry real bucket_bounds and an le label — cgroup_manager, pleg_relist, pleg_relist_interval, pod_start, pod_worker. So the “k8s-monitoring allow-list drops buckets” theory is dead, since five siblings keep theirs. That asymmetry points at a CAPTURE GAP specific to this family, most plausibly that a fresh k3d cluster performs few or no volume operations during the window.
Routed to lane L21 while it has the lab running: capture whether the family publishes a _bucket series at egress, provoking a volume operation if that is cheap. Either answer unblocks this task — no buckets means the current counter shape is RIGHT, buckets means convert using the captured set.
TWO FURTHER DEFECTS FOUND IN THE SAME FAMILY, neither in this task original scope:
- This task states “labels already match”. They do not. Synth omits the node label, which the corpus family carries. It is invisible today only because a separate name mismatch stops the comparator pairing the families at all. internal/construct/k8scluster/kubelet.go passes a base label set that, unlike every sibling kubelet emitter, has no node.
- Synth emits the literal name storage_operation_duration_seconds_count, which the comparator never folds to the base name, so the family shows as a name coverage gap rather than being compared.
Left untouched so the family gets fixed once, coherently, when the capture question is answered.
BLOCKING QUESTION ANSWERED 2026-08-27 (lane L21, from the k3d matrix run). The answer is decisive and it changes what this task should do.
DO NOT CONVERT storage_operation_duration_seconds TO A HISTOGRAM. The current counter shape is RIGHT.
Two independent proofs, and the second is the one that closes it:
- grafana/k8s-monitoring 4.4.0 default-allow-lists/kubelet.yaml lists storage_operation_duration_seconds_COUNT and nothing else for that family. The five kubelet histograms that DO carry real bucket bounds in the corpus are exactly the five with a _bucket entry in that same allow-list. No bucket series can reach egress on this path at ANY window length.
- Live from a fresh capture: the family arrived with POPULATED ACTIVITY LABELS — operation_name=verify_controller_attached_volume|volume_mount, status=success, volume_plugin=kubernetes.io/configmap|host-path|projected|secret, migrated=false. So the kubelet genuinely DID perform and record storage operations, and still emitted no le. Exactly six families in that capture carry le and this is not one of them.
That also settles the follow-up I had queued: provoking a PVC would have proven nothing, because this is a collector ALLOW-LIST property, not a capture gap. Good thing the lane checked the allow-list rather than running the experiment.
So this task remaining scope is the two defects found alongside it, both real and neither yet fixed:
- Synth omits the node label, which the corpus family carries. Invisible today only because the name mismatch stops the comparator pairing the families at all.
- Synth emits the literal name storage_operation_duration_seconds_count, which the comparator never folds to the base name, so the family shows as a name coverage gap rather than being compared.
Fix those two and the family is correct. Its ACs as originally written are now wrong on the histogram half and should be read as superseded by this note.
The dependency on SKT-0013.06 stands but for a narrower reason: that task removes the suffix-inferred classic flag which made this task premise look settled. The premise was wrong; the flag is why nobody noticed.
2026-08-30 closeout: the in-scope CoreDNS half remains delivered and final gates exited 0. SKT-0040 owns observed bucket-evidence policy; SKT-0041 owns the storage node label, literal _count/base-family pairing, and evidence-gated shape decision.
2026-08-31 correction: AC1 and AC2 are superseded by observed egress and remain unchecked because the real delivery contract is not a classic histogram. The five-minute capture showed active storage operations but only literal _count, with no buckets, sum, histogram block, or le. SKT-0040 and SKT-0041 completed the evidence policy, node label, comparator pairing, corpus, and signals corrections without inventing bounds.
2026-09-06 Rule Zero disposition, case (b): AC1 and AC2 remain open deliberately. The recorded collector-egress evidence proves only the literal storage counter on this path, not buckets; the histogram premise was superseded, and SKT-0040/SKT-0041 completed the actual evidence policy and shape corrections. Task stays Done; no invented bucket set or new task.
Final Summary
Parked at AC#1, AC#2, and AC#4. Resume with SKT-0040, then SKT-0041; do not choose histogram bounds. This task closes only after the storage family corrections and both signals catalogues carry sourced provenance.
2026-08-31: Closed the two real shape defects: the CoreDNS hosts metric remains absent when the plugin is absent, and storage duration now matches observed counter egress with its node label and correct comparator pairing. The original histogram criteria were disproved, not weakened.