Description
Found 2026-08-27 while checking the lab cluster. The k3d lab pins k8s-monitoring 4.4.0 and both k3d-lab.json corpus documents record that as their collector_version. The EKS lab cluster now runs 4.5.0 (read live from its release-info ConfigMap).
So the corpus is evidence for a chart version the reference estate no longer runs. That is not a crisis — chart minor versions rarely move metric names — but it is exactly the kind of quiet drift the corpus exists to prevent, and it gets worse every release. It also means the fidelity gate is comparing synthkit against a snapshot that no longer matches the deployment synthkit claims to model.
Two things this needs beyond a version bump. First, the diff between 4.4.0 and 4.5.0 in the captured shape is itself a finding worth recording — if nothing moved, that is a useful negative result and a datapoint on how often the corpus really needs refreshing. If something did move, it is a real drift the audit would otherwise have missed entirely. Second, whatever makes the version visible when it drifts: today nothing compares the pinned lab version against the estate, so this was found by a human looking, which does not scale.
Sequence this alongside the permutation work rather than before it — the lab is being reworked for SKT-0013.01 anyway, and bumping the chart twice is wasted capture.
Acceptance Criteria
- #1 The k3d lab runs the chart version the reference estate runs, and the corpus records it
- #2 The captured shape difference between the old and new chart version is recorded, including a negative result if nothing moved
- #3 Drift detection is not implementable until a shipping series carries the observed chart version; SK-107 records the gap; the k3d lab remains on captured current version 4.5.0
Definition of Done
- #1 make gate (build vet test race rw-proto-check spdx-check forbidden-words)
- #2 make blueprint-schema (only if a blueprint field or construct/workload config struct changed)
- #3 DRY_RUN=true go run ./cmd/synthkit -once -dump — inventory diffed against signals/
Implementation Plan
Lane E order 1: refresh the disposable k3d lab from chart 4.4.0 to 4.5.0, capture the new shape, and record the measured diff including an unchanged result. Add a non-human drift detector without weakening the lab check.
2026-09-03 wave: After the Operator discriminator work, add an authorised automated estate-versus-lab chart-version drift check. Preserve the recorded empty 4.4.0-to-4.5.0 shape diff and do not recapture it; park on the exact missing read source if infrastructure change would be required.
2026-09-04 ordered after SKT-0013.02: preserve the recorded empty 4.4.0 to 4.5.0 shape diff, then add automated estate-versus-lab chart-version drift detection against the observed live metric using runtime-resolved read identity and a freshness contract.
2026-09-05 wave plan: Lane C preserves the recorded empty 4.4.0-to-4.5.0 shape diff and adds read-only live chart-version drift detection using a shipping series, runtime-resolved query identity, and an explicit maximum age; no k3d and no mutating corpus read.
2026-09-06 execution: preserve the empty 4.4.0-to-4.5.0 shape diff, replace the unreachable drift-detector criterion with the recorded not-implementable-until-observed-version-source boundary in SK-107, and close unless the Operator capture incidentally carries a version.
Implementation Notes
2026-09-02 evidence: the disposable lab and committed corpus now use chart 4.5.0. A controlled 4.4.0 versus 4.5.0 inventory diff under identical promotion exclusions was empty: both contained 100 metric and 3 log shapes. AC3 remains open because no automatic estate-versus-lab version source or scheduled comparison was implemented. Resume by adding an authorized non-human source for the estate version and making drift fail a scheduled check. just lab-check, integrated just check, and just dump passed; generation was not applicable.
2026-09-03 evidence: the live estate version is observable as grafana_kubernetes_monitoring_build_info{version}, while the current lab check validates only local pins and the sibling Terraform value is desired configuration rather than live evidence. No authorised scheduled authenticated read or publish path exists, and no infrastructure write was made. just lab-check passed; the recorded empty 4.4.0-to-4.5.0 shape diff remains the final negative result. Resume by providing a scheduled non-human read or publisher for that observed metric with a freshness contract, then add hard match and mismatch checks.
2026-09-04 closeout: the recorded old-to-current chart shape diff remains the valid empty result and was not recaptured. No authorised non-mutating scheduled estate-version source with runtime identity and freshness exists; the confirm-gated corpus reader mutates state and Terraform is desired state only. No source change was justified. just check, just dump, and just e2e passed; chart and published-compose e2e cases were skipped because their opt-ins were absent. Resume by adding an authorised read-only observed-version source with explicit freshness and testing both exact match and mismatch.
2026-09-05 evidence: the retained 4.4.0-to-4.5.0 shape diff remains the final negative result, with 100 metric families and 3 log shapes on both sides. The fresh observed grafana_kubernetes_monitoring_build_info series exposes cluster, instance, job, k8s_cluster_name, namespace, and source, but no version label. A detector against it would invent a version source. just lab-check and focused k8scluster/archtest tests passed; no Docker, k3d, capture, source, or corpus mutation occurred. SK-107 already records the missing observed version-bearing source. Resume when a shipping series actually exposes chart version and an authorized read-only scheduled query with an explicit freshness bound exists.
2026-09-04 decision by root under relaxed overnight authority: STOP hunting the observed chart-version source. Three waves have looked; the fresh build-info series carries cluster, instance, job, cluster-name, namespace and source, and no version label. Building a detector from it would invent the version source. AC#3 is to be re-specified as not-implementable-until-a-shipping-series-carries-it (SK-107 already records the gap), the empty 4.4.0-to-4.5.0 shape diff kept as the valid recorded negative result, and the task closed. A criterion nothing can satisfy is not a boundary. If the SKT-0013.02 capture happens to observe a version-bearing series, reopen on that evidence - do not go looking.
2026-09-06 criterion correction: preserved the recorded empty 4.4.0-to-4.5.0 diff. The authorized Prometheus Operator capture exposed no chart-version-bearing series, so it supplies no new detector source. AC#3 is re-specified as the SK-107 evidence boundary rather than an impossible implementation request.
Final Summary
2026-09-02: Parked after refreshing and capturing chart 4.5.0 with a useful negative shape diff; automatic estate-version drift detection remains unimplemented.
2026-09-03: Parked at the absent automated live estate-version source. Local pin validation cannot prove or detect estate drift.
2026-09-04: Parked at the missing observed estate-version source. The negative shape diff remains recorded; AC 3 cannot close from desired state or a mutating reader.
2026-09-05: Parked at the missing observed version source. The empty historical shape diff is preserved; the live build-info series has no version label, so AC3 remains unproven.
2026-09-06: Closed after criterion correction. The empty 4.4.0-to-4.5.0 shape diff remains the valid negative result; automatic drift detection stays unimplementable until a shipping series exposes the observed chart version.