k3d image import reports success while the nodes loaded nothing

Description

The nightly signal-fidelity-k3d matrix has been red on three of the last five runs, losing a different permutation each time. Run 34101218418 lost alloy-otlp-podlogs; run 34020767181 lost otel-collector-prom and otel-receivers. The retained diagnostic shows the cause is not the permutation under test: k3d writes the image tarball to the per-cluster image volume, both nodes fail to read it back with “ctr: open /k3d/images/-images-.tar: no such file or directory”, and k3d then prints “Successfully imported image(s)” and exits 0. e2e/lab/permutation.sh trusts that exit code, so the receiver Deployment is admitted with an image that is never pulled, sits in ErrImageNeverPull, and the permutation dies in deploy-receiver having observed nothing. A harness failure is being spent as a permutation outcome, which is the confusion the existing wait_for_apiserver comment in that file exists to prevent. The race is timing-dependent, so a non-zero exit code cannot be relied on and residency must be proven directly.

Acceptance Criteria

• #1 The import phase proves the receiver image is resident in every cluster node containerd namespace after import, rather than trusting the k3d exit code
• #2 A failed or partial import is retried once, and a still-absent image fails the import-images phase with a message naming the node and the image reference
• #3 A retained diagnostic records the per-node residency check result on both the passing and the failing path
• #4 The skcapture lab runner applies the same residency proof, because it imports through the same unreliable path

Definition of Done

• #1 just check (fmt-check, lint, gen-check, env-check, docs-check, test, race, hygiene, ui-check, compose-check, helm-test, lab-check, signal-fidelity)
• #2 just gen (only if a blueprint field, construct/workload config struct, or a skill under plugins/synthkit/skills/ changed)
• #3 just dump — inventory diffed against signals/

Implementation Plan

Prove exact image residency in every node containerd namespace after import in both runners, test the false-success case before the fix, retry once on missing residency, retain per-node diagnostics, validate statically, then root reviews and integrates.

Implementation Notes

Implemented direct per-node containerd k8s.io residency proof in both runners, one bounded import retry, and retained diagnostics on pass and failure. Simulations cover immediate pass, pass after two imports, persistent miss after two imports naming both nodes/reference, and untagged-reference normalization. Retained evidence: codex/scratch/wave-2026-09-11/residency/. bash -n, shellcheck and static lab-check passed. L1/L2 CodeRabbit completed; the sole minor normalization finding was reproduced and fixed in both scripts. No live lab or live containerd proof was run.

Final integrated just check passed. One safe explicit-selection dump exists; its machine-name comparison against signals is incomplete for prose/expanded families and is not claimed full catalogue conformance. No renderer or construct changed; no live k3d run or e2e was authorized for this lane. Conditional generation DoD is not applicable to these scripts. The unchecked dump DoD is explicitly unproven, not a hidden pass.

New local live proof: five captured permutations, partial0 empty0 failed0 in the successful matrix; both nodes per permutation passed residency on the first import. Executed at base7cbafe0, which includes d7e0281, plus isolated harness patch cda6bda011c23aa52c9dbab132894cac3c46dd01e5d8b6ccb6f62e745be12ecb. The first attempt failed in harness setup and remains separate. All ten created clusters across both attempts were deleted; none retained. Final harness corrections integrated in4dbe491. This is local execution proof, not a green nightly claim.

First post-fix scheduled CI proof: run 34203580977 at 0240066787a3b0d8777d4f4246f03785b5bd7825 completed successfully. All five permutations captured, with no partial, empty, failed or skipped result. Retained per-permutation residency artifacts show both nodes PASS on attempt 1 and retry not needed for all five. Every permutation reports teardown confirmed. No nightly was manually triggered.

First scheduled post-fix nightly 34203580977 at 0240066787a3b0d8777d4f4246f03785b5bd7825 succeeded: five captures, ten residency passes, five teardown confirmations, zero failures/partials/skips/retries. No nightly triggered. This adds CI evidence to historical completed acceptance; no task source change.

Final Summary

The import decision no longer trusts k3d exit status. Static and mocked acceptance is proven; next nightly supplies live residency evidence.

Scheduled CI now proves the residency fix: run 34203580977, exact source 0240066787a3b0d8777d4f4246f03785b5bd7825, five captured permutations, ten node residency passes and five confirmed teardowns. This supersedes the earlier statement that the next nightly is still needed.

First scheduled post-fix nightly 34203580977 at 0240066787a3b0d8777d4f4246f03785b5bd7825 succeeded: five captures, ten residency passes, five teardown confirmations, zero failures/partials/skips/retries. No nightly triggered. This adds CI evidence to historical completed acceptance; no task source change.