Description
The lab runs one configuration, serially, and prints what it found. SKT-0013 needs it to run several permutations, and the emission-validation work needs it to run alongside them, so the harness itself is now the constraint.
Restructure it so each permutation is an independent job that can run in parallel, and the run ends in ONE report covering all of them.
The report is the deliverable, not a side effect. A matrix that leaves an operator diffing several per-permutation outputs by hand has moved the work rather than done it. It needs to say, in one place: which permutations ran, which produced a capture, where two permutations DISAGREE about the same metric family, and what that disagreement means — because two permutations differing is a real property of the estate, not a defect to reconcile.
Three properties that are easy to lose and expensive to get back:
- A failed permutation must be distinguishable from one that captured nothing. Both produce an empty corpus entry, and they mean opposite things.
- Partial success must not read as success. If three of four permutations capture and the fourth times out, the combined report says so loudly rather than presenting three green rows.
- Parallel jobs must not collide on the corpus. Each permutation writes its own tagged entry; merging is a deliberate step, not a race between jobs.
Docker is an exclusive resource, so the parallelism has to be bounded and the bound stated rather than discovered by exhausting the host.
Acceptance Criteria
- #1 Permutations run as independent jobs, in parallel, with a stated concurrency bound
- #2 One combined report covers every permutation in the run
- #3 A failed permutation is distinguishable from one that captured nothing
- #4 Partial success is reported as partial, never as success
- #5 Where permutations disagree about a metric family, the report names the disagreement rather than reconciling it
- #6 Parallel jobs cannot corrupt each other corpus output
Definition of Done
- #1 make gate (build vet test race rw-proto-check spdx-check forbidden-words)
- #2 make blueprint-schema (only if a blueprint field or construct/workload config struct changed)
- #3 DRY_RUN=true go run ./cmd/synthkit -once -dump — inventory diffed against signals/
Final Summary
The lab is a matrix. run.sh stays the entrypoint and became the dispatcher; a permutation is a self-describing directory carrying its own meta, deploy script and a named-check acceptance predicate. All logic with a decision in it moved into a unit-tested Go package; bash owns only Docker, k3d and Helm orchestration.
FIVE OUTCOMES, and only one is success — this is the deliverable, not the parallelism. captured; partial (evidence arrived, predicate unmet, explicitly NOT for promotion); empty (every step completed, collector ready, receiver decoded ZERO requests — rendered as “evidence about the permutation, not a broken lab”); failed (the harness could not complete, rendered as making NO claim about what the permutation emits); skipped (still a row, so the matrix always shows the whole estate).
Three mechanisms make that robust rather than aspirational: the worker writes its result from its EXIT trap so every path including signals classifies itself; the dispatcher PRE-WRITES a failed placeholder so a job dying before its own teardown leaves a row rather than vanishing (a real hole — one attempt reported “0 of 0 selected permutations” before this); and an unreachable receiver is failed, never empty, because the harness cannot claim to have observed nothing.
CONCURRENCY BOUND 2, hard cap 4, measured not guessed: one permutation is a 2-node k3d cluster plus a full Alloy collector set at about 1.3 GiB resident and roughly a core. Memory is not the binding reason — the capture window is a fixed wall-clock deadline, so a CPU-starved job would report partial for a HARNESS reason, which is exactly the confusion this task exists to prevent.
A subtle one worth keeping: every permutation now observes the WHOLE window even after acceptance passes. Without it, capture depth was a function of how fast a permutation satisfied itself, and one run produced 76 bogus family_scope disagreements that were purely one job looking for 79s and another for 383s. With a common dwell they collapsed to zero, leaving one real disagreement. The report states the common dwell and flags the comparison as unconfirmed if windows differ.
Corpus safety: no worker or dispatcher writes reality-corpus at all, enforced by a static check. Each job owns a fixed cluster name, a distinct port and its own output dir. Promotion is a separate deliberate step, and nothing is promoted — the permutation-tagged entries cannot be until the corpus schema carries a permutation field.
SKT-0013.01 was NOT absorbed: the OTel-native permutation is expressed and renders, but is declared unproven, is not in the default selection, and the report says “Do not read an absent family here as evidence.”