Reality corpus format and report-only PR fidelity gate

Description

Layer 1: the always-on leg that actually protects the contract. Runs in seconds on every PR with no cluster and no credentials.

A committed corpus of real captured inventories becomes the reference that synthkit output is diffed against. The corpus is the durable product of the capture paths in the sibling tasks; this task owns its on-disk format, its provenance requirements, and the CI leg that consumes it.

Provenance is load-bearing: a k3d capture must never be able to “correct” an EKS-specific claim in signals/k8s.md, because k3s has no aws-node, no provider_id and different node labels. Every corpus entry records which substrate produced it, the chart/collector version, and the capture date.

Report-only on first landing (explicit decision): the gate prints findings and does not fail the build, until the corpus has settled. Promoting it to fail-closed on contradictions is a follow-up, not part of this task.

Acceptance Criteria

• #1 A documented on-disk corpus format stores captured inventories per signal area and per source, each stamped with substrate, collector/chart version and capture date
• #2 A corpus entry produced on one substrate is never applied as authority to a claim scoped to a different substrate
• #3 A make target and a CI job diff synthkit inventory output against the corpus and print a findings report grouped by finding class
• #4 The gate is report-only: it surfaces findings and exits successfully
• #5 The report separates contradictions from coverage gaps and names the signals/.md file each finding belongs to
• #6 Coverage gaps produce copy-pasteable cantfind.md PENDING entries rather than being dropped
• #7 The corpus refresh procedure is documented, including how a reviewer judges whether a corpus change is real drift or capture noise
• #8 Capture volume non-determinism is handled: the corpus does not churn between two captures of the same lab, and the format or the gate absorbs a varying captured set rather than treating volume as contract
• #9 inventory.Diff covers logs, traces, profiles and sigil, not metrics alone, so the gate can report drift on every signal class the schema carries
• #10 Comparison is scoped to the families the corpus actually covers, so an absent-from-corpus family does not surface as a coverage gap
• #11 The unused Finding*/Contradiction/CoverageGap alias constants in internal/inventory/diff.go are removed rather than adopted

Definition of Done

• #1 make gate (build vet test race rw-proto-check spdx-check forbidden-words)
• #2 make blueprint-schema (only if a blueprint field or construct/workload config struct changed)
• #3 DRY_RUN=true go run ./cmd/synthkit -once -dump — inventory diffed against signals/

Implementation Plan

1. Freeze and document the measured, substrate-scoped cumulative-union corpus format and commit the canonical two-run k3d corpus split by signals area.\n2. Integrate Lane A’s sole-owned all-signal Diff/scoping changes with focused tests.\n3. Integrate Lane B’s report renderer using Lane M’s complete area mapping, then wire the root-owned make target and report-only CI job.\n4. Integrate Lane D’s refresh documentation and verify links/rendering.\n5. Run CodeRabbit on code-bearing changes, integrate the read-only Lane G make gate, run the corpus diff and structural dry-run checks, then push and require exact-SHA ci including hygiene before finalizing SKT-0006.02.\n6. Only after SKT-0006.02 is integrated and green, ask the human which live stack to query; complete SKT-0006.04 with root-only read-only gcx, merge substrate-tagged k8s and cw entries, re-run review/gates, push, require exact-SHA ci including hygiene, and finalize.

2. Freeze and document the measured substrate-scoped cumulative-union format; commit the canonical two-run k3d corpus by signals area.

3. Integrate Lane A all-signal Diff/scoping changes and focused tests.

4. Integrate Lane B using Lane M mapping; wire root-owned make and report-only CI surfaces.

5. Integrate Lane D refresh documentation and verify links/rendering.

6. Run CodeRabbit, Lane G make gate, corpus diff, structural dry-run, push, exact-SHA ci including hygiene, then finalize SKT-0006.02.

7. After .02 is integrated and green, ask for the live stack and complete only SKT-0006.04 with root-only read-only gcx and the same terminal gates.

Implementation Notes

2026-08-24 wave-1 findings (root assessment of SKT-0006.01/.03 as landed; these are inputs to this task, not speculation):

1. CAPTURE VOLUME IS NON-DETERMINISTIC — the load-bearing one. The same lab captured 89 metric contracts locally and 44 on the hosted runner (evidence: .03 implementation notes; hosted artifact 9524811269). e2e/lab/run.sh waits only until its four positive-evidence assertions pass (RW1 receipt > 0, otlp_logs receipt > 0, k8s_cluster_name present, source="kubernetes"), then captures whatever has been scraped by that instant. A corpus that a PR gate diffs against cannot move 2x between refreshes. Options: settle-until-stable before capture, key the corpus on shape presence rather than on the captured set, or record volume as metadata explicitly excluded from comparison. Decide deliberately; do not inherit the current behaviour by accident.

2. inventory.Diff IS METRICS-ONLY. It satisfies SKT-0006.01 AC #5 literally, because all six finding kinds are metric-centric, but it never walks Schema.Logs, Schema.Traces, Schema.Profiles or Schema.Sigil even though the schema carries them (verified: no references to those fields in internal/inventory/diff.go). The gate cannot report log or trace drift until this is extended, and SKT-0006.05 needs log diffing to compare the two pod-log transports at all.

3. CORPUS/SYNTH SCOPE MISMATCH. The k3d corpus covers 44-89 metric contracts; a complete-catalog synth dump produces 2645 distinct metric series-name/label-key shapes. Diffed naively, the coverage-gap side is almost entirely noise and the report is unreadable. Scope comparison to families the corpus covers.

4. Minor: internal/inventory/diff.go declares eight alias constants (FindingMissingMetric, …, Contradiction, CoverageGap) with zero call sites outside their own declarations. Delete them before this task starts depending on one spelling arbitrarily.

Also inherited from wave 1: internal/construct/k8scluster/conformance.go holds NO hardcoded chart-version constant — it emits the blueprint-provided km.ChartVersion. Wave 1’s goal wrongly asserted such a constant existed; the audit finding is recorded on SKT-0006.03. Do not chase it again.

NAMING CONSTRAINT: this repository is public and make hygiene runs a forbidden-words guard. Never name a live Grafana Cloud stack, account or tenant in tracker text, code, docs or commit messages. A term committed here fails CI on every subsequent push until removed (it did, for six consecutive pushes across wave 1).

Wave-2 non-determinism decision (measured 2026-08-25): two fresh runs of the pinned k3d lab produced 103 and 39 metric contracts. Their structural metric sets shared 29 contracts and had a union of 113 (Jaccard 29/113 = 25.7%); raw log entries varied 2 versus 4 while projecting to the same structural log shape. Decision: the corpus is keyed on observed structural shape presence and refreshed by cumulative union for the same source/substrate/configuration. Capture counts are retained only as provenance metadata and are excluded from Diff. A shape absent from a later capture is capture noise, not deletion authority; removal requires separate confirmed drift evidence. This directly absorbs capture-volume variation and prevents identical lab refreshes from churning established entries.

Frozen corpus format v1alpha1 (2026-08-25):\n\nPath: reality-corpus//.json, where maps one-to-one to signals/.md and source-id is a generic producer name with no live identifiers.\n\nEnvelope:\n{\n “corpus_version”: “synthkit.telemetry.reality-corpus/v1alpha1”,\n “area”: “”,\n “source”: {\n “kind”: “k3d_lab | gcx_readback”,\n “substrate”: “k3s | eks”,\n “collector”: “<public collector/chart name>”,\n “collector_version”: “”,\n “captured_on”: “YYYY-MM-DD”\n },\n “authority”: {\n “substrates”: [“<exact substrate(s) this entry may contradict>”]\n },\n “capture_volume”: {\n “runs”: ,\n “observed_contract_counts”: []\n },\n “inventory”: <synthkit.telemetry.inventory/v1alpha1 Schema>\n}\n\nRules:\n- One document contains exactly one signals area and one producer/source. Empty signal-class arrays and receipts encode as [].\n- The source envelope applies to every inventory observation in the document. Comparison never unions documents across different substrates before Diff; every finding retains its source substrate. An entry may contradict only a synth claim evaluated for a substrate listed in authority.substrates.\n- capture_volume and inventory.receipts are provenance only and excluded from Diff.\n- Refresh is a cumulative structural union for the same path/source/configuration. Metric identity is name; log identity is transport plus structural label/metadata shape; trace identity is service; profile identity is profile_type; sigil identity is ingest_kind. Missing observations never delete existing shapes.\n- Set-valued structural fields are unioned and sorted. For attributes, keys are sticky. Values identical across the measurement runs are retained as observed evidence. A value set that varies between runs is canonicalized to values: [] with values_elided: true, making it presence-only/open-ended; once elided, it stays elided until a reviewer supplies confirmed drift evidence. Dynamic log-source exemplars are provenance-only and do not form an authoritative family identity.\n- captured_on changes only when an accepted refresh adds structural evidence or confirmed stable value evidence. Re-running the same lab with a subset produces byte-identical canonical corpus output.\n- Deletion or narrowing requires separate confirmed drift evidence; a single absent capture is never deletion authority.\n- The report compares only families present in each corpus document and remains report-only/exit 0 in this wave.

Rendering correction: the following restates the frozen format above with real line breaks and is authoritative.

Path: reality-corpus//.json. Each JSON document is one signals/.md area and one generic producer.

Required envelope fields: corpus_version=synthkit.telemetry.reality-corpus/v1alpha1; area; source.kind; source.substrate; source.collector; source.collector_version; source.captured_on (YYYY-MM-DD); authority.substrates; capture_volume.runs; capture_volume.observed_contract_counts; and inventory containing synthkit.telemetry.inventory/v1alpha1 with all signal arrays plus receipts.

Authority and comparison: source provenance applies to every observation; documents from different substrates are never unioned before Diff; a document may contradict only a claim evaluated for one of authority.substrates. capture_volume and receipts are provenance-only. Comparison is limited to families present in that document and remains report-only/exit 0.

Canonical refresh: cumulative structural union for the same path/source/configuration. Identities are metric name; log transport plus structural label/metadata shape; trace service; profile_type; and sigil ingest_kind. Missing observations never delete shapes. Set fields are sorted unions. Stable attribute values are retained; values varying across the measurement runs become values=[] and values_elided=true, sticky until confirmed drift. Dynamic log-source exemplars are provenance-only. captured_on changes only when accepted structural or confirmed stable-value evidence is added. Deletion or narrowing requires separate confirmed drift evidence.

Parked resume boundary (2026-08-25): Lane M was spawned with the required gpt-5.6-luna/medium request and fork_turns=none, but both the collaboration client and the child runtime exposed model and effort as unexposed. The lane correctly stopped before reading the catalogue. Because the run contract requires post-spawn route inspection and treats an uninspectable or substituted route as a hard stop, Lane M did not run and its mapping dependency for Lane B is unsatisfied. The same metadata gap prevents compliant preflight/audit of Lanes A, D and G. Resume when the client exposes effective post-spawn model and reasoning-effort metadata; re-dispatch Lane M from a fresh thread and do not credit the stopped thread. SKT-0006.04 remains To Do and untouched.

Root-owned progress while parked: materialized the frozen v1alpha1 on-disk contract in reality-corpus/README.md. git diff –check and make docs-check passed. Corpus entries remain intentionally uncreated because their per-area split depends on the required Lane M mapping.

Routing evidence for resumed v2 run: pre-dispatch confirmed the enabled collaboration feature and the Appendix A MAPPING route gpt-5.6-luna/medium with fork_turns=none. post-spawn route metadata unexposed by the client; this is recorded as a client limitation, not a misroute or stop, under the corrected v2 contract.

Resumed v2 implementation evidence (2026-08-25): fresh Lane M completed the read-only 27-area catalogue mapping. The mapping makes the corpus document area authoritative and identifies the capture split as core Kubernetes/cAdvisor/kubelet/node plus Kubernetes pod-log shapes to k8s, and CoreDNS shapes to k8s-addons. Generic go_, process_, scrape_, up, and fixture-specific synthkit_lab_ observations were excluded because the structural projection does not prove a unique catalogue owner.

Execution lanes completed under the requested Codex routes and fork_turns=none: Lane A (EXECUTION) extended Diff across logs, traces, profiles and sigil with signal-class-specific coverage kinds, reality-scoping, sticky elision, and alias removal; Lane B (EXECUTION) added strict corpus loading, canonical cumulative merge, scoped comparison and deterministic grouped report rendering, including rejection of trailing JSON documents; Lane D (EXECUTION) added the reviewer refresh guide. Children made no tracker, commit, push, Docker, cluster or live-service mutations.

Root corpus/integration evidence: committed candidates contain 32 k8s metrics plus two pod-log structural shapes and 25 k8s-addons/CoreDNS metrics, derived as the cumulative two-run union. Capture volume remains provenance-only at 39 and 103 contracts; deployment-specific values are presence-only with sticky values_elided. The root report command and make target use the full-catalog inventory JSON, load corpus documents independently by substrate authority, print grouped contradictions and coverage gaps with signals-area paths and copy-pasteable PENDING stubs, and return success with findings present. The target is mirrored in both CI workflows and included in the GitHub ci-success dependency set. Focused inventory and command tests, inventory race tests, vet, docs-check, workflow lint, git diff checks, and the end-to-end report target passed.

Correction to the preceding progress note: “committed candidates” means the candidate corpus documents prepared under reality-corpus for the pending root commit; no wave-2 commit or push had occurred at the time of that note.

Final local validation evidence before the first implementation push: the routed read-only Gate lane ran make gate exactly once and returned exit 0 without changing the tree. The report-only make target returned exit 0 with 214 findings grouped into four contradiction classes and six coverage-gap classes, named only signals/k8s.md and signals/k8s-addons.md, and emitted 97 copy-pasteable PENDING stubs. The full-catalog structural dump reported 2,644 distinct metric names, 187 generations, 50 workflow steps, 80 scores and 15 profile types. The exact unqualified DoD dry-run command also exited 0 but selected no blueprints in this local environment and therefore printed an empty inventory; the full-catalog target and BLUEPRINT_NAMES=* dump provide the comparison evidence. make blueprint-schema was intentionally skipped because no blueprint field or construct/workload config struct changed.

CodeRabbit review evidence: the first complete review raised three Major and two Minor issues. Fixed the Make fail-fast path, distinct structural identities for log coverage findings, and redundant report wording. Dismissed the histogram-elision suggestion because the frozen inventory Histogram has no elision field and both raw captures explicitly recorded a classic histogram with no observed bounds; inventing a field or le values would violate the seam. Dismissed cleanup of historical literal-newline tracker text because tracker sections are append-only and replacing them is forbidden. The second complete review raised two Major and one repeated Minor: fixed authority validation so authority.substrates is exactly the singleton source substrate, and documented source-id/path plus collector/version as the frozen producer-configuration identity. A new configuration-fingerprint field was not added because the v1alpha1 envelope is frozen.

Exact-SHA implementation CI evidence: GitHub ci run 32868942896 completed success for fd5aba271177160bbe87c8618880e2f86072ac19. hygiene, signal-fidelity, go, secret-scan, ui, docker, e2e and ci-success all passed. This is the secret-backed forbidden-words proof that the local gate cannot provide.

Final Summary

Landed the frozen v1alpha1 substrate-scoped cumulative-union reality corpus, canonical two-run k3d entries, all-signal Diff/scoping, deterministic grouped report with PENDING stubs, reviewer refresh guide, and mirrored report-only CI gate. Two measured lab runs produced 103 and 39 contracts with 29 shared of 113 (25.7% Jaccard); counts remain provenance-only and absence is never deletion authority. The local routed gate passed once; the report exited 0 with 214 findings and 97 PENDING stubs; the full-catalog dump showed 2,644 metric names and 15 profile types; exact-SHA CI including hygiene passed.