Record real instrument types in both corpus producers

Description

Every metric entry in both committed corpora carries instrument_types: ["unknown"], which is why SKT-0010.01 has to treat that value as absent evidence. That is the fail-safe; this is the actual fix. The instrument type is the single field most likely to catch a real defect (counter emitted as a gauge, _sum treated as a rate), so leaving it permanently unobserved would gut the audit.

Both producers can do better than the sentinel:

• k3d lab (e2e/lab/, decoded by the capture receiver). Prometheus Remote-Write v2 carries the instrument type in its symbol-table metadata, and a scraped /metrics exposition carries # TYPE. Establish which is actually available at the capture point before encoding either.
• gcx read-back (internal/inventory/gcx_readback.go, cmd/reality-corpus-gcx). A Prometheus-compatible API exposes metric metadata; whether the target stack serves it for cloud-scraper-ingested CloudWatch series is the open question. If it does not, that is a real limitation and the entry keeps the sentinel with the reason recorded, rather than a guessed type.

Where a producer genuinely cannot observe the type, say so in the corpus source block. Do not synthesise a type from the metric name suffix: inferring counter from _total would launder a guess into the ground truth, and the corpus is the only ground truth in the repository.

Acceptance Criteria

• #1 The k3d lab records a real instrument type per metric where the capture point exposes one, with the mechanism it read it from recorded
• #2 The gcx read-back records a real instrument type where the stack serves metric metadata, or records why it cannot
• #3 No instrument type is inferred from a metric name suffix or any other heuristic
• #4 Corpus entries that still carry the unknown sentinel state the reason in the corpus source block
• #5 Regenerated corpus files keep their existing provenance fields and gain a capture date for the refresh
• #6 The instrument types the refresh reveals are compared against synth, and any resulting contradictions are reported rather than absorbed

Definition of Done

• #1 make gate (build vet test race rw-proto-check spdx-check forbidden-words)
• #2 make blueprint-schema (only if a blueprint field or construct/workload config struct changed)
• #3 DRY_RUN=true go run ./cmd/synthkit -once -dump — inventory diffed against signals/

Implementation Notes

DELIVERED 2026-08-27 (lane L2). The sentinel went from universal to the exception: typed coverage 12/676 -> 513/706, and no type anywhere is derived from a metric name.

Mechanisms

gcx read-back: the stack Prometheus metadata API, read ONCE per run and looked up by EXACT metric name only, mapped through a fixed table onto the instrument vocabulary. A miss keeps the sentinel.

k3d lab: two mechanisms, neither a name. (1) prompb.WriteRequest.metadata field 3 decoded at the capture receiver, with field numbers verified against the prompb protos in the pinned Prometheus 3.12.0 module rather than from memory. (2) The reserved label names of the exposition contract — Prometheus RESERVES le for classic histogram buckets and quantile for summary quantiles, so a series carrying one IS that instrument; a native histogram sample counts equally.

ROOT DECISION on the second mechanism: KEEP it. The lane flagged it as a possible AC #3 violation and offered to delete it. Reserved label names carry normative force where a name suffix does not — the AC prohibition targets deriving type from a metric NAME (_total implies counter), which is a naming convention. Without this mechanism the k3d producer contributes zero type evidence at all.

THE HEADLINE VALIDATION RESULT

ZERO synth-vs-reality instrument contradictions. All 596 instrument_mismatch findings were the sentinel artefact. Every one of the 200 overlapping CloudWatch families AGREES with the live stack — including all 101 _sum families and every _sample_count reading as gauge. So AGENTS.md rule that a CloudWatch _sum is a per-period GAUGE and never a rate is now LIVE-VERIFIED rather than asserted.

Report effect: 1806 -> 883 findings; instrument_mismatch 596 -> 103, and all 103 survivors are reality=[unknown], the absent-evidence class.

Live-verified facts that cost real calls

• The metadata API answers only for SOME ingest paths. For the CloudWatch cloud-scraper path it answers (598 aws_* names in one probe). For the Prometheus remote-write path it answers for NOTHING — kube_node_info, kube_pod_info, kubeproxy_, awscni_, node_cpu_seconds_total, container_cpu_usage_seconds_total, alloy_build_info and up were each probed and returned empty.
• Its answer is a TIME-WINDOWED SNAPSHOT, not a catalogue. Two dumps ~20 minutes apart differed by 34 names dropped and 21 added. So a family missing from one read-back is UNRESOLVED, not untyped — which is why the cumulative union plus sentinel-drop is the right merge shape.
• grafana/k8s-monitoring 4.4.0 sends ZERO remote-write v1 metadata: 12,651 RW1 series decoded, 0 MetricMetadata records. The chart has no metadata_config override and Alloy default is send=true, so the metadata simply never leaves the collector on protocol 1.

A corpus contradiction found and corrected

storage_operation_duration_seconds carried instrument_types: [“histogram”] purely from its _sum/_count suffix, with EMPTY bucket_bounds proving no bucket series was ever captured. Now unknown. The claim happens to be true of the real kubelet, but the corpus had no evidence for it, and a ground-truth corpus must not carry a lucky guess.

Two latent bugs fixed in cmd/reality-corpus-gcx

Both would have broken any agent-run refresh, neither related to instrument types. (1) execute used CombinedOutput while gcx writes an advisory hint to STDERR when it detects an agent harness — that hint fed straight into a DisallowUnknownFields decoder. (2) gcx version emits JSON not a table under the same detection, and the parser died on it.

A receiver defect fixed

RW2 probing returned zero receipts. Cause: the receiver never sent the X-Prometheus-Remote-Write-*-Written response headers that the Remote-Write 2.0 spec makes MANDATORY, so a compliant sender reads their absence as nothing having been written. Fixed with header names taken from the spec. Re-probing RW2 needs a lab run AND a decision not taken: switching the lab pinned protocol changes what the k3d corpus is evidence of, since the chart real default is v1. The values file is restored to v1.

Docker was used for three full k3d lab runs and the corpus was regenerated. The refresh is a genuine cumulative union — 618 CW families = 588 committed + 30 newly observed, with ten firehose families absent from the new window PRESERVED.

Still open: the InstrumentTypeSource corpus field and the merge-identity problem, both tracked as SKT-0010.10 with the exact values to stamp.

2026-09-06 disposition, case (a): inspected every current corpus document carrying unknown instrument types. Original read-back and lab source blocks now contain explicit instrument_type_source reasons. The seven capture-v2 source blocks carry METADATA_ABSENT_FOR_SCRAPED_METRICS and UNTYPED_FAMILIES_ARE_A_SENDER_PROPERTY warnings, and the unknown family entries record instrument_type_source=undetermined. RKSY-0027 documents the measured ingest-path distinction. Source-block absence reasons are now present; no capture was changed.

Final Summary

Both corpus producers now read a real instrument type from evidence: typed coverage went from 12 of 676 to 513 of 706, and no type anywhere is derived from a metric name.

The headline is a validation result rather than a mechanism. ZERO synth-vs-reality instrument contradictions — all 596 instrument_mismatch findings were the sentinel artefact, and every one of the 200 overlapping CloudWatch families AGREES with the live stack, including all 101 _sum families reading as gauge. The AGENTS.md rule that a CloudWatch _sum is a per-period gauge and never a rate is now live-verified rather than asserted.

Three mechanisms, none a name. The gcx read-back uses the stack Prometheus metadata API, looked up by exact metric name. The k3d lab decodes remote-write metadata, and separately uses the reserved label names of the exposition contract — Prometheus RESERVES le and quantile, so a series carrying one IS that instrument. The root kept that second mechanism deliberately: reserved label names carry normative force where a _total suffix is only a convention, and without it the k3d producer contributes no type evidence at all.

Three facts that cost real calls: the metadata API answers for the CloudWatch scraper path but returns NOTHING for the remote-write path; its answer is a time-windowed snapshot, not a catalogue, with two reads twenty minutes apart differing by 34 names dropped and 21 added; and k8s-monitoring 4.4.0 sends ZERO remote-write v1 metadata across 12,651 decoded series.

A corpus contradiction was also found and corrected: storage_operation_duration_seconds was typed histogram purely from its suffix with empty bucket bounds. The claim is true of the real kubelet, but the corpus had no evidence for it, and a ground-truth corpus must not carry a lucky guess.

AC #4 is unchecked: it needs a CorpusSource field that did not exist and could not be added from this lane. The exact field, validation, and the four final provenance strings are recorded here and tracked as SKT-0010.10, which also carries the merge-identity data-loss trap the refresh exposed. Two latent bugs in the read-back CLI and a mandatory-response-header defect in the capture receiver were fixed along the way. make gate green.

2026-09-06 reconciliation supersedes the earlier unchecked AC4 statement and deferred ownership: source-block reasons are now present and AC4 is checked against the current corpus.