k3d capture lab: real k8s-monitoring egress into a capture receiver, local one-shot plus nightly CI

Description

Layer 2a, the primary corpus producer. Stands up a throwaway k3d cluster, a workload deck shaped like the reference otel-demo deployment, and the real grafana/k8s-monitoring Helm chart pinned to the version synthkit claims to model — with the capture receiver as its only destination. The captured egress becomes corpus input for the generic k8s-monitoring and application-observability surface that the base blueprints model and that most users will exercise.

Capture must sit at the collector egress, not at the scrape targets: the ambient label set synthkit models (cluster, k8s_cluster_name, job, instance, source=“kubernetes”) is produced by the chart relabelling, so only Alloy output shows the real shape.

Reference for chart version and values: m7kni/rkps-awsinfra applications/core/k8s-monitoring.yaml (chart 4.4.0) and applications/k8s-monitoring/values.yaml. Strip cloud destinations and credentials; keep the feature toggles.

Known blockers found during scoping:

• Alloy remote-writes RW1 by default and chart 4.4.0 does not expose protobuf_message on a destination, while the existing receiver only decodes RW2 (io.prometheus.write.v2.Request). The capture receiver needs RW1 decode, or the lab needs a raw-Alloy-config escape hatch.
• The receiver has no OTLP logs decode, and the reference values set podLogsViaOpenTelemetry: true, so real pod logs arrive as OTLP logs.
• Substrate fidelity boundary: k3s has no aws-node, no provider_id, a different kube-proxy and different node labels. This lab cannot validate EKS identity; that is the sibling gcx task.

Acceptance Criteria

• #1 A single make target stands up the lab, captures, writes a corpus candidate, and tears the cluster down, on a developer machine with only Docker available
• #2 The k8s-monitoring chart version and the workload deck are pinned, and the pinned chart version is asserted against the versions hardcoded in internal/construct/k8scluster/conformance.go
• #3 The capture receiver decodes what the real chart actually sends, including Prometheus remote-write v1 and OTLP logs
• #4 Capture happens at collector egress so chart-added ambient labels are present in the captured inventory
• #5 Every captured corpus entry is tagged with the k3s substrate so it cannot be applied to an EKS-scoped claim
• #6 A nightly CI workflow runs the same target on a standard hosted runner and publishes the findings report
• #7 The nightly job does not require any Grafana Cloud credential
• #8 Teardown is reliable: a failed or interrupted run leaves no k3d cluster or container behind

Definition of Done

• #1 make gate (build vet test race rw-proto-check spdx-check forbidden-words)
• #2 make blueprint-schema (only if a blueprint field or construct/workload config struct changed)
• #3 DRY_RUN=true go run ./cmd/synthkit -once -dump — inventory diffed against signals/

Implementation Plan

Run contract and ownership

• Root: DESIGN+INTEGRATION on gpt-5.6-sol/high. Root owns Makefile, GitHub workflow, every Docker/k3d/Helm/kubectl invocation, live chart debugging, integration, tracker mutations, commits, pushes, and final evidence.
• Lane 4: EXECUTION on gpt-5.6-luna/max with fork_turns=“none” and delegation forbidden. Sole ownership is new files under e2e/lab/**. It may author orchestration and pinned manifests but must not run Docker, k3d, Helm, or kubectl.
• Lane 5: GATE on gpt-5.6-terra/low with fork_turns=“none” and delegation forbidden. It is read-only and runs make gate exactly once after integration.

Dependencies and frozen seams

• SKT-0006.01 is Done; internal/inventory.Schema is frozen with schema_version, optional provenance, metrics, logs, traces, profiles, sigil, and receipts. Provenance fields are substrate, chart_version, and captured_at. The candidate must set substrate=k3s and chart_version=4.4.0.
• e2e/receiver already decodes RW1, RW2, OTLP logs/metrics/traces, Loki, profiles, and emits the canonical schema at /__inventory.
• Chart reference is read-only /Users/rob/repos/rkps-awsinfra. The lab pins grafana/k8s-monitoring 4.4.0, retains the applicable generic reference feature toggles, strips every cloud destination and credential, and routes collector egress only to the capture receiver.
• Current inspection found no chart-version constant in internal/construct/k8scluster/conformance.go: it emits the blueprint-provided km.ChartVersion. This is recorded as an audit finding; no silent signal-contract edit is permitted.

Implementation sequence

1. Lane 4 adds e2e/lab scaffolding: pinned versions and workload images, sanitized chart values, workload deck, receiver deployment/service/TLS setup, exact-cluster cleanup trap, capture wait, provenance-stamped candidate, and a human-readable findings report.
2. Root reviews and renders Helm 4.2.4 with chart 4.4.0, then runs the lab end to end using exclusive local Docker/k3d/Helm/kubectl access. Debug only the real collector path and preserve collector-egress capture.
3. Root adds one Make target and a scheduled plus workflow_dispatch GitHub workflow that invokes the same target without Grafana credentials and uploads candidate/findings artifacts. Render/lint the workflow and run one manual workflow_dispatch.
4. Run focused checks and CodeRabbit before the code commit, dispatch the read-only Terra/low gate lane to run make gate once, then run the required structural dump comparison and final local lab evidence.
5. Commit explicit paths, push main, inspect exact-SHA CI, finalize SKT-0006.03 through Backlog CLI, commit/push terminal tracker state, then stop.

Implementation Notes

2026-08-24 integration evidence: Helm 4.2.4 rendered grafana/k8s-monitoring 4.4.0 into 2975 lines and 30 Kubernetes objects with only the three in-cluster capture endpoints. The exact make signal-fidelity-k3d target completed locally: 89 metric contracts, 2 log contracts, RW1 count 294, OTLP-log count 14, and PASS for cluster, k8s_cluster_name, job, instance, and source=kubernetes. The candidate provenance is substrate=k3s and chart_version=4.4.0. Exact-name teardown was observed after ordinary success, multiple failures, and Ctrl-C. conformance.go contains no hardcoded chart-version literal; it emits the blueprint-provided km.ChartVersion, recorded as the task-required audit finding without changing the signal contract. CodeRabbit completed with 7 findings: both Major findings and two valid Minors were fixed; three Minors were dismissed because the workflow has ample remaining timeout, the live stateful receiver was healthy under its memory-only limit and repository policy avoids CPU limits, and the real chart successfully scraped both extensionless BusyBox metrics.

2026-08-24 final gate evidence: the read-only GATE lane confirmed gpt-5.6-terra/low, ran make gate exactly once, and exited 0. Root then ran make dump exactly once; the complete-catalog structural inventory exited 0 with 2645 distinct metric series-name/label-key shapes and 15 profile types, with no error, fatal, or panic output. Per SKT-0004, assessment used the authoritative structural inventory rather than sampled exemplars or the wall-clock-dependent sigil generation/score counts. make blueprint-schema was skipped because no blueprint field or construct/workload config struct changed.

2026-08-24 hosted-run evidence: the first workflow dispatch exposed an invalid upload-artifact pin and the second exposed a non-portable rg dependency; both failed before creating a cluster or tore down safely. The fixes pinned verified actions/upload-artifact v7.0.1 at 043fb46d1a93c77aae656e7c1c64a875d1fc6a0a and changed the executable conformance audit to ubiquitous grep. actionlint, zizmor, the static lab validator, ShellCheck, and git diff checks passed; the follow-up CodeRabbit review completed with zero findings. Manual workflow_dispatch run 32739962451 at head 863dec936bf36d38472d9b466d7e4013f2a9afbb succeeded. Published artifact 9524811269 contains schema synthkit.telemetry.inventory/v1alpha1 with substrate=k3s, chart_version=4.4.0, 44 metric contracts, 2 log contracts, RW1 count 494, OTLP-log count 18, PASS for cluster/k8s_cluster_name/job/instance/source=kubernetes, and explicit successful teardown. Exact-head CI run 32739955181 passed Docker, Go, E2E, UI, and secret-scan; hygiene alone failed on the known pre-existing forbidden term in parent SKT-0006 and ci-success followed that failure. The parent task is outside this run scope.

Final Summary

Added a disposable, credential-free k3d capture lab and nightly/manual GitHub workflow for grafana/k8s-monitoring 4.4.0. Verified real collector-egress RW1 and OTLP-log decoding, k3s provenance, chart-added ambient labels, reliable teardown, local gate and structural dump, focused static checks, CodeRabbit, exact-head CI, and successful hosted workflow artifact publication in run 32739962451.