Model the Prometheus operator remote-write envelope

Description

The prometheus/remote-write-helm-operator/ permutation: kube-prometheus-stack running Prometheus Operator, remote-writing to Grafana Cloud, instead of Alloy.

It is an envelope difference, not a namespace one — and that decides the design. The metric FAMILIES are the same kube_*, container_* and node_* catalogue synthkit already models; what changes is the label envelope every series arrives in. Confirmed from the Prometheus Operator source:

• job defaults to the Kubernetes Service name (__meta_kubernetes_service_name), overridden by jobLabel when set. That is a completely different value space from the Alloy path literal integrations/kubernetes/cadvisor style, and any dashboard selecting on job is affected.
• service is relabelled unconditionally from __meta_kubernetes_service_name onto every ServiceMonitor target, regardless of targetLabels or jobLabel. It is always present.
• prometheus and prometheus_replica external labels are added by default (prometheusExternalLabelName / replicaExternalLabelName), so they appear on EVERY series. Neither exists on the Alloy path.

Verdict on the shape of the change: a discriminator on k8s_cluster, not a new construct, and not only a blueprint. A new construct would duplicate a hundred-plus families to vary three labels, which is exactly the boundary rule AGENTS.md warns against — the smallest independently-declarable boundary with a distinct identity, not a delivery-pipeline boundary. The right shape is a collector-envelope discriminator, mirroring how pod_logs_method selects the log transport on the same construct. A blueprint demonstrating it is warranted on top, not instead.

Capture before encoding. The three bullets above come from the operator source and its API reference, not from a live capture, and the interaction with kube-state-metrics and node-exporter subcharts (which ship their own ServiceMonitors) is unverified. The lab already runs ServiceMonitors and the monitoring.coreos.com CRDs, so a k3d capture of this permutation is tractable.

Watch for one trap: service is currently declared as read-path enrichment on the CloudWatch corpus, where it genuinely cannot be an operator relabel because CloudWatch series do not come from a ServiceMonitor. If this permutation is captured, service becomes a REAL collector-egress label on the k8s side. Those two facts must not be collapsed into one declaration.

Acceptance Criteria

• #1 The permutation is captured before its shape is encoded
• #2 The envelope is a discriminator on the existing k8s_cluster construct, not a new construct duplicating the family catalogue
• #3 job, service, prometheus and prometheus_replica behave as captured, including job carrying Service-name values rather than the Alloy literal style
• #4 A blueprint demonstrates the operator envelope end to end
• #5 The service label is not conflated with its read-path-enrichment declaration on the CloudWatch corpus
• #6 signals/k8s.md records both envelopes and which one a given series shape implies

Definition of Done

• #1 make gate (build vet test race rw-proto-check spdx-check forbidden-words)
• #2 make blueprint-schema (only if a blueprint field or construct/workload config struct changed)
• #3 DRY_RUN=true go run ./cmd/synthkit -once -dump — inventory diffed against signals/

Implementation Plan

Lane E order 2: after the 4.5.0 refresh, capture the Prometheus Operator remote-write permutation before encoding it. Return the existing-k8s_cluster discriminator and demonstrating-blueprint wiring as root handoffs, preserving the separate k8s collector-egress service-label fact.

2026-09-03 wave: Map only captured Operator families onto the existing k8s_cluster catalogue; add an envelope discriminator, document the distinct collector-egress labels, and add the sole demonstrating blueprint blueprints/k8s-prometheus-operator.yaml. Return root-owned schema/wiring changes as exact handoffs.

2026-09-04 ordered Lane C: retain the immutable Operator family inventory in the disposable k3d window, map only captured families to the existing k8s_cluster catalogue, add the envelope discriminator, preserve the two distinct service-label facts, document both envelopes, and add blueprints/k8s-prometheus-operator.yaml.

2026-09-05 wave plan: Lane C adds a new immutable hash-keyed Operator inventory capture, maps only captured families, implements an operator-envelope discriminator on the existing k8s_cluster construct, preserves the distinct CloudWatch read-enrichment versus k8s collector-egress service facts, documents both envelopes, and adds one demonstrating blueprint.

2026-09-06 execution: take a new immutable hash-keyed Prometheus Operator capture, review per-family envelope semantics, encode only what it proves as a discriminator on k8s_cluster, document both envelopes, and add one demonstrating blueprint.

Implementation Notes

2026-09-02 capture evidence: after the 4.5.0 refresh, the isolated Prometheus Operator permutation captured 842 metric families through remote-write v1. The predicate proved ServiceMonitor job equals Service name, service is retained, and prometheus plus prometheus_replica external labels arrive. Encoding is deliberately parked: there is no evidence-backed mapping from this self-scrape-heavy capture to the existing k8s family catalogue. Resume by reviewing that mapping, then add a discriminator on existing k8s_cluster, document both envelopes, and add the demonstrating blueprint without conflating the CloudWatch service enrichment fact. just lab-check and integrated gates passed; generation was not applicable.

2026-09-03 evidence: retained capture proves only the four-label Operator envelope for count:up0, not the captured family inventory. Repository history contains no immutable Operator inventory or reviewed mapping to existing k8s_cluster families, so adding a discriminator now would require forbidden name inference. just lab-check passed; no k3d resource was created and the recorded empty 4.4.0-to-4.5.0 diff was not recaptured. Resume by retaining or recovering the immutable Operator inventory and reviewing an exact captured-family mapping, including observed job, service, prometheus, and prometheus_replica values.

2026-09-04 closeout: a 300-second Operator observation reconfirmed that job, service, prometheus, and prometheus_replica are real collector-egress labels and that job and service are not universally identical across families. The immutable task record does not retain a reviewed per-family inventory authorizing a universal transform, so no discriminator, catalogue duplication, signals entry, or blueprint was added. The CloudWatch service fact remains separate read-path enrichment. just check, just dump, and just e2e passed; chart and published-compose e2e cases were skipped because their opt-ins were absent. Resume with an immutable reviewed per-family envelope mapping or current Operator configuration evidence that explains the differing values.

2026-09-05 evidence: an exhaustive retained-evidence search found no immutable 842-family Operator inventory or per-family catalogue mapping. The only durable Operator evidence is the count:up0 predicate proving job=lab-catalog, service=lab-catalog, and nonempty prometheus/prometheus_replica for that family. Encoding a universal envelope would therefore invent behavior. The distinct CloudWatch read-path service fact remains separate. just lab-check and focused k8scluster/archtest tests passed; no Docker, k3d, capture, source, or corpus mutation occurred. SK-106 already records this missing evidence. Resume by recovering or retaining a hash-keyed reviewed inventory with per-family job, service, prometheus, and prometheus_replica semantics.

2026-09-06 implementation: a new immutable Prometheus Operator capture observed 845 families, 272,984 RW1 samples, and 3,952 metadata records. Exactly 30 captured families authorize the existing k8s_cluster discriminator. The demonstrating blueprint loads 2 constructs and 0 workloads; job, service, prometheus, and prometheus_replica follow the captured egress envelope. CodeRabbit found that applying the envelope before allow-list filtering retained three excluded series; a failing regression test reproduced it, filtering was moved first, and the test passed.

Final Summary

2026-09-02: Parked after capture-before-encoding proved the Operator envelope; the evidence-backed catalogue mapping, k8s_cluster discriminator, documentation, and demonstrating blueprint remain.

2026-09-03: Parked without encoding. The envelope predicate is real, but the immutable family inventory and reviewed catalogue mapping required for the k8s_cluster discriminator are absent.

2026-09-04: Parked at reviewed per-family envelope semantics. The envelope remains a discriminator on k8s_cluster, but encoding it now would invent behavior beyond the durable evidence.

2026-09-05: Parked at the same reachable evidence boundary. No immutable per-family Operator inventory survives; the one-family predicate cannot authorize the k8s_cluster discriminator or demonstrating blueprint.

2026-09-06: Done. Added the capture-backed Prometheus Operator remote-write discriminator to k8s_cluster, documented its 30-family boundary separately from CloudWatch read enrichment, and added a non-agent demonstration blueprint. Full local gates passed.