Description
Mission Control is the control plane for NVL72 racks and ships its own kube-prometheus-stack, Alertmanager, Loki and Grafana, which operators leave unmodified. Designs that also want the data in Grafana Cloud collect it twice: poll the Mission Control API, scrape the in-cluster exporters for the autonomous recovery services with an Alloy deployed into the admin and user clusters, collect the same logs in parallel, and forward Mission Control Alertmanager alerts by webhook. Recovery telemetry covers hardware-side workflows (firmware update state, validation, diagnostics, component health actions) and, where Autonomous Job Recovery is deployed, failed-job detection, eligibility, requeue or restart, checkpoint recovery, resource isolation and outcome. Job IDs belong in logs, never metric labels.
Acceptance Criteria
- #1 The real Mission Control surfaces (API resources, recovery-service exporters, log sources) are identified per release with provenance; anything undocumented is PENDING in cantfind.md
- #2 Recovery events are emitted as logs, and as metrics where a real exporter exists, carrying the affected node, GPU, tray, service or job, the attempted action, the outcome and whether resources were drained
- #3 Forwarded alerts are emitted as log events using the real Alertmanager webhook payload schema
- #4 A scenario ties a GPU fault to a recovery workflow and a workload requeue across the DCGM, recovery and scheduler outputs on the same identities and time window
- #5 Every metric, label, attribute and log field is sourced from vendor documentation or a capture at a pinned version and recorded with provenance in the named signals/ area file; anything unconfirmed is a PENDING in cantfind.md, never an invented name
- #6 Rates, counts and cardinality in any shipped blueprint are plausible for a real deployment, with the real-world basis stated in a comment; nothing is raised to force emission
Definition of Done
- #1 just check (fmt-check, lint, gen-check, env-check, docs-check, test, race, hygiene, ui-check, compose-check, helm-test, lab-check, signal-fidelity)
- #2 just gen (only if a blueprint field, construct/workload config struct, or a skill under plugins/synthkit/skills/ changed)
- #3 just dump — inventory diffed against signals/
Implementation Notes
loop47: admitted R11 Mission Control vendor-contract research; implementation blocked on fixture/DCGM and parked BCM source contract
loop47: two research passes pin Mission Control 2.3.1/AHR/AJR/PRS docs and actual upstream Alertmanager v0.28.1 webhook Message schema. Recovery-service/API/exposition/log join contracts remain unavailable; PRS or training inputs are not recovery outputs. Resume with pinned AHR/AJR API/exporter/event schema and BCM dependency; never invent fields. Implementation 0; review-repairs 0; infra retries 0; not rejected.