Description
When Run:ai is the scheduler of record on Kubernetes, the failed-job and slow-job journeys start from it. Run:ai exposes a Prometheus metrics path (MetricsExporter and WorkloadExporter) plus control-plane Metrics and Telemetry APIs for data not exposed as metrics. Telemetry covers scheduler and workload state, queue depth and pending workloads, project and department quota, GPU allocation, available and idle-allocated GPUs, wait time and failures. No source emits exemplars, so the workload-to-GPU join comes from DCGM pod labels combined with Run:ai and Kubernetes metadata mapping pod to workload, project and node pool. Project and workload names can be sensitive in regulated environments and are often pseudonymised before egress, so they must stay blueprint-only.
Acceptance Criteria
- #1 The construct emits Run:ai metric families with names and labels from the Run:ai documentation for a pinned version, documented in a new signals/ area file
- #2 Projects, departments, quotas and node pools are declared in the blueprint; where real Run:ai series carry workload identifiers, their cardinality is bounded by plausible concurrent workloads and documented
- #3 Workload lifecycle (pending, running, completed, failed, preempted) drives GPU allocation, and DCGM utilisation and pod labels follow that allocation on the same GPUs
- #4 A documented query walks from a failed workload to its GPUs and nodes using only emitted series
- #5 Scheduler logs and Kubernetes events (unschedulable, preemption, eviction) are emitted for the same workloads
- #6 Failure modes cover quota exhaustion with a queue backlog, a preemption storm and workload failure caused by a GPU fault
- #7 Every metric, label, attribute and log field is sourced from vendor documentation or a capture at a pinned version and recorded with provenance in the named signals/ area file; anything unconfirmed is a PENDING in cantfind.md, never an invented name
- #8 Rates, counts and cardinality in any shipped blueprint are plausible for a real deployment, with the real-world basis stated in a comment; nothing is raised to force emission
Definition of Done
- #1 just check (fmt-check, lint, gen-check, env-check, docs-check, test, race, hygiene, ui-check, compose-check, helm-test, lab-check, signal-fidelity)
- #2 just gen (only if a blueprint field, construct/workload config struct, or a skill under plugins/synthkit/skills/ changed)
- #3 just dump — inventory diffed against signals/
Implementation Notes
loop47: admitted vendor-contract research R12; implementation waits for fixture and DCGM landing
loop47: vendor-contract research confirms public versioned API identifiers, not MetricsExporter/WorkloadExporter exposition. Core names/types/labels and failed-workload GPU query are PENDING. Resume only with pinned exporter reference/source or capture proving lifecycle and workload-to-GPU series; do not infer API identifiers as Prometheus names. Implementation attempts 0; review-repairs 0; infra retries 0; no rejection.