Task · SKT-0087.18

Add Slurm scheduler telemetry and DCGM HPC job mapping

Description

Many AI Factories schedule on Slurm (BCM-managed, or Slurm on Kubernetes) instead of or alongside Run:ai. There the job-to-GPU join comes from dcgm-exporter’s HPC job mapping, written by Slurm prolog and epilog scripts, which replaces a label per job and has its own churn profile. Lower priority than the Kubernetes scheduler path, but needed to cover the generic AI Factory case.

Acceptance Criteria

Definition of Done

Implementation Notes

loop47: admitted R18 Slurm vendor-contract research only. Real job-ID Mimir labels remain forbidden; implementation HPC-mapping acceptance blocked on that explicit fence, never silently logs-only accepted.

loop47: Slurm research selects tagged maintained v1.8.5 exporter with documented scheduler/partition/node/queue shapes; actual DCGM hpc_job mapping clones series per job and conflicts with binding no-job-ID Mimir-label fence. AC2/AC3 remain unfulfilled, never logs-only accepted. Resume requires an operator-authorized policy decision and sourced job-log writer contract. Implementation attempts 0; review-repairs 0; infra retries 0; not rejected.

References

View the source file on GitHub