Description
GPU telemetry is the centre of every AI Factory journey: utilisation, memory, temperature, power, clocks, throttle reasons, ECC, row remapping, Xid and NVLink errors, and DCGM health. dcgm-exporter runs per GPU node and is scraped either by a host Alloy or through the k8s-monitoring DCGM integration. Cardinality facts established while sizing real estates: the stock default-counters.csv at dcgm-exporter 4.6.x to 4.8.x has 26 rows, one of which (DCGM_FI_DRIVER_VERSION) is a label-type row, giving about 25 series per GPU; adding the counters that GPU fault alerting needs (ECC, throttle violation, clock events, Xid, health) takes it to about 49. DCGM_EXP_GPU_HEALTH_STATUS emits one series per health watch (about 11 per GPU); DCGM_EXP_CLOCK_EVENTS_COUNT emits one series per GPU and reason; DCGM_EXP_P2P_STATUS is N x (N-1) per node and belongs off by default. Kubernetes mode adds pod, namespace and container labels only while a pod holds the GPU, which is the standard job-to-GPU join when no source emits exemplars. NVL72 nodes add NVSwitch and Grace CPU entities.
Acceptance Criteria
- #1 For sourced hgx pools with h100_sxm_80gb only, emit the pinned stock counter set by default with blueprint-selectable stock, fault-detection and DCP profiles as frozen in the accepted DCGM rev3 contract; stock and alternate DCP have 24 captured families per GPU at rest, raw Xid is absent until an event, and unsupported or blank fields are omitted. The exact fault-detection selected additions have 46 captured at-rest samples per GPU with documented event-created expansion; P2P status is off by default. just dump matches the documented selected-profile inventory and arithmetic. Products without a sourced envelope emit no DCGM series.
- #2 Host mode and Kubernetes mode are both supported for sourced H100 SXM HGX pools; in Kubernetes mode pod, namespace and container labels exist only while a workload pod is bound to the GPU and join to actual kube-state-metrics pod series and the GPU fixture, with released attribution removed.
- #3 NVL72 nodes emit the NVSwitch and Grace CPU entity families the way dcgm-exporter does on that platform, or the gap is PENDING in cantfind.md; MIG is supported or explicitly deferred
- #4 For sourced H100 SXM HGX pools, failure modes cover Xid bursts, ECC error growth with row remapping, sustained thermal or power throttling, a GPU falling off the bus (its series disappear), DCGM health failure and NVLink errors, using the pinned source-supported observables and health metadata pairings; they are exercised through real fixture/state tests and listed in docs/incidents.md. An unsourced physical NVLink endpoint map is not invented or waived.
- #5 For sourced H100 SXM HGX pools, values are physically consistent: utilisation follows scheduler allocation when a scheduler construct is present; power, clocks and GPU temperature stay within the captured product/driver envelope and sourced T.Limit-derived limits; framebuffer free, used and reserved are distinct and use the sourced usable capacity rather than advertised GB; counters are cumulative with source reset semantics. Unsupported product or memory thermal limits remain nullable and fail closed where required.
- #6 signals/dcgm.md documents every field against a pinned dcgm-exporter tag and counters file
- #7 Every metric, label, attribute and log field is sourced from vendor documentation or a capture at a pinned version and recorded with provenance in the named signals/ area file; anything unconfirmed is a PENDING in cantfind.md, never an invented name
- #8 Rates, counts and cardinality in any shipped blueprint are plausible for a real deployment, with the real-world basis stated in a comment; nothing is raised to force emission
Definition of Done
- #1 just check (fmt-check, lint, gen-check, env-check, docs-check, test, race, hygiene, ui-check, compose-check, helm-test, lab-check, signal-fidelity)
- #2 just gen (only if a blueprint field, construct/workload config struct, or a skill under plugins/synthkit/skills/ changed)
- #3 just dump — inventory diffed against signals/
Implementation Plan
Implement only the sourced H100 SXM HGX dcgm exporter profile using frozen fixture identities, allocation/physics and the accepted exporter/DCGM contract. Register owned dcgm stub; validate fail-closed product/driver/capability admission, stock/DCP/fault counts, labels, omission/event semantics and each failure with real collaborators. Public loader/KSM join proof uses one owned integration test, not construct-directory imports. Generate, custom inventory/source diff, exact committed check; root owns batch review/CI/live.
2026-10-15 resumption: reconcile pure GPUConstraintPoints with exporter event and duration semantics; agree additive configuration/admission seams with root before implementation. Preserve exact stock/DCP/fault inventories and real Kubernetes join proof. No acceptance weakening.
Root scope correction: additive shared demand/fault admission is necessary engineering inside the authorized exporter task, not a separate owner approval. Proceed isolated at /Users/rob/.local/share/synthkit-worktrees/interactive-dcgm-20261015; preserve old thermal/physics and compatibility surfaces, implement new explicit shared scenarios and then the narrowed H100 consumer. Root integrates after first batch gates; no weakening of accepted fault/profile criteria.
Implementation Notes
loop47: admitted vendor-contract research R04; implementation waits for frozen fixture landing
loop47 I04 bounded feasibility commission1 stopped before edits, implementation performed0/infra0, clean d592c9a. Frozen catalogue has NO thermal envelope for any4products; advertisedGB is not usable/reserved driver framebuffer; stock clock/memory-temperature/vGPU encoding/support and failed-health GetErrorMeta pairing unsourced. Names/units and generic allocation/physics do not grant these facts. AC4/5 and default/fault eligibility cannot pass; NVSwitch/Grace PENDING and MIG deferral allowance does not waive them. Resume with pinned product/driver bounds, framebuffer/profile support and actual health pairing, root-owned sourced catalogue update, then consumer+serialized I06 KSM join proof. No unchanged retry. Evidence codex/scratch/loop47/I04/attempt1-return.md.
loop48 R0: admitted narrowed AI Factory research/design under Rob 2026-10-03 decisions; implementation dependency-gated; prior commission1 retained for syslog/DCGM.
loop48: Rob 2026-10-03 narrow-sourced-profile decision applied after accepted codex/packets-2026-10-03-loop48/contracts/dcgm-rev3.md sha256 951168f1287d26fc0afbfaa149aaeca9fb84ca2389a3f9fe75e61879b9ad3e9d. AC7/8 and DoD1-3 unchanged verbatim; broad deferred product/platform criteria preserved verbatim in a new non-admitted To Do follow-up. Source-supported H100 failure criteria retained; no induced-capture requirement invented. Prior commission1 retained; implementation2 waits for HGX frozen/landed and physical link evidence, not reissued unchanged.
loop48 implementation2 admitted after source-grounded AC amendment and published HGX fixture at89f4c81fe2368df95f695c5d0a7868d384b9235c, local integrated gate and root CodeRabbit completed. Landing CI remains root-owned pending, not a blanket barrier to independent source consumer work. Previous implementation1 no-edit feasibility charge retained; changed premise accepted DCGMrev3 plus sourced model/map, discriminator real profile inventory, physics/failure paths and actual existing KSM pod join; implementation2 is last ordinary worker attempt.
loop48 in-loop park: admitted implementation2 stopped before edits, conservatively consumed after carried implementation1; infra0. Independent read-only resolution confirmed owner/seam amendment required, not missing vendor names. Existing snapshots can reach zero T.Limit margin and declared power limit but do not establish requested/achieved clocks, engaged limiter or causal violation duration. DCGM-local hardware toggles would exceed the collection-only exception. Supported AC4/5 stay unchanged and unmet. Resume after owner-frozen additive shared active-constraint/requested-achieved-clock semantics and independent review; then commission changed-premise root3/specialist4 and real public tests. Evidence codex/scratch/loop48/I04/implementation2-boundary.txt and completed I04-resolution return at immutablebasee4920f8fa807d22dae21e8d4cc347400c95f307d. No unchanged retry or capability weakening.
2026-10-15 reconciliation: added /Users/rob/repos/synthkit/signals/dcgm.md as durable pinned source catalogue, explicitly not implementation acceptance. Read-only raw-capture comparison verified stock/DCP each 24 families and 192 samples; selected fault 36 families and 368 samples (46 per GPU), without exposing hardware identifiers. Bounded just docs-check passed. GPUConstraintPoints supplies causal flags/clocks but runtime demand projection and explicit fault-profile thermal admission remain root-owned seams; current ThermalEnvelope rejects worst-case above 87 C, so cannot exercise >89 C slowdown. Proposed additive helper leaves legacy capability unchanged. No emitter, registry, tests, gate, CodeRabbit or commit claimed complete.
Independent root challenge accepted a separate exact-driver fault admission path below 95 C while preserving legacy ThermalEnvelope; vendor reason attribution and temporal counters still need explicit source-grounded semantics. Additional shared power-demand incident extension proposed to owner via async prompt. Retained catalogue is indexed as source-only pending emission. No emitter completion or acceptance waiver.
Root froze additive shared-seam requirements at /Users/rob/.local/share/synthkit-worktrees/interactive-dcgm-20261015/codex/dcgm-owner-seam.txt for isolated implementation. Prior attempts retained; changed premise is explicit extension authority and independent thermal/temporal review. No local hardware toggles, vendor curve inference or legacy compatibility weakening admitted.
Implemented narrowed H100 HGX stock/DCP/fault consumer and additive shared constraint inputs. Public runner proofs match source-derived 24/36 baseline family sets and 192/368 samples for eight GPUs, verify actual KSM joins/released attribution, thresholds, temporal edge/window/duration semantics, ECC/Xid/NVLink and bus loss. Negative overlays reject missing stock fields, repeated-edge overcount and unsourced drivers. Legacy full fixture output is byte-identical (SHA256 cec2ab6b5b5afc4f4f65d6d441fdb724ddb63320901377592a19be485a0dbcf1). Pinned exporter health aliases and valid-zero clock-window omission corrected from source. CodeRabbit completed zero findings. Integrated composed gate currently running; no task completion or live deployment claimed.
First integrated just check failed only catalogue admission checks: missing native-OTLP verdict/count entry for the new kind and a barred generic token in a comment. Added source-grounded SCRAPE-ONLY verdict for pinned dcgm-exporter, updated the explicit catalogue count 46 to 47 because the intended catalogue grew, and rephrased the comment. No architecture guard weakened. Focused full archtest passes; changed candidate receives a fresh composed gate. These are metadata/comment corrections after completed CodeRabbit, not a new runtime branch.
Completed on source commit 5e2bb9a, integrated with concurrent HA/CLI work and validated in final published SHA 86634ea40952074beedeb4751609f63b5b9cf8ac. Full local just check passed; CodeRabbit completed with zero exporter findings. Exact CI 38753576146 attempt 2 passed every required job including Go coverage/race, real e2e, chart tests and ci-success. Attempt 1 chart failure was a Go module proxy HTTP2 download error before test compilation; unchanged-SHA retry passed. Earlier c9793d4 CI privacy failure was corrected through the tracker CLI, without a scan exception or history rewrite. Public dump/source inventory proofs and negative tests remain retained under /Users/rob/.local/share/synthkit-worktrees/interactive-dcgm-20261015/codex/. Final composed gate and CI evidence are under /Users/rob/repos/synthkit/codex/backlog-pickup-20261015/. No live GPU deployment claimed; other platform gaps remain explicit.
Final Summary
Implemented sourced H100 SXM HGX DCGM stock/DCP/fault profiles with shared demand and fault admission, source-correct temporal semantics, actual Kubernetes attribution joins and fail-closed unsupported products. Stock/DCP 24 families and 192 samples, selected fault baseline 36 families and 368 samples for eight GPUs. Legacy fixed-input fixture output remains byte-identical. Canonical contracts, schema and docs updated; local gate, independent review and exact-SHA CI pass. Shipped GPU compute emission remains disabled until remaining adapters are verified.