Description
AI Factory telemetry is only useful if every source agrees on the same physical and logical identity: site, rack, node, GPU index and UUID, NVLink domain and partition, compute tray and switch tray, and the fabric port each NIC plugs into. No fixture models GPUs today; host and Kubernetes nodes carry no GPU identity. Two hardware shapes are needed: PCIe GPU servers (typically 4 to 8 GPUs per 2U/4U node, air-cooled) and rack-scale NVL72 systems (18 compute trays of 4 GPUs plus Grace CPUs, 9 NVSwitch trays, one NVLink domain per rack, liquid-cooled). DCGM, driver logs, Redfish, NMX, BCM, Mission Control, the scheduler, storage clients and both fabrics all join on this fixture, so it is a frozen seam: design and review it before any dependent construct starts.
Acceptance Criteria
- #1 A blueprint can declare GPU node pools of both shapes with node count, GPU model, GPUs per node, site and rack placement; GPU UUIDs, serial numbers and BMC identities are deterministic per seed and stable across restarts
- #2 The fixture exposes node-to-Kubernetes-node binding, NIC-to-switch-port attachment for a frontend and a backend fabric, storage-client identity, and NVL72 rack-to-NVLink-domain, partition and tray mapping
- #3 Blueprint load rejects collisions and inconsistent shapes, such as more GPUs than the shape allows, a tray in two racks, or a node bound to two Kubernetes clusters
- #4 ARCHITECTURE.md or docs/ describe the fixture and its join keys; the task emits no telemetry on its own
- #5 The seam is reviewed and frozen before any task that depends on it starts
Definition of Done
- #1 just check (fmt-check, lint, gen-check, env-check, docs-check, test, race, hygiene, ui-check, compose-check, helm-test, lab-check, signal-fidelity)
- #2 just gen (only if a blueprint field, construct/workload config struct, or a skill under plugins/synthkit/skills/ changed)
- #3 just dump — inventory diffed against signals/
Implementation Plan
loop47: implement root-frozen design rev3, all exact fixture/blueprint types and functions, sourced nullable catalogue, pure allocation/physics and load-time rejections. Shared tick/nettopo and renderer adapters stay root/consumer-owned; no consumer AC inferred from the generic seam.
Implementation Notes
loop47: admitted
loop47: generic seam frozen after independent second challenge with bounded period/origin/coverage/aggregate corrections. No fixture consumer implementation started before freeze. Source-dependent consumer profiles and runtime adapter gates remain explicit.
loop47: impl1 stopped blocked on shared-only CoversTarget versus source-local selected-PDU validation; scaffold preserved uncommitted, no AC proof. Root froze bounded clarification rev4, shared APIs unchanged and disjoint local registration/axis/selected-profile validation with no physics effect. impl2 resumed same worktree preserving scaffold; second worker attempt, infra0. Runtime tick adapter still root prerequisite.
loop47: generic seam frozen revision4 after two independent design/challenge rounds and bounded clarification. Attempt2 candidate47eb03c correctedgate exit0 but independent implemented-candidate review BLOCKING: unrelatedEKS ordering,host sibling falsecoverage,VAST storage-only false-reject,YAML merge fractional truncation,PCI suffix acceptance. Root rescue1 (third implementation reservation) will repair these five concrete defects against frozen API; no consumer admission or AC1-4 acceptance. Impl2 inventoryinfra retries2 retained; correctedsetup gate is not code pass for these uncovered counterexamples.
loop47 finalized: implementation attempts4; separate review-repair1of3; historical original inventory retries2 retained, final rescue/review-repair infrastructure retries0. Landing d592c9af0ed3b3b5992fbe77e8d2647685af3d17 ci37081070813 completed success. Root scratch Go poisoning caused two failed integrated vet gates before full green; preserved evidence as .go.txt, no assertions weakened. Eight BLOCKED_EXTERNAL troubleshooting rows unexecuted, static lab validation not collector capture. F1 root-owned genotype stanza declares compute-user with collection disabled pending sourced adapters.
loop47 post-fixture integration finding: root-owned referencecompute addition usedk8s_monitoring.enabledfalse incorrectly asbase-emissiongate; I21realCLI exposed10249unadaptedcomputeK8sseries. Corrected explicitcluster.emitfalse in001de3485e6e3815f7e7dada053dd01819141016 withpublicloaderbinding/strictdisabledconfigtests, actualCLI2379series, wholelocalgate, CodeRabbit0findings andci37090183079SUCCESS. FrozenGPUAPI/20nodes80GPUs retained; genericfixtureDone remainsseparatefromvendor/adapters and I21unmetGPU-sizingAC4.
Final Summary
loop47: landed generic GPU topology fixture, frozen rev4 API, deterministic physical joins, pure allocation/physics, strict loader validation and source-gated nullable catalogue. Four implementation reservations plus separate CodeRabbit review-repair-1; original five public-boundary defects, alias expansion resource defect, and generated-IP collision regression repaired with red/green evidence. Final full local just check green at d592c9af0ed3b3b5992fbe77e8d2647685af3d17; gen and actual ai-factory CLI dump validated. Full unfiltered safe inventory equal under identical fixed clock/random process edges for exact repaired isolated d4e6425133d30bb2040233b0e6e3f9e46d8b9b8d versus blob-verified692bb91; independently challenged by driftcheck2. Two unmodified randomized inventory reds retained, not claimed passes. CodeRabbit complete31files2major both fixed flagged hunks, no dismissed findings. Exact landing ci37081070813 SUCCESS at d592c9a. Generic fixture acceptance grants no vendor/consumer/live acceptance; thermal/usable-framebuffer/physical-NVLink metadata remain nullable and fail closed.