Description
The value of AI Factory observability is correlation: a GPU fault, node log, scheduler event and storage latency spike lined up on the same identities and time window. The six operational journeys (slow job, failed job, GPU fault, degraded NVLink domain, hardware replacement, observability failure) each need a scenario whose effects span constructs, so dashboards, alerts and runbooks can be rehearsed before hardware exists.
Acceptance Criteria
- #1 Named scenarios exist for an Xid-driven job failure, backend fabric congestion causing a slow job, storage metadata latency causing a slow job, a degraded NVLink domain, a BMC hardware fault leading to a node drain, a cooling fault leading to thermal throttling, and a gateway WAN outage
- #2 Each scenario’s effects land on consistent fixture identities and one time window across every construct involved
- #3 docs/incidents.md documents each scenario, its effects and the signals that change
Definition of Done
- #1 just check (fmt-check, lint, gen-check, env-check, docs-check, test, race, hygiene, ui-check, compose-check, helm-test, lab-check, signal-fidelity)
- #2 just gen (only if a blueprint field, construct/workload config struct, or a skill under plugins/synthkit/skills/ changed)
- #3 just dump — inventory diffed against signals/