Description
AI Factory designs send all egress through a clustered pool of identical Alloy gateways with separate roles (OTLP receive, central scrape, syslog, cloud forwarding) and disk-backed sending queues for metrics, logs and traces; host collectors hold no buffer. Collector configuration comes from Fleet Management through a proxy at the gateway. The observability-failure journey needs queue depth and capacity, failed and dropped sends, config reload failures, cluster peer state, scrape failures, syslog parse errors and credential expiry, and telemetry-gap alerts must distinguish a silent source from a failed component. alloy_health already emits part of this (signals/fm.md).
Acceptance Criteria
- #1 A gap analysis against signals/fm.md and the alloy_health construct is recorded before any new series is added
- #2 Gateway roles and pool size are declarable, and the role-appropriate Alloy and otelcol series (persistent queue size and capacity, send failures per signal, clustering peers, scrape health, config reload) are emitted with names from the Alloy documentation
- #3 Failure modes cover a gateway instance loss with pool rebalancing, a WAN outage where queues grow and drain without loss inside capacity, queue overflow with drops, a config reload failure and an expired cloud credential
- #4 A per-source telemetry-gap scenario stops one source’s series while its component stays healthy
- #5 Every metric, label, attribute and log field is sourced from vendor documentation or a capture at a pinned version and recorded with provenance in the named signals/ area file; anything unconfirmed is a PENDING in cantfind.md, never an invented name
- #6 Rates, counts and cardinality in any shipped blueprint are plausible for a real deployment, with the real-world basis stated in a comment; nothing is raised to force emission
Definition of Done
- #1 just check (fmt-check, lint, gen-check, env-check, docs-check, test, race, hygiene, ui-check, compose-check, helm-test, lab-check, signal-fidelity)
- #2 just gen (only if a blueprint field, construct/workload config struct, or a skill under plugins/synthkit/skills/ changed)
- #3 just dump — inventory diffed against signals/
Implementation Plan
Extend the existing alloy_health addon with optional declared gateway pool and roles, preserving its omitted-config baseline and the landed syslog mechanic. Use the retained pinned gap matrix before adding series; implement role-specific queues, peers, scrape/config health, failure transitions and source-series cessation with real fixture/state tests. Validate custom fixture inventories and source names, gen/check on the isolated candidate; root owns integration, batch review and exact-head CI.
Retry2 uses the explicit same-process remote-config source and omission contract. Verify exact translator/default names and component healthy enum from pinned Alloy source, implement pool roles/queue/peer/reload/failure state and member-local data omission, prove public loader and before/during/after inventory in the specifically owned integration test, then own gen/dump/check.
Implementation Notes
loop47: admitted R16 vendor-contract research; implementation waits for shared syslog landing
loop47 dependency park, implementation0: shared syslog SKT-0087.02 (Alloy receiver mechanic) blocked on unchanged OTel parse-error AC3 versus confirmed source absence. Resume after authoritative signal/acceptance decision and green mechanic landing; shared alloyhealth ownership remains serial I02 then I16. R16 gateway contract retained, no gateway output/queue/failure acceptance inferred.
loop48: admitted implementation1 after the shared syslog code and real-health AC landed on main at ea510aa3198e3775105f9b05b963760374be67ee with local integrated gate and CodeRabbit complete. Landing CI remains root-owned pending; this independent source consumer may work against the published mechanic, but final acceptance and any dependent publication remain gated on applicable CI. Loop47 consumed no implementation attempt, only a dependency park. Prior blocker changed by owner-approved AC3 and verified implementation; discriminators pool transitions, source-gap omission with healthy component, source inventory and own full gate.
loop48 attempt1 stopped before code on the root packet omission of a named source profile and ownership; AC1 gap matrix retained, implementation1 conservatively consumed, infra0. Root supplies same-process Alloy remote-config profile from pinned gateway contract: successful-load gauge and attempt/failure counters per declared gateway member, 60s expected cadence, owned collection omission before Metrics.Write while independently sourced healthy component observations remain. No duplicated external producer or shared runner filtering. Retry2 is the last ordinary worker attempt, with real before/during/after source-absence proof and translator verification before emission.
loop48 implementation2 partial manifest2eda7ad5ee20914dc9159092e3f843d58ac2361ae0c51ae557e5f6e4348e7bd6 stopped on first implementation red: the real public gateway runner helper again omitted the log writer required by the unchanged Kubernetes event lane. Root verified runner.Sinks has Metrics only. Draft source/role/queue code and pinned translator/healthy-enum evidence retained, AC2-6 still unmet. Attempt2 settled, infra0; root-directed rescue3 reserved with explicit real LogCapture correction, no sibling-runtime change, assertion weakening or renewed worker allowance.
loop48: implementation1-3, infra0, review-repair0; landed78d5f6dbdfbd1fc72d1232eed200e1a2b1b37851, exactci37146837472 all10 jobs success. Root gen/dump and neutral check tested4193b5f1ed6d7f70e13aa81f8a00a4db55eea344; rebase difference is foreign canonical Backlog document only, application/blueprint identical. CodeRabbit0 findings/full16-file batch coverage. Inventory327->343 patterns, intentional legacy gateway replacement shapes. Eight optional external rows remain BLOCKED_EXTERNAL. Final LIVE preparation stopped on expired AWS SSO before env/build/container; resume live verification after owner auth refresh and exact source rebuild.
Final Summary
Delivered optional role-specific gateway pools with bounded modeled request queues, peer/scrape/reload health, five fault/recovery sequences and selected-member same-process source cessation before the public writer. Public loader captures and eight assertion-negative controls prove conservation, retained backlog, role isolation, healthy witnesses and continued polling. Shipped pilot rates retain one aggregate syslog record/minute, no steady trace/app-log intake is fabricated. Persistence is modeled emitter state, not disk WAL; generic send failures do not uniquely diagnose expiry. Final-candidate live readback remains auth-blocked, not a substitute for these ACs.