Task · SKT-0158

Diagnose intermittent HA renewal-loss observation timeout under loaded gates

Description

The Go 1.27.3 dependency/security candidate hit TestHACoordinatorRenewalLossSuppressesCleanup/active in cmd/synthkit/ha_loss_test.go:381: HA local behavior was not observed within bound. The same unchanged test passed plain and race in the preceding full gate and passed a focused retry. Preserve the real observation synchronization and current failure bounds; diagnose the occasional missed observation rather than loosening timing or weakening assertions.

Acceptance Criteria

Definition of Done

Implementation Plan

Loop60: bounded focused native measurement with load context; retain existing proof bounds.

Implementation Notes

loop52 BATCH2: landed e2cea58 (pushed 45ff131). Cause 1 (reserve-and-close port race) fixed in ha_loss_test.go; failures now report helper exit/output or a SIGQUIT dump. Cause 2 unfixed: under gate CPU starvation the helper misses its 100ms startup Lease lookup or the 300ms renew deadline (65/130 under 256-hog load on gate-1); removing it needs a bound or production change. Parked pending owner ruling Q05 (park / isolated race tier / bound change). Evidence: codex/loop52/lha/return.md (machine-local). Sibling port pattern: SKT-0161.

Owner ruling 2026-10-22 (loop52 Q05): isolated race invocation with bounds unchanged; tracked in SKT-0162.

Loop54 authorized bounded remote measurement at source8f5a75e (runtime source unchanged from a0554b6), run20261023T194726Z-synthkit-f2a5755f. One quiet-host race job ran HA then Fleet sequentially: 10 top-level HA iterations/40 scenario subtests and 10 Fleet iterations/20 restart-teardown subtests all passed, zero skipped. The same 40 HA cases cover helper readiness, not an extra independent sample. Bounds/source/assertions/isolation unchanged. No reproduction, repair, cleared full-gate failure, loaded A/B or measured isolation benefit established; retain Parked pending naturally failing same-context request/startup/scheduler evidence.

Loop54 closeout: remains Parked. Read-only quiet-host HA measurements passed40HA/20Fleet scenarios without reproducing the failure or proving isolation benefit. Resume on fresh attributable reproduction; caller deadlines, test bounds and isolation unchanged.

This loop performed only route discovery for the permitted quiet-host measurement. The installed runner has slot admission but no documented idle-only or preemptible measurement path guaranteeing required gates are not blocked. No remote job or test started; this is an unavailable measurement, not a green result or proof the host is busy. Retain the park until a compliant route or attributable reproduction is available.

Closeout defect park retained: no documented idle/preemptible nonblocking measurement route was available, so no new measurement ran. Resume on a compliant route or attributable failing reproduction; a free slot is not idle admission and full-gate success does not close the timing defect.

Fresh bounded remote race measurement ran ten iterations of the unchanged HA loss test: forty scenario passes, zero failures/skips/race warnings. Passing does not reproduce or repair the original scheduling/observation fault or establish isolation benefit; retain Parked and preserve every bound.

loop57 closeout: parked/open, see codex/closeout-2026-10-24-loop57.md.

loop58 evidence (SKT-0186 lane): gate-1 ran at load ~50 on 32 cores during loop58 gates; cmd/synthkit HA tests (TestFleetHAGracefulRestartKeepsIDsAndExplicitTeardown 100ms lease lookup deadline, TestHAExpiredGlobalDrainDoesNotRelease) went red under that load. Same oversubscription class; fix routes to the gate-runner per-host heavy-slot cap. Evidence: codex loop58 tlsf return.

loop58: TestFleetUnregisterOwnedHAAuthorityInterruptionAndRecovery failed once in a loaded gate-2 just check (slot 12, ‘interruption consumed intent’); passes -race -count=5 isolated on candidate and base.

loop58: runner TestHAActualFaro5000WholeWrite failed once in a loaded gate-2 just check (f1 lane), passed 100/100 isolated.

Owner closeout: No new measurement: scout lacked native wait helper; no fallback. Preserve existing timeout defect and bounds; resume bounded actual-context native measurement.

One bounded current-context focused native race run passed all four unchanged renewal-loss scenarios, with CPU cap/load and actual readiness/fatal-exit timings retained. Non-reproduction only, not original-cause attribution or repair. Resume with a new failing actual-context log or discriminating controlled scheduling hypothesis; no unchanged repeated sampling.

Loop60 closeout: The bounded current-context four-scenario pass is non-reproduction, not repair or cause attribution. Keep every deadline, observation and cleanup assertion; resume only from attributable failing evidence or a discriminating controlled scheduling hypothesis, not unchanged sampling.

Final loop closeout: original attribution remains unresolved; ordinary later gate passes do not reproduce or repair it. Retain the prior task-specific diagnostics and exact failure evidence; resume only on new attributable actual-context evidence, not unchanged green sampling.

View the source file on GitHub