Task · SKT-0099

Make Synthetic Monitoring provisioning work without a shared volume

Description

SM provisioning is a separate two-boot handoff: cmd/sm-provision takes an O_EXCL file lock and writes a registration the emitter reads at boot, and the registration binding includes the release version (internal/smstate/state.go), so every image bump silently disables SM emission until the provisioner re-runs and the emitter restarts. Under stateless HA: run sm-provision against the kubernetes state backend with a Lease-based lock, and stop a release-version bump with unchanged SM specs from invalidating the registration.

Acceptance Criteria

Definition of Done

Implementation Plan

Implement approved dedicated provisioner Lease, aggregate CAS sm-state key, v2 release-independent activation with strict tool/snapshot version compatibility and explicit format migration. Refresh consumer census against current main before changing. Prove local CLI with fake named Kubernetes and SM HTTP plus actual emission/restart behavior; preserve private file evidence, compare inventory before/after, regenerate config/skills outputs, full local gate and pre-land guarded reviews. Chart wiring is a separate unadmitted follow-up.

One owner-raised lineage cycle via SKT-0127 (synchronize lost-response fixture observations), five total. Resume exact retained feature; no chart wiring or independent fresh budget.

Implementation Notes

Implementation preflight found no decision-changing engine drift. Chart named-state HA wiring has landed, while SM Job remains file-only; follow-up will extend that existing seam. Zero implementation attempts: required CLI HTTP proof needs an explicit backend client injection seam. Root grants backend_kubernetes.go solely for additive explicit injection using the existing validated gated backend, matching the Lease adapter precedent. Production credential discovery and accepted backend/gate/CAS contracts stay unchanged.

Candidate partial after two change-and-verify cycles (implementation/full gate, then owned architecture repair/focused race); initial ownership inspection consumed zero. Baseb8a0ef1da85ed9b0942be2398d35083ce90561f5, treeb6c51fac54a9b98f98dea061a5bd03d8a650ca59, complete39file patchd9b91adf56df730fffd8c9f1c3b904f75ede725287e5e94651b7be92238d4e91 verified and retained by root. Dedicated Lease/CAS/v2 activation/file evidence implemented with actual fake HTTP CLI and runner release-bump/restart positives, inventory byte-identical, focused race green. Full gate ran before final architecture correction and was red; unchanged chart test combines Helm mutually exclusive flags and fails before fixture API calls on available4.3.0/3.20.0. No assertion changed, no final fullgate pass claimed. Full composed-runner negative matrix still misses removal, target/schema mutation and all incomplete/duplicate/bad-Modified evidence cases. Root dispatched compatible-tool prerequisite mapping and independent full-candidate security review before next rescue. Initial CodeRabbit39files had2contract-conflicting majors under review; post-architecture checkpoint review emitted all39files0findings, a disclosed second-full-review cadence defect. Nothing accepted or landed; no CI.

Full independent security review BLOCK on treeb6c51fac: major torn file activation read in internal/smstore/file.go:33,45 can combine stale nil Pending with newly complete registration while durable pending remains; source-confirmed interleaving, no runtime reproduction yet. Both initial CodeRabbit majors rejected against approved mapping/lock requirements. Compatible official Helm3.19.0 focused unchanged chart proof passed all4cases with actual fake API reads/precreation and byte-identical state. Third counted change-and-verify cycle routed to retry worker for fail-first race repair and complete required composed-runner negative matrix, then exact finalfullgate and guarded delta reviews. Ceiling remains4; no reset. Local retained checkpoint dd4102104ae75654ada28c5d29882fbf8b5885ad identifies exact last-reviewed tree for limited delta review and is not a land.

Retry preflight stopped before edits because narrowed repair brief omitted cmd/sm-provision/lease.go file-mode lock composition. Root inspected and added that branch only for positively acquired store/release composition, preserving all Kubernetes Lease behavior and frozen APIs. Candidate remains exact prior tree, real index empty, no gate or CodeRabbit run; zero additional cycles consumed. Third counted cycle continues on same retry route.

Third counted cycle repaired the torn file read with positively acquired O_EXCL store lifetime and post-release refusal; actual concurrent apply/handoff regression failed before and passed after. Expanded runtime persisted-evidence matrix90cases green; bound-spec matrix39cases observed no failures in prior focused invocation, but whole invocation red for two new fixture-index errors later corrected. CodeRabbit exact seven-file delta completed0findings. Final fullgate exit1 before tests/race: ignored .go evidence copies under scratch became a Go package and redeclared symbols. Candidate treed0f02189e64c743b78e2241db0bc09af3851c7d2, patch81683d810e4ae34414a1500b40554e775e965b871132dee1fb4d7ba6d10744e2, source bytes unchanged after gate and root verified. Fourth/final rescue cycle owns artifact preservation/rename and overlay-map update only, then one fullgate; delta security and native pinned history/candidate scans run independently. No fifth cycle permitted.

Scoped delta security CONDITIONAL PASS: F1 resolved by positively owned consistent read, actual concurrent apply/handoff proof accepted, new runtime negative source matrix complete; finalfullgate remains external. Minor inherited PublishSnapshot lifecycle caveat recorded separately as SKT-0126 (fence snapshot publication on acquired provisioner stores): no production call or lock bypass, left unchanged. Native pinnedgitleaks8.30.1 history and exact1727file materialized candidate scans both exit0/no leaks, existing config unchanged. Count reconciled883reachable=15no-diff merges+1empty+1deleted-file-only+866fragment-eligible, consistent with tagged scanner/parser semantics; no unexplained text coverage gap or scan rerun. Final artifact-only fullgate still running.

Four-cycle ceiling reached. Code provisionally landed as d850591d71efcdcbca80df2597bd89625d36d08e after exact local fullgate0, guarded full/delta review and pinned scanner evidence. Mandatory composed just check at that exactSHA failed in race tier: TestKubernetesProvisionLostSMResponseRetainsPending reads fakeSM.posts in kubernetes_test.go:429 concurrently with ServeHTTP write in main_test.go:110. This is a genuine new test-fixture race, not hostload/infrastructure; prior localgreen does not clear it. Root reverted only its code with c4c61fb437e05bfc4841e289ff79abe94679906a, restoring the exact pre-code tree while preserving all tracker/instruction work and complete candidate evidence. No root rescue, unchanged rerun or fifth cycle. Original and restored exact CI remain owned; restoration composed gate running. No criteria checked. Park defect at lineage ceiling4; resume exact retained treed0f02189/patch81683d81 only after owner raises ceiling for synchronized HTTP fixture observations, reviewed repair and exact gates/CI. A separately recorded repair task does not reset that budget.

Original pushed d850591 CI38170368433 terminal overallcancelled; required ci-success114564142236 failed because go race and e2e jobs were cancelled by superseding revert push. This is not a CIpass or infrastructure-rerun authorization; no rerun requested. Root reverted immediately on proven composed fixture race before original CIterminal, a disclosed never-cut-land-mid-CI timing deviation. Restoration has its own exactSHA watcher and composedgate. Source defect remains independently reproduced locally; original CI cancellation is not substituted as code-failure proof.

Restoration verified: exactc4c61fb437e05bfc4841e289ff79abe94679906a full composed just check exit0, clean detached source, plus exact CI38171230322/ci-success114569308907 success. This proves restored baseline only; feature stays reverted, Parked and all criteria unchecked. Complete code candidate, fail-first proofs, clean localgate/reviews/scans and failing composed race are retained. Chart wiring SKT-0099.01 (extend existing named-state chart) is To Do/unadmitted; fixture repair SKT-0127 (synchronize lost-response observations) is Parked at original lineage budget. Resume only with owner-raised cycle authority. Root finalpublication contains tracker receipts only; no engine implementation survives this run.

loop16: one additional owner-raised cycle through SKT-0127 (synchronize lost-response fixture observations), five total lineage. Two fixturetestfiles synchronized after genuine count10racefailfirst; all37otherretainedfeaturefiles unchanged. Focusedrace0, twofileCodeRabbit0, securitydeltaPASS, rebasedfull laneG1270. Feature still notpublished: finalcurrentmaincomposed/CI blocked by harness missing nopush launcher; no toolmodefallback, rescue or freshbudget. Resume exact retainedcandidate through SKT-0127 after runtimerestored and precedingHelmpublication; SKT-0099.01 (chart wiring) staysToDo and unadmitted, SKT-0126 (fence snapshot publication) deferredminor unchanged. Allcriteria remainunchecked.

loop18: completed through SKT-0127 (synchronize SM lost-response fixture observations), no fresh implementation cycles. Reviewed39file diskless provision/activation source re-landedb676b0dfb548fabb6763c19e6485f09e405d0ed1; exact CI38271867692 all11required jobs green. Prior pending-evidence, no-replay/activation, file ownership and named-state/fencing proofs retained. Chart job extension remains SKT-0099.01 (wire diskless SM provisioning into named-state chart).

Final Summary

Engine provision/activation accepted atb676b0dfb548fabb6763c19e6485f09e405d0ed1 after retained reviews and all-required exact CI38271867692.

View the source file on GitHub