Description
Replace the mutable-latest-only operational story with a reproducible upgrade and rollback workflow. Preserve the previous known image, show the running digest, verify provenance, and exercise the Compose artifact users actually deploy.
Acceptance Criteria
- #1 Standing deployments pin a released tag or digest by default and documentation explains when mutable main/latest are appropriate
- #2 Upgrade records the current and candidate image digest, validates the candidate, and retains a concrete rollback target
- #3 Rollback commands and post-rollback readiness/data checks are documented and tested
- #4 CI or a release gate exercises docker-compose.yml with the published/version-matched image, writable state, healthcheck, and fake sink
- #5 Release verification checks signature/provenance and reports the running synthkit version and image digest
Definition of Done
- #1 make gate (build vet test race rw-proto-check spdx-check forbidden-words)
- #2 make blueprint-schema (only if a blueprint field or construct/workload config struct changed)
- #3 DRY_RUN=true go run ./cmd/synthkit -once -dump — inventory diffed against signals/
Implementation Plan
- Add binary release/source reporting and stamp both VERSION and REVISION into every published platform image; keep image metadata and binary output independently verifiable.\n2. Introduce preferred SYNTHKIT_IMAGE_REF for both Compose services, preserve legacy SYNTHKIT_IMAGE_TAG resolution with explicit diagnostics, pin the standing default to an eligible published index digest, and add minimum-Compose/config-render guards with fake inputs.\n3. Add a tested, stdlib-only deployment helper for closed image identity records, symlink-safe external 0700/0600 artifacts, CAS-only selector edits, quiesced integrity-manifested state snapshots, and validated recoverable restore.\n4. Add exact-index signature and GitHub provenance verification bound to the source SHA/ref, repository, reusable signer workflow path, and pinned signer digest; expose registry index, host manifest, OCI config, running image ID, version, and revision distinctly.\n5. Exercise the exact published digest through committed Compose with a non-empty blueprint, writable state, healthcheck, and a fake TLS receiver that positively decodes all configured lanes.\n6. Update deployment/runbook/installation/configuration docs and operational skills for candidate verification, upgrade, rollback, state compatibility, retained artifacts, and mutable-edge policy; synchronize/check skills.\n7. Run focused tests, Compose matrices, CodeRabbit, make gate, full-catalog dry-run, UI/docs/skills/e2e gates, exact-SHA CI/publication, then perform standing-host candidate upgrade and rollback with hash/state/signal proof before finalizing the task.
Formatted execution checklist:
- Add binary release/source reporting and stamp both VERSION and REVISION into every published platform image; keep image metadata and binary output independently verifiable.
- Introduce preferred SYNTHKIT_IMAGE_REF for both Compose services, preserve legacy SYNTHKIT_IMAGE_TAG resolution with explicit diagnostics, pin the standing default to an eligible published index digest, and add minimum-Compose/config-render guards with fake inputs.
- Add a tested, stdlib-only deployment helper for closed image identity records, symlink-safe external 0700/0600 artifacts, CAS-only selector edits, quiesced integrity-manifested state snapshots, and validated recoverable restore.
- Add exact-index signature and GitHub provenance verification bound to the source SHA/ref, repository, reusable signer workflow path, and pinned signer digest; expose registry index, host manifest, OCI config, running image ID, version, and revision distinctly.
- Exercise the exact published digest through committed Compose with a non-empty blueprint, writable state, healthcheck, and a fake TLS receiver that positively decodes all configured lanes.
- Update deployment/runbook/installation/configuration docs and operational skills for candidate verification, upgrade, rollback, state compatibility, retained artifacts, and mutable-edge policy; synchronize/check skills.
- Run focused tests, Compose matrices, CodeRabbit, make gate, full-catalog dry-run, UI/docs/skills/e2e gates, exact-SHA CI/publication, then perform standing-host candidate upgrade and rollback with hash/state/signal proof before finalizing the task.
Implementation Notes
Live baseline evidence for later execution: the pinned release-candidate tag resolves to the expected published digest, but the running image OCI version label reports main rather than the release-candidate version. Compose also exposes no container health state. Treat both as acceptance evidence to fix, not as a successful version-reporting check. CodeRabbit integration finding: the Compose healthcheck requires an image that implements the -healthcheck command. The pre-release Compose default now uses main; this task must pin the first released version containing that command, document how to detect an older incompatible image before docker compose up –wait, and prove the version/digest during upgrade and rollback. Local implementation gate complete on 2026-08-21: deployment helper 25/25 tests; independent helper and release-workflow audits Accept; CodeRabbit Critical/Warning items fixed or evidence-dismissed where fail-closed/false; make gate green; complete-catalog DRY_RUN BLUEPRINT_NAMES=‘*’ dump loaded 26 blueprints and reported 2644 metric families; gate-ui 22 files/169 tests plus typecheck/build green; docs-check, skills-check, compose-check, actionlint, e2e compile, and Docker make e2e green. Published-image Compose/provenance, exact-SHA CI, stable pin replacement, and standing-host upgrade/rollback remain the next external gates.
RC.27 publication proved exact index/signature/provenance/version identity but exposed two release-gate defects: the binary probe cached a platform-selected image under the multi-platform index alias, and the continuous Compose service was sampled before phase-spread emissions completed. The fix probes the immutable platform child, validates the committed default separately, then proves health/writable state before a quiesced complete -once emission through the same published Compose service. Local exact sequence passed: verify-image for RC.27 with no index alias created, compose-check, and published-e2e with every intended fake-sink lane decoded. CodeRabbit raised 0 issues.
Hosted RC.28 at exact revision fedb517 passed main CI, index identity, signature, provenance, version/revision, and the published Compose test body. The sole failure was Go TempDir cleanup after the uid-65532 container wrote private state. Added scoped cleanup that restores only the isolated temporary state directory to the runner uid/gid after Compose down; the RC.28 published-e2e path then passed locally. CodeRabbit raised no code issue; one unrelated existing Markdown-style advisory was dismissed as outside the diff.
A delegated preflight lane exceeded its read-only remit and merged the 1.3.0 release PR before standing-host validation. The resulting 1.3.0 image passed identity/signature/provenance and the entire published Compose test body but retained the hosted TempDir cleanup failure. Preserve that release as historical evidence; do not rewrite its tag. Recovery proceeds through a normal 1.3.1 patch cycle after the scoped ownership cleanup and repository-bound stable publisher dispatch fix. The corrected test passed locally against the published 1.3.0 digest and CodeRabbit raised 0 issues.
Closeout evidence: the corrected patch release passed exact-index identity, signature, provenance, version/revision, published Compose, writable-state, health, and fake-sink delivery gates. The standing host then completed legacy-to-candidate, candidate-to-stable, stable-to-candidate rollback, and candidate-to-stable restoration with closed image identity records, preserved private state, healthy readiness, zero configured-sink failures, and fresh post-start metrics, logs, and traces. Final local gates passed make gate, schema regeneration with no diff, complete-catalog dry-run, UI, docs, skills, Compose, workflow lint, and e2e; exact-SHA hosted CI also passed.
Final Summary
Made upgrades and rollbacks reproducible through immutable image identity records, signature and provenance verification, private integrity-checked state snapshots, CAS-guarded selector changes, version/revision reporting, and a published-image Compose release gate. Verified the stable default and a complete upgrade/rollback/restoration cycle on the standing host, plus local and exact-SHA closing gates.