Expose delivery loss, queue pressure, and deployment sizing

Description

Make best-effort delivery failure visible and document operational capacity. Failed batches are currently discarded after sink retry exhaustion while troubleshooting claims they remain buffered, and default queue/log/resource ceilings are not explained.

Acceptance Criteria

• #1 Documentation states the actual retry and loss contract and no longer promises failed batches are buffered until recovery
• #2 Status and self-observability expose items dropped after failed flush, queue depth/backpressure, and last loss timestamp
• #3 Operators can distinguish recovered current delivery from cumulative historical failures
• #4 Deployment guidance gives queue-memory sizing, tick/backpressure consequences, and recommended CPU/memory/log rotation limits
• #5 Tests prove failed-flush loss accounting and recovery-state transitions

Definition of Done

• #1 make gate (build vet test race rw-proto-check spdx-check forbidden-words)
• #2 make blueprint-schema (only if a blueprint field or construct/workload config struct changed)
• #3 DRY_RUN=true go run ./cmd/synthkit -once -dump — inventory diffed against signals/

Implementation Plan

Run contract: root owns the loss-state contract, shared runner/self-observability/control/UI wiring, docs, tracker, commits, pushes and gates. One bounded execution lane owns only internal/sink/queue; no lane commits or touches live systems.

1. RED: extend queue tests to prove exact failed-subbatch loss accounting and that a subsequent wholly successful flush is the recovery transition while cumulative loss remains.
2. GREEN: extend the stdlib queue observer flush event with exact dropped-item count and preserve existing batching, depth and retry behaviour; align the package fallback capacity with the configured 500000 default.
3. Root wiring: fold per-sink queue depth, blocked enqueues, cumulative dropped items, last-loss/recovery times and current-loss state into pushstatus; export queue dropped/current-loss/last-loss self-observability metrics; expose the sanitized operational state through authenticated /control/status and the UI.
4. Correct the retry/loss contract and add measured capacity, backpressure, CPU/memory and log-rotation guidance without promising a WAL or recovery buffering.
5. Run focused queue/pushstatus/selfobs/control/UI tests, review the integrated diff, run CodeRabbit, make gate and the explicit complete-catalog dry-run inventory, then live-failure/recovery proof on standing host before finalization.

Security-reviewed v4 supersedes the earlier implementation steps where they conflict. Root owns shared sink/pushhook/runner/pushstatus/selfobs/control/UI/docs wiring, tracker, review, gates and live proof. One leaf owns internal/operationalerr only; one leaf owns internal/sink/queue only.

6. RED/GREEN: add closed operational error codes and queue tests for exact failed sub-batches, queue-owned immutable completion sequence, shard identity, sanitized return/log paths and 500000 fallback capacity.
7. Root converts push observers to closed codes before fanout, removes raw error storage/span recording, and adds broad encoded-secret canaries.
8. Root folds affected-shard queue loss/backpressure history into pushstatus, fans identical queue events to selfobs and status, exposes queue state in authenticated status/UI, and exports bounded loss metrics.
9. Correct retry/loss and deployment capacity/resource/log guidance.
10. Run focused Go/UI checks, CodeRabbit, make gate, complete-catalog dry-run inventory, then isolated standing host fake-sink loss/recovery proof with standing-deployment hash checks before finalization.

Implementation Notes

Research baseline at f13da008eaf4db62ec0c377b97a22f8585598e86: failed queue sub-batches are discarded after bounded sink retry; queue depth, blocked enqueue and flush outcomes exist, but exact loss count, last-loss/recovery state and status exposure do not. docs/troubleshooting.md overpromises buffered recovery. Focused existing tests passed for queue, pushstatus, selfobs and control. Implementation begins with the queue leaf seam; shared wiring remains root-owned.

Security review accepted immutable implementation contract v4. It requires closed errors before observers/logs, queue-owned monotonic completion sequence, affected-shard recovery, broad encoded-secret canaries, and an isolated standing host fake-sink project.

Final verification 2026-08-21: focused Go packages passed; control UI passed 22 files/167 tests and production build; git diff –check passed; paid-path CodeRabbit completed with no Critical or Warning findings after three minor corrections; make gate passed including build, vet, plain/race tests, schema/docs/skills checks, RW2 provenance, SPDX and forbidden-words. Complete-catalog DRY_RUN=true BLUEPRINT_NAMES=* go run ./cmd/synthkit -once -dump exited 0 for all 26 blueprints; no blueprint field, construct/workload config, or signals catalogue change required schema regeneration. Isolated standing host proof used project synthkit-loss-proof, fake credentials, TLS fake receiver, port 19088 and separate uid-65532 state. Receiver loss changed readiness true-to-false, Loki current_loss false-to-true with 2 affected shards and dropped_items 30; recovery restored ready/live_ready, cleared affected shards, retained cumulative Loki 51 and PromRW 462 drops, and advanced recovery timestamps. Standing deployment remained byte/identity stable: .env SHA-256 3d0886016f9fcc3fba6c2418ff14cec3b3c9fb056a200c15009fceb11a6c2216, /data manifest 0bc0e04202c9ef9bb48aa6d3280180e2345e909109d69e93878d4b52711a4a1a, container 04686bc6cfc332bcaa1f2895fc7b906ce09808515ae60886744161e546c3804a, image sha256:732841ee377b2cc5c263d3486e23ab8b58dc26a2a8e7102016dfc8eaeabcf968, healthy. Temporary proof containers/network and fake-secret directory were removed.

Final Summary

Implemented closed operational errors, exact queue-loss and shard-aware recovery accounting, authenticated status/UI and self-observability exposure, plus corrected loss/sizing/resource/log guidance. Verified with focused Go/UI checks, paid CodeRabbit, full make gate, 26-blueprint dry-run inventory, and an isolated standing host TLS fake-sink failure/recovery proof that left the standing deployment unchanged.