Description
The hot store is roughly 23,000 files across 10 namespaces and clients append to a transcript for the life of a session, checkpointing at most once every five minutes. Reading from the start each pass is unaffordable and reading only new files misses everything still being written.
The tree is read-only. Retention and archiving are owned by agent-session-archive.timer on camden and this service is never in that chain.
Acceptance Criteria
- #1 Walks configured namespace roots and tracks each file by a content-prefix identity, not by name
- #2 Resumes from a stored byte offset and never re-emits or skips a record across a restart
- #3 A partial trailing line is not an error: the reader stops at the last complete line and resumes there
- #4 Never writes, renames, locks or touches anything under source.root; a test asserts the tree is unmodified after a run
- #5 Idle files fall back to a slow sweep rather than being polled at full cadence
- #6 Emits bytes_read, files_tracked and ingest_lag
Definition of Done
- #1 just check passes: fmt-check, lint, build, test-short, probe-ci, dashboard-check and docs-links all clean
Implementation Plan
Wave 1 lane 1: implement source discovery, tailing, checkpointing and restart safety within internal/source; run focused source tests; root integrates and runs the final gate.
Implementation Notes
Mechanism independently re-verified against source on 2026-09-22, and it is worse than the park recorded.
contentIdentity (internal/source/source.go:583-610) hashes the first line when a newline falls inside DefaultPrefixBytes=4096, otherwise the raw first 4096 bytes. Checkpoints are keyed by that identity, not by path (internal/source/checkpoint.go:39).
The park described a one-time replay. It is not one-time: two files sharing an identity share one map entry, and source.go:337 writes the checkpoint back under that shared key every sweep, so the two files continually stomp each other’s offset. That loses records as well as repeating them, and nothing in the output distinguishes either case. Fix the identity, not just the replay symptom.
Trap on the fix: adding size or mtime to the identity reintroduces replay on every append, because an append changes both and the old identity’s checkpoint entry is then orphaned. TestContentPrefixIdentitySurvivesRename and TestSourceRestartResumesAtByteOffset must both stay green.
The memory ceiling this task’s resume boundary referred to does not exist anywhere in internal/source or internal/config. That work moved to CAOT-0040; repairing the identity alone will not survive a full sweep.
Final Summary
Parked after live Camden expansion found 14,412 files but only 14,392 first-record identities. Collided identities can reset offsets and replay multi-gigabyte files; the bounded installed namespaces remain healthy. Resume by making identities collision-safe, adding a live duplicate-first-record regression, and re-running the full namespace start below a bounded memory ceiling.