Description
The use case needs real agent behaviour over fictional study data: five agents (orchestrator, source, drafter, reviewer, readability), pure fixture-backed tools, a committed corpus of authoring requests for the load generator, and experiment suites. Mirrors the Touchline agents design (apps/agents, apps/mcp-tools, apps/corpus); see doc-0003 for names, tools and the observability story. Fixtures must be obviously fictional.
Acceptance Criteria
- #1 apps/regulatory runs all five roles locally with STUB_MODEL=1 and its npm test suite passes
- #2 Every request ends approved, needs_revision or escalated and records that outcome as telemetry
- #3 Tool results pass the preflight guard before a model sees them; a source-note injection probe path exists in the load generator
- #4 At least two experiment suites exist, one comparing section-prompt variants and one guard policy suite, each runnable by the experiment runner
- #5 tools/scrub_check.py passes and no fixture names a real company, drug, trial or person
- #6 apps/regulatory runs all five roles locally with STUB_MODEL=1 and its npm test suite passes
- #7 Every request ends approved, needs_revision or escalated and records that outcome as telemetry
- #8 Telemetry records the guarded tool result, the subject-identifier guard fails closed, and number-grounding and reading grade are also computed in-app as metrics so panels fill when content capture is off
- #9 Tool results pass the preflight guard before a model sees them; a source-note injection probe path with its own marker list exists in the load generator
- #10 One experiment suite compares section-prompt variants; the guard-policy suite is built in AIO-0018.05
- #11 tools/scrub_check.py passes; sponsor, compound and study codes follow doc-0003 and nothing names a real company, drug, trial or person
Definition of Done
- #1 just check
Implementation Plan
Implement the corrected design and frozen regulatory contracts, with disjoint file ownership for parallel work. Prove local runtime/rendered/mock-plan behaviour, integrate into just check, complete CodeRabbit review before landing code, and record live-only acceptance separately. No live deployment, image publication, new credential or CVE exception is authorized by this local implementation pass.
Implementation Notes
Local integration evidence: all five roles start with the production OTel preload under STUB_MODEL=1; child-process smoke proves shared trace propagation and bounded shutdown. Runtime tests prove guarded evidence only, fail-closed transport/untransformed IDs, actual SDK metadata export omits content while preserving usage, fabricated-statistic revision, and parts-based transforms. Root added a red/green zero-rate regression and verified Helm replicas=0 at zero requests/minute. Full isolated-PyYAML just check passed with 13 regulatory tests and all existing suites. No cloud calls.
CodeRabbit completed full79-file review and correction delta. Two original minor issues fixed: loadgen Deployment now runs until shutdown unless an explicit MAX_REQUESTS bounds a manual/test run, and new remote injection control follows warn-first rollout. Follow-on priority/short-circuit setting corrected as declarative config and tofu validate passed. Corrected full gate passes15 regulatory tests. Remaining terminal step is secret-backed CI scrub against the committed fixture names.
Post-delivery dependency validation: added the same legacy-peer-deps npm setting already used by agents and by the regulatory Docker install. Regenerated the lock with zero retained-package version changes and zero new entries, removing298 unused optional/peer packages. Clean npm ci and all15 regulatory tests pass; audit returns the existing eight-high SDK tree instead of33 findings including one critical. This is pure dependency/config cleanup, so CodeRabbit skipped under policy; runtime/clean-install/image validation used.
Final Summary
Built the separate five-role fictional regulatory app, fixture tools, corpus, continuous load generator and section/guard suites. Fail-closed preflight controls precede model calls and tool recording, including SDK parts transforms; capture-off export retains numeric scores and usage without content. Fifteen runtime/experiment tests and12 Terraform scenarios pass in the full gate. CodeRabbit full and delta reviews completed; all reported issues were resolved. CI39506282163 passed check, private scrub and ci-success for exact source commit3b05093ff1324d1380ac83c9aef8728e20327ee6. No real model calls or deployment were performed.