Description
The main pharma-facing story is controls on generated content: grounding, readability, patient-identifier redaction and unsupported-claim detection. Add evaluators, hook rules and evaluation rules for the regulatory agents with the regulatory prefix, gated on regulatory. Mind the repo gotchas: guard priority tiers (deny rules below 10), parallel rule creation can partially fail (apply with -parallelism=1 and read back).
Acceptance Criteria
- #1 Evaluators exist for citation and number grounding, reading grade, unsupported claims and review outcome, all prefixed with regulatory_name
- #2 A preflight guard redacts subject and investigator identifiers in tool results and a second guard flags injected source notes
- #3 Evaluation rules score live drafter and reviewer conversations and produce scores visible in the dashboard
- #4 Objects are created only when regulatory is selected and Touchline’s evaluators are unaffected
- #5 Evaluator and rule ids follow the ids frozen in doc-0003; evaluators cover citation and number grounding, reading grade, unsupported claims and review outcome, prefixed with regulatory_name
- #6 A preflight guard redacts subject and investigator identifiers in tool results and a second guard flags injected source notes; deny rules stay below priority 10
- #7 Evaluation rules score live drafter and reviewer conversations; a guard-policy experiment suite exists, with the runner extended for preflight tool-result redaction
- #8 Objects are created only when regulatory is selected; shared evaluators and Touchline’s are untouched; rules are applied with -parallelism=1 and read back
Definition of Done
- #1 just check
Implementation Plan
Implement the corrected design and frozen regulatory contracts, with disjoint file ownership for parallel work. Prove local runtime/rendered/mock-plan behaviour, integrate into just check, complete CodeRabbit review before landing code, and record live-only acceptance separately. No live deployment, image publication, new credential or CVE exception is authorized by this local implementation pass.
Implementation Notes
Local controls and both suites complete. Root integration fixed consumption of SDK parts-based transforms before recording/model input. Runner dry-run defaults, bounded sequential trials, scheduled Lease, explicit preflight tool-result cases and stored numeric score readback are tested. Terraform validates; no live rule apply, readback or scores. Resume after regulatory image publication and an authorized target deployment, applying new rules serially and reading back actual objects before retrying.
CodeRabbit corrections implemented: remote source-note control starts warn at detector priority10 with short_circuit=false so identifier redaction still runs; app known-marker rejection remains fail closed. Guard-policy expectations now require allow plus a failed evaluation of the exact matching rule, rather than claiming remote denial. Regression rejects allow with no matching failed evaluation. Promotion to remote deny is deferred until reviewed warning evidence. Live scoring/apply/readback remain the resume boundary after image availability and authorized deployment.
Continuing live acceptance now that image publication is available. Static preflight found a real experiment SDK coercion bug: recordIO(object) exported [object Object]. Fixed with JSON serialization and real SDK transport regression spanning trial creation, generation export, evaluation, stored score readback and finalization. No paid evaluation occurred during this regression.
The serialized-source-evidence regression now passes through actual SDK trial/create/export/evaluate/score/finalize HTTP transport. Final public repository gate and CodeRabbit review pass after the corrective changes. Live readback is next against the configured isolated consumer.
Live continuation: serial Terraform apply created controls successfully, and product-route readbacks confirm both guards, expected priorities10/20, warn-first/no short-circuit and online drafter/reviewer rules. Stored rule-score readback returns actual numeric outcomes and judge usage, including passing citation/unsupported-claim/review-outcome scores. Bounded injection app request escalates with0model usage. Guard-policy experiment Job running with backoff0/deadline180seconds; normal app requests exposed a separate fenced reviewer-JSON parsing defect, now being fixed before further paid calls. Public note contains no private target identifiers.
Final Summary
Live serial apply and product-route readback verified all regulatory controls and sample-rate1 drafter/reviewer rules. Stored online numeric scores and judge usage exist for both roles; dashboard pass/fail metrics are ingested. Actual guard-policy experiment regulatory-52f8a4ae-4e21-4eea-bf90-d03eca269245 has three stored passed final scores for redaction, matched warn-first source-note detection and benign evidence. Injected app request escalates with zero model usage. Source evidence serialization and SDK transforms are covered by real-transport regressions. Live testing found fenced reviewer JSON was falsely escalated; strict single-fence parser correction has red/green coverage, full19-test regulatory gate and CodeRabbit zero findings. Capture-off/snapshots/teardown remain on the end-to-end task.