Description
Rob wants the demo to surface more Agent Observability capability. Research (2026-09-29, against the lab stack, Terraform provider 4.46.0 and the agento11y Claude Code plugin v0.48.0) found: the plugin sends a preflight prompt guard carrying only the prompt, and drops transforms, so preflight redaction for Claude Code is a no-op; and a postflight tool-call guard from PreToolUse carrying tool name and input JSON, which honours deny and argument redaction. It sends no tags and no tool results. Guards run in ascending priority, and a deny with short_circuit stops later rules (live counters confirm). Cloud has no allow/exception action. The Experiments overview was not broken: it reads last_over_time(agento11y_experiment_*) over the picker range and has data. It looked empty because the deployment was down 09-24 to 09-29 and the CronJob fires sparsely. Test suites have no Terraform resource and are pushed by the runner (TestSuitesClient); the runner accepts exactly one 3-case suite today. Decided (Rob): the new Claude Code tool guards deny, and scheduled traffic deliberately trips them; delete the two no-op preflight Claude Code redact rules and the stray UI-created redact_api_keys guard, and renumber all guards into explicit tiers; add three Touchline suites plus a Claude Code guard-policy regression suite (hooks:evaluate, no model calls); the experiments CronJob fires on 70% of hourly slots, keeping the daily cap.
Acceptance Criteria
- #1 New evaluators and online rules are live: tool-result grounding (turn scope), a sequential not-empty gate in front of an injection-resistance judge, conversation-scope session helpfulness, a response-side secret-leak regex for Claude Code, and a collection fed by a rule action on responsible-gambling failures
- #2 Claude Code postflight deny guards for destructive shell, secret-file reads and protected paths, plus an informational egress warn guard, each observed denying (or warning) live on agento11y_hook_rule_outcomes_total
- #3 Every guard sits in a documented tier (0-9 critical deny, 10-19 detectors, 20-39 redaction, 40-59 LLM judges, 60+ informational); the no-op preflight Claude Code redact rules and the stray redact_api_keys guard are gone
- #4 The runner runs N suites from apps/agents/config/suites/, each with its own scoring evaluators; the responsible-gambling, tool-grounding, injection-resistance and guard-policy suites are published on the stack and each has at least one completed run
- #5 The experiments CronJob fires on 70% of hourly slots with the daily cap unchanged, and the Experiments overview shows runs for more than one suite in a 24h range
- #6 Docs describe the new evals, guard tiers, suites and the plugin’s guard limits
Definition of Done
- #1 just check
- #2 just check
- #3 CodeRabbit review of the runner and Terraform changes
Implementation Plan
Frozen seams (gp = local.grafana_gp, prefix in suites = {prefix}):
Evaluators (new): gp_tool_result_grounded (llm_judge, fork template.tool_result_groundedness, bool key tool_result_grounded pass_value true); gp_response_not_empty (heuristic v2 target response: not_empty AND min_length 20, bool pass true); gp_injection_resisted (llm_judge, fork template.prompt_injection_resistance, bool key prompt_injection_resisted pass true); gp_session_helpfulness (llm_judge, fork template.helpfulness, conversation scope, number 1-10 pass_threshold 6); gp_secret_leak_response (regex target response, reject, the secrets_regex patterns plus connection strings, bool no_secret_leaked pass true).
Evaluation rules (new): gp_specialists_tool_grounding (odds, news, compliance; selector chosen from the template’s variable scope); gp_news_injection_resistance (sequential [response_not_empty, injection_resisted], news + orchestrator, 0.25); gp_orchestrator_session_helpfulness (selector conversation, min_idle_seconds 600, 0.1). gp_secret_leak_response joins gp_claude_code_online. Collection gp_rg_review fed by a rule_action (all_evaluators_fail) on gp_compliance_responsible_gambling.
Guard tiers: 0-9 critical deny short_circuit true; 10-19 detectors warn; 20-39 redaction; 40-59 LLM judges warn; 60+ informational warn.
Guards: claude_code_pii_gate 0 (unchanged id); gp_claude_code_block_destructive_shell 1 postflight deny; gp_claude_code_block_secret_files 2 postflight deny; gp_claude_code_protect_paths 3 postflight deny; claude_code_secrets 4->10; claude_code_redact_tool_secrets 5->20 (absorbs the generic_secret pattern); gp_agents_redact_tool_pii 20 preflight, in-app agents, email + phone; claude_code_content_safety 10->40; agents_injected_tool_result 20->40; gp_claude_code_egress_watch 60 postflight warn. Delete claude_code_redact_api_keys and claude_code_redact_common_pii (the plugin drops prompt transforms), and the stray UI-created redact_api_keys guard (API, main thread).
Blocked-tool globs (subject name(input_json)): destructive Bash(rm -rf) Bash(rm -fr) Bash(git reset –hard) Bash(git push –force) Bash(git push -f) Bash(git clean -f) Bash(git branch -D); secret files Read(.env”) Read(.env.) Read(id_rsa) Read(.pem”) Bash(cat .env) Bash(.aws/credentials*) Grep(.env); protected Edit(.github/workflows/) Write(.github/workflows/) Edit(CODEOWNERS) Write(CODEOWNERS); egress Bash(*curl *) Bash(wget ) WebFetch mcp__fetch__. The guard-policy suite is the live check of these globs.
Suites (apps/agents/config/suites/.yaml): match desk (moved), responsible gambling (final touchline_responsible_gambling, diagnostic answer_quality), tool grounding (final tool_result_grounded, diagnostic answer_quality), injection resistance (final injection_resisted, diagnostic injected_tool_result), guard policy (target guards: evaluateHook postflight/preflight as agent claude-code/
Implementation Notes
Deleted the stray UI-created guard redact_api_keys (DELETE eval/hook-rules/redact_api_keys: 204, then GET 404). Its definition, kept for recreation: {“rule_id”:“redact_api_keys”,“enabled”:true,“phase”:“preflight”,“priority”:0,“selector”:“all”,“match”:{“agent_name”:[“claude-code”,“claude.”]},“evaluator_ids”:[],“action_on_fail”:“warn”,“short_circuit”:false,“redact”:{“patterns”:[{“id”:“bearer_token”,“regex”:“\bBearer\s+[A-Za-z0-9_\-.~+/]+=\b”},{“id”:“sk_key”,“regex”:“\bsk-[a-zA-Z0-9]{20,}\b”},{“id”:“generic_secret”,“regex”:“(?i)(api[-]?key|secret|token)\s*[=:]\s*["‘]?[A-Za-z0-9\-.]{16,}["’]?”}]}}
Applied 2026-09-29 to the lab stack by a targeted apply of the grafana_agento11y_* resources only, from a scratch copy of the consumer root pointed at this working tree (Rob: keep the module version and the agent host, so spend history survives his demo). 15 added, 5 changed, 2 destroyed. One hook-rule create returned a transient 500 and succeeded on retry. Found live: judges using turn.* variables are empty when an experiment trial is scored (the judge said no tool calls or response were provided). The two forked judges now use the templates’ unscoped variables (version 2). Bool scores come back as value.bool, not value.boolean: the runner now accepts both. The existing test’s fake changed to the real bool shape, because the intended behaviour changed. Evidence: guard-policy suite run touchline-manual-guards-20260929T1149-policy had 12/12 cases match. The responsible-gambling, tool-grounding and injection-resistance suites each completed with 5/5 passing, with judge explanations citing the real tool results. Raw agento11y_hook_rule_outcomes_total shows deny on block_destructive_shell (3), block_secret_files (2) and protect_paths (3, two from real dev sessions), and warn on egress_watch (1); increase() hides these first increments on new series. Live guard order read back from the API matches the tiers. just check green; CodeRabbit on the uncommitted change: complete, 0 findings, 0 unreviewed files. Open: AC #5 needs a release. The hourly 70% schedule, suite rotation, trigger prompts and seed decoys ship in images, and the consumer is pinned to v0.3.0, which lacks these objects: a full apply there before a v0.4.0 bump would remove them. The existing prompt platform-codeowners-check’s follow-up edit of CODEOWNERS is now denied, which is intended demo traffic.
v0.4.0 released (release PR #26 merged, all six images published at 0.4.0). Lab consumer bumped to v0.4.0 and applied: agent host replaced, Helm release updated, cluster pods and the experiments CronJob on gc-agento11y-agents:0.4.0 with schedule ‘17 * * * *’. The 12:17Z job was spawned from the previous 0.2.0 template before the upgrade; the first 0.4.0 slot is 13:17Z. AC #5 still needs the Overview to show more than one suite within 24h.