Responsible AI Playbook for Agentic AI

Guides

Mohammed Sanjeed

Forward Deployed Engineer

From input guardrails, trajectory evaluation to session outcomes: playbook for regulated teams buil...

Traditional ML and RAG evaluation gave teams a familiar set of measures: accuracy and drift for models, then retrieval precision and recall, ROUGE, BLEU, faithfulness, and coherence for generated responses. Agentic systems require a broader measurement model.

Agents introduce new failure surfaces. They can choose the wrong tool, pass dangerous arguments, loop through failed calls, or lose context during a handoff. Some failures can sit behind an acceptable final response. Others produce a bad response whose cause becomes clear only when you inspect the trajectory. Evaluating agents therefore requires monitoring both execution and outcomes.

In our work with responsible AI, risk, and platform teams at large global enterprises across banking, insurance, and other regulated sectors, the recurring challenge is deciding what to measure and where each check belongs. This guide maps metrics for task resolution, tool use, handoffs, guardrails, and system reliability across CI, asynchronous production monitoring, and the synchronous request path.

Agents fail in the middle, not just at the end

A banking agent may select the wrong tool or pass arguments outside its authority. An insurance claims agent may retrieve the correct policy but make an unsupported coverage determination. An internal agent may complete the task only after looping through failed calls and consuming unnecessary time and tokens.

These failures require measurement at several levels. Response-level metrics assess individual calls. Trajectory metrics cover tool selection, argument validity, loops, and recovery. Inter-agent and conversational metrics cover handoffs, context preservation, task completion, and goal resolution. Guardrail and system metrics cover policy violations, unsafe actions, latency, cost, and reliability.

Coverage should match the workflow. A banking advisory agent and an internal research agent do not need the same controls. SR 26-2 explicitly excludes generative and agentic AI, but its risk-based principle is still useful: validation and monitoring should scale with purpose, exposure, materiality, and the cost of failure.

The registry below is a broad reference, not a checklist. Review traces, identify the failure modes relevant to the workflow, and enable the checks that cover them.

Metrics across the agent lifecycle

A conversation is a series of messages between participants (customer, staff, or AI agent) aimed at resolving an intent. It can contain one or more threads, each tied to an intent and an agent. Messages can run through guardrails whose actions range from masking PII to handing the conversation to a human.

Risk shows up before the model sees input, after each call, before the user sees output, across the full trajectory, at handoffs, and at session close. System reliability and cost sit under all of it. Evaluation levels describe how far you zoom (one call → one agent → multi-agent → session). Stages describe when the check runs in that lifecycle.

Evaluation levels

Level	Scope	Primary evaluators
Response	Individual LLM call: single input/output pair	Code, LLM-as-Judge, Guardrails
Single agent	Full agent trajectory: tool calls, any exposed reasoning, final answer	LLM-as-Judge, Code
Inter-agent	Handoffs and coordination between agents in a thread	LLM-as-Judge, Hybrid
Conversational	End-to-end conversation across threads and intents	LLM-as-Judge, Hybrid, human oversight

‍

Evaluator types

Type	Description	When to use
Code	Deterministic functions that score event attributes	Objective, reproducible checks (schema validation, length, overlap scores)
LLM-as-Judge	An LLM scores quality from a structured prompt	Subjective dimensions (coherence, faithfulness, tone) once calibrated against a human
Guardrail	Pre/post checks in the request path that can block or transform	Real-time safety and compliance (jailbreak, PII, advice)
OTEL	Metrics captured via OpenTelemetry instrumentation	System performance (latency, errors, throughput)
Hybrid	Automated scoring plus human-in-the-loop review	High-stakes decisions, annotation queues, edge cases

‍

For LLM-as-Judge metrics, prefer binary pass/fail criteria tied to a real failure mode over vague 1-5 scores. Before a judge becomes a CI or production gate, calibrate it against a domain expert: measure true positive and true negative rates on labeled examples, and correct for bias. An uncalibrated judge looks precise and still drifts.

Guardrails also tag an implementation approach: Non-LLM (regex, keywords, patterns, classic ML models), LLM, or Cascading (fast heuristic first, LLM only when ambiguous). If a metric says LLM-as-Judge under an input or output stage, treat it as async or review-path scoring unless it's on the critical path.

Risk surfaces across the agent execution lifecycle — *Risk changes at each stage, while system reliability affects the entire run.*

Some risks appear at multiple stages. Toxicity, privacy, and profanity, for example, need separate checks at the input and output hook points.

‍

Stage 1: Input guardrails

Hook: before the inbound message is forwarded to the agent.

Input guardrails execute on every inbound user message before it reaches the agent. They evaluate in priority order. If a guardrail triggers, the configured action fires (for example eject to a human, send a canned response, or mask content) and downstream processing may halt. High-consequence risks here usually belong on the synchronous path. We list metrics separately so you can map coverage and actions; in production, overlapping LLM checks such as Harmful Content, Off Topic, and Toxicity can share one structured model call when labels and actions stay distinct.

Guardrail checks

Name	Detects / measures	Type of use	Evaluator type	Action on trigger
Harmful Content	Sexual, hateful, or violent content in inbound messages	Input	Guardrail (LLM)	Block + canned response
Jailbreak	Attempts to bypass safety mechanisms or alignment constraints	Input	Guardrail (Cascading)	Eject to human
Off Topic	Politics, sensitive issues, or illegal activities outside scope	Input	Guardrail (LLM)	Canned response + redirect
Vulnerability	Customer may be in a vulnerable circumstance requiring escalation	Input	Guardrail (Cascading)	Eject to human
Customer Complaint	Customer complaints (e.g. as defined in AU RG 271); map to your complaint rules	Input	Guardrail (Cascading)	Flag + route to complaints team
Profanity	Profane language in customer messages	Bidirectional	Guardrail (Non-LLM)	Mask in place
Privacy (Input)	PII before forwarding to the agent	Bidirectional	Guardrail (Non-LLM)	Mask PII in place
Competitor	Mentions of predefined competitors in customer text	Bidirectional	Guardrail (Non-LLM)	Flag for review
Language Check	Non-English text requiring language-specific routing	Bidirectional	Guardrail (Non-LLM)	Route to language-specific agent

‍

Input safety metrics

Name	Detects / measures	Evaluator type	Required inputs
Prompt Injection Attack	Input techniques designed to override system instructions	LLM-as-Judge	Input message
Moderation Filter	Unsafe content categories via an external moderation API (e.g. Google, Azure, or OpenAI)	Code	Input message
Toxicity (Input)	Toxicity level of the inbound message on a 0 to 1 scale	LLM-as-Judge	Input message

‍

Stage 2: Response-level evaluation

Hook: server-side evaluator on the model event, runs after each individual LLM call.

Response-level metrics evaluate a single LLM call in isolation. They compare the generated output against ground truth, assess output quality, and verify structural correctness. These metrics run as server-side evaluators attached to the model event within the trace, so they can use that call's inputs, retrieved context, and references, not only the generated text.

Quality metrics

Name	Detects / measures	Evaluator type	Required inputs
Answer Faithfulness	Response faithful to source documents without hallucination	LLM-as-Judge	Output, source documents
Answer Relevance	Response directly addresses the user query	LLM-as-Judge	Output, input query
Context Relevance	Retrieved context relevant to the query (RAG)	LLM-as-Judge	Input query, retrieved context
Context Precision	Proportion of retrieved documents that are relevant	Code	Retrieved docs, ground-truth relevant docs
Context Recall	Proportion of relevant documents successfully retrieved	Code	Retrieved docs, ground-truth relevant docs
Coherence	Logical flow and readability of the response	LLM-as-Judge	Output
Summary Quality	Summary captures key points without distortion	LLM-as-Judge	Output, source text
Format Adherence	Output follows prescribed format (JSON, markdown, template)	LLM-as-Judge	Output, format specification
Tone Appropriateness	Tone matches expected register (formal, empathetic, etc.)	LLM-as-Judge	Output, tone guidelines
G-Eval	Chain-of-thought scoring against custom criteria	LLM-as-Judge	Output, evaluation criteria

‍

Textual similarity metrics

Name	Detects / measures	Evaluator type	Required inputs
Semantic Similarity	Cosine similarity between output and ground-truth embeddings (e.g. text-embedding-3-small)	Code	Output, ground truth
ROUGE-L	Longest-common-subsequence F1 vs reference text	Code	Output, ground truth
BLEU	N-gram overlap precision with brevity penalty vs reference	Code	Output, ground truth
Levenshtein Distance	Normalized edit distance as 0 to 1 similarity	Code	Output, ground truth
Response Length	Word count of model output (verbosity monitoring)	Code	Output
Flesch Reading Ease	Readability from sentence length and syllable count (0 to 100)	Code	Output
Keyword Assertion	Required or prohibited keywords present in output	Code	Output, keyword list

‍

Structural reliability metrics

Name	Detects / measures	Evaluator type	Required inputs
JSON Schema Validation	Output conforms to expected JSON schema	Code	Output, JSON schema
SQL Parse Check	Generated SQL is syntactically valid and parseable	Code	Output (SQL string)
JSON Key Coverage	Proportion of expected keys present in JSON output	Code	Output, expected keys
Compilation Success	Generated code compiles successfully	Code	Output (code string)

‍

Stage 3: Output guardrails

Hook: after the agent produces a response, before delivery to the customer.

Output guardrails inspect the generated response before the user sees it. They enforce compliance, prevent hallucinated advice, and ensure grounded, safe responses. Actions below are example policy responses; adapt them to your product and jurisdiction.

Guardrail checks

Name	Detects / measures	Type of use	Evaluator type	Action on trigger
Financial Advice	Financial product advice (e.g. AU Corporations Act s766B); map to your advice rules	Output	Guardrail (Cascading)	Block + canned disclaimer
Legal Advice	Response may constitute legal advice	Output	Guardrail (Cascading)	Block + canned disclaimer
Groundedness	RAG answer grounded in retrieved source documents	Output	Guardrail (Cascading)	Block + fallback response
Privacy (Output)	PII leaked into the agent response	Bidirectional	Guardrail (Non-LLM)	Mask PII in place
Profanity (Output)	Profane language in the agent response	Bidirectional	Guardrail (Non-LLM)	Mask in place
Competitor (Output)	Competitor names in the agent response	Bidirectional	Guardrail (Non-LLM)	Flag for review

‍

Output safety metrics

Name	Detects / measures	Evaluator type	Required inputs
Toxicity (Output)	Toxicity level of the generated response on a 0 to 1 scale	LLM-as-Judge	Output
Moderation Filter (Output)	Unsafe categories via an external moderation API (e.g. Google, Azure, or OpenAI)	Code	Output
Policy Compliance	Adherence to organizational policies and guidelines	LLM-as-Judge	Output, policy document
Harm Avoidance	Response avoids potential harm to the customer	LLM-as-Judge	Output, context

‍

Groundedness here is the blocking twin of Answer Faithfulness in Stage 2: score the call asynchronously if you want; put a groundedness guard on the delivery path when an ungrounded answer is costly.

In banking, an output can be grounded and well-written while still crossing a jurisdiction- or product-specific advice boundary. Map AU examples (complaint handling, financial advice) to the laws, policies, products, and customer contexts that apply to your workflows.

‍

Stage 4: Single-agent trajectory

Hook: after the full agent execution completes (chain / agent-run event).

Trajectory evaluation examines the observable action sequence of a single agent from intent receipt to final response: tool selection, intermediate steps, and loops. Some models or runtimes expose a reasoning trace, but current frontier APIs often return only a summary or keep the reasoning state opaque. Reasoning Trace Consistency applies only when the trace itself is exposed. Evaluating the observable trajectory still requires span-level visibility, not a score on the final message alone.

Name	Detects / measures	Evaluator type	Required inputs
Intent Identification	Agent correctly identified customer intent from the input	LLM-as-Judge	Input, agent's classified intent, ground-truth intent
Correct Tool Use	Right tools selected with correct parameters	LLM-as-Judge	Agent trajectory (tool calls), expected tool calls
Agent Looping	Non-productive loops in tool calls or, when exposed, reasoning steps	LLM-as-Judge	Agent trajectory
Reasoning Trace Consistency	Whether an exposed reasoning trace supports the final answer and observed actions	LLM-as-Judge	Exposed reasoning trace, tool calls, final output
Plan Coverage	Execution covered all steps in the stated plan	LLM-as-Judge	Agent plan, execution trace
Trajectory Plan Faithfulness	Agent followed its stated plan without deviation	LLM-as-Judge	Agent plan, execution trace
Failure Recovery	Graceful recovery from tool errors or unexpected states	LLM-as-Judge	Agent trajectory (including error events)
Task Completion	Binary assessment of whether the agent resolved the stated intent	LLM-as-Judge	Agent trajectory, intent definition
Tool Correctness	Tool call outputs match expected schemas and return values	Code	Tool call inputs/outputs, expected schemas
Intent Resolution	Quality and completeness of resolving the customer intent	LLM-as-Judge	Agent trajectory, intent definition, final output

‍

Stage 5: Inter-agent evaluation

Hook: after agents in a thread complete, or on handoff events.

Inter-agent evaluation assesses communication, handoff quality, and coordination between multiple agents in one conversation thread. Relevant when conversations span multiple intents or an orchestrator delegates to specialists.

Name	Detects / measures	Evaluator type	Required inputs
Handoff Quality	Context correctly preserved and communicated during agent-to-agent handoffs	LLM-as-Judge	Handoff messages, pre/post-handoff context
Context Preservation	Critical information from prior agent interactions retained across handoffs	LLM-as-Judge	Full thread context, individual agent contexts
Agent Trajectory Consistency	Multiple agents on related sub-tasks produce consistent, non-contradictory outputs	LLM-as-Judge	All agent outputs in thread
Escalation Appropriateness	Decision to escalate to another agent or human was appropriate	Hybrid	Escalation trigger, conversation context

‍

Stage 6: Conversational-level evaluation

Hook: at conversation close, or on a scheduled batch pass over the session.

Conversational-level evaluation assesses the end-to-end experience across all threads and turns. These metrics capture holistic quality that you cannot see at a single response or single-agent level.

Name	Detects / measures	Evaluator type	Required inputs
Conversation Coherence	Logical flow and consistency across all turns	LLM-as-Judge	Full conversation transcript
Goal Achievement	Customer's original goal or intent ultimately achieved	LLM-as-Judge	Full conversation, original intent
Customer Sentiment Trend	Sentiment progression to identify satisfaction trajectory	LLM-as-Judge	Full conversation transcript
Conversation Efficiency	Turns-to-resolution and duration relative to intent complexity	Code	Conversation metadata
Context Coverage	Relevant knowledge sources consulted during conversation	LLM-as-Judge	Full conversation, available knowledge sources
Translation Fluency	Natural fluency of translations in multi-language conversations	LLM-as-Judge	Translated messages, source messages
Human Oversight Review	Structured human annotation of sampled conversations via annotation queues	Hybrid	Full conversation transcript

‍

System performance and reliability

Hook: OpenTelemetry auto-instrumentation on LLM and tool-call spans, captured continuously.

System performance metrics are captured automatically via OpenTelemetry. They feed real-time monitoring dashboards and CI/CD quality gates.

Performance metrics

Name	Detects / measures	Evaluator type	Required inputs
Latency	End-to-end response time from request receipt to delivery (P50, P95, P99)	OTEL	Span timestamps
Time-to-First-Token (TTFT)	Time from request submission to the first generated token	Code	Streaming span events
Tokens per Second	LLM generation throughput in output tokens per second	Code	Token count, generation duration
Cost	Estimated cost per request from model, input/output token counts, and pricing tables	Code	Model name, token counts

‍

Reliability metrics

Name	Detects / measures	Evaluator type	Required inputs
API Errors & Exceptions	Count and categorization of API errors, timeouts, and unhandled exceptions per span	OTEL	Span status, error attributes
Guardrail Trigger Rate	Percentage of messages that trigger each guardrail; use to detect drift and tune thresholds	Code	Guardrail event logs
Fallback Rate	Percentage of requests that fell back to a canned response or human handoff after guardrail triggers	Code	Orchestration / conversation logs

‍

Sustainability metrics

Hook: batch computation aggregated on a schedule from OTEL spans and infrastructure telemetry.

Sustainability metrics estimate the environmental impact of AI inference workloads. They are extrapolated from system performance data and infrastructure metadata.

Name	Detects / measures	Evaluator type	Required inputs
Power Consumption	Estimated power draw (kWh) from GPU utilization, latency, and request volume	Code	Latency data, GPU metadata
Carbon Emissions	Estimated CO₂ equivalent from power consumption and regional grid carbon intensity	Code	Power consumption, grid region

Guardrail configuration properties

Each guardrail needs a few properties in configuration:

Property	Description
Priority	Evaluation order. Critical checks such as Jailbreak and Vulnerability should rank highest.
Trigger condition	Input, Output, or Bidirectional
Guardrail type	Non-LLM, LLM, or Cascading (fast heuristic first, LLM if ambiguous)
Action on trigger	Flag only, canned message, mask in place, eject to human, or terminate
Canned message	Pre-written, non-model text shown when the action needs a fixed reply

Recommended guardrail priority order

The following order is a starting point for guardrail evaluation. Lower priority numbers execute first.

Priority	Guardrail	Rationale
1	Jailbreak	Block adversarial inputs before any processing
2	Harmful Content	Block dangerous content immediately
3	Vulnerability	Detect at-risk customers for immediate escalation
4	Privacy	Mask PII before content reaches the LLM or logs
5	Profanity	Clean language before further processing
6	Customer Complaint	Route complaints to appropriate handling team
7	Off Topic	Redirect off-topic queries before agent processing
8	Language Check	Route non-English to appropriate agent
9	Competitor	Flag competitor mentions for awareness
10	Financial Advice	Block non-compliant financial advice in output
11	Legal Advice	Block non-compliant legal advice in output
12	Groundedness	Verify output grounding before delivery

‍

Priority alone is not enough. How a check is implemented, and whether it sits on the synchronous path, decides whether that order is usable in production.

Production, CI, drift, and human review

Operating loop for measuring and mitigating agent risk — *The same risk definitions connect production monitoring, CI, review, and the next release.*

Real-time monitoring. System performance metrics (latency, TTFT, error rates) and guardrail trigger rates stream to dashboards via OpenTelemetry. Alerts fire when metrics breach defined thresholds (for example P95 latency exceeding 5 seconds, or jailbreak trigger rate exceeding 2%). Set the actual numbers from your workflow baseline and risk tolerance.

CI/CD quality gates. Before any agent, prompt, or model change is promoted to production, the evaluation pipeline runs response-level and agent-level metrics against a curated dataset of historical user journeys and synthetic test cases. Changes are blocked if any calibrated metric regresses beyond the defined tolerance.

Batch evaluation and drift detection. On a scheduled cadence (for example weekly), evaluate the full metric registry you care about against a representative sample of production conversations. Compare results to baseline benchmarks to detect quality drift, emerging safety risks, or shifts in customer intent distribution.

Annotation queues. Conversations flagged by guardrails, low-confidence evaluators, or random sampling are routed to human annotation queues. Annotators provide ground-truth labels that feed back into the evaluation dataset, closing the feedback loop for metric calibration and model improvement. Prefer binary pass/fail with short critiques before a judge becomes a CI or production gate.

Operationalizing with HoneyHive

HoneyHive is one way to wire those modes together. Use CI regression checks to catch metric degradation before release. After deployment, online evaluations score matching traces asynchronously, and alerts surface threshold breaches. Trajectory View helps you debug trajectory failures in long-running agents. Send ambiguous cases to annotation queues to calibrate judges and grow the evaluation dataset.

The HoneyHive skills and CLI support coding-agent workflows in Cursor, Claude Code, and similar tools. They can instrument the application, create evaluators, update datasets, debug alerts, and configure online scoring as part of the improvement loop.

Blocking, masking, and escalation still live in your orchestration layer. HoneyHive helps you trace, monitor, and improve your agents.

How to start

Pick one high-risk workflow and document its lifecycle steps and map risks, available actions, and potential failures.
Review a sample of real traces and list the observed failure modes. If traces are not yet available, use simulation to create synthetic traces.
Map each priority risk to a lifecycle stage and metrics.
Calibrate the first judges and guardrail thresholds against those traces. Include latency, cost, and error metrics for the controls themselves.
Expand coverage after the first set agrees with human review on held-out data.

‍

For runnable evaluators and monitoring patterns, see evaluator templates. To work through your agent's risk surface and connect controls to production alerts and CI gates faster, book a 30-minute session with our team.