Responsible AI Playbook for Enterprise Agents
From input guardrails, trajectory evaluation to session outcomes: playbook for regulated teams buil...
Traditional ML and RAG evaluation gave teams a familiar set of measures: accuracy and drift for models, then retrieval precision and recall, ROUGE, BLEU, faithfulness, and coherence for generated responses. Agentic systems require a broader measurement model.
Agents introduce new failure surfaces. They can choose the wrong tool, pass dangerous arguments, loop through failed calls, or lose context during a handoff. Some failures can sit behind an acceptable final response. Others produce a bad response whose cause becomes clear only when you inspect the trajectory. Evaluating agents therefore requires monitoring both execution and outcomes.
In our work with responsible AI, risk, and platform teams at large global enterprises across banking, insurance, and other regulated sectors, the recurring challenge is deciding what to measure and where each check belongs. This guide maps metrics for task resolution, tool use, handoffs, guardrails, and system reliability across CI, asynchronous production monitoring, and the synchronous request path.
Agents fail in the middle, not just at the end
A banking agent may select the wrong tool or pass arguments outside its authority. An insurance claims agent may retrieve the correct policy but make an unsupported coverage determination. An internal agent may complete the task only after looping through failed calls and consuming unnecessary time and tokens.
These failures require measurement at several levels. Response-level metrics assess individual calls. Trajectory metrics cover tool selection, argument validity, loops, and recovery. Inter-agent and conversational metrics cover handoffs, context preservation, task completion, and goal resolution. Guardrail and system metrics cover policy violations, unsafe actions, latency, cost, and reliability.
Coverage should match the workflow. A banking advisory agent and an internal research agent do not need the same controls. SR 26-2 explicitly excludes generative and agentic AI, but its risk-based principle is still useful: validation and monitoring should scale with purpose, exposure, materiality, and the cost of failure.
The registry below is a broad reference, not a checklist. Review traces, identify the failure modes relevant to the workflow, and enable the checks that cover them.
Metrics across the agent lifecycle
A conversation is a series of messages between participants (customer, staff, or AI agent) aimed at resolving an intent. It can contain one or more threads, each tied to an intent and an agent. Messages can run through guardrails whose actions range from masking PII to handing the conversation to a human.
Risk shows up before the model sees input, after each call, before the user sees output, across the full trajectory, at handoffs, and at session close. System reliability and cost sit under all of it. Evaluation levels describe how far you zoom (one call → one agent → multi-agent → session). Stages describe when the check runs in that lifecycle.
Evaluation levels
| Level | Scope | Primary evaluators |
|---|---|---|
| Response | Individual LLM call: single input/output pair | Code, LLM-as-Judge, Guardrails |
| Single agent | Full agent trajectory: tool calls, any exposed reasoning, final answer | LLM-as-Judge, Code |
| Inter-agent | Handoffs and coordination between agents in a thread | LLM-as-Judge, Hybrid |
| Conversational | End-to-end conversation across threads and intents | LLM-as-Judge, Hybrid, human oversight |
Evaluator types
| Type | Description | When to use |
|---|---|---|
| Code | Deterministic functions that score event attributes | Objective, reproducible checks (schema validation, length, overlap scores) |
| LLM-as-Judge | An LLM scores quality from a structured prompt | Subjective dimensions (coherence, faithfulness, tone) once calibrated against a human |
| Guardrail | Pre/post checks in the request path that can block or transform | Real-time safety and compliance (jailbreak, PII, advice) |
| OTEL | Metrics captured via OpenTelemetry instrumentation | System performance (latency, errors, throughput) |
| Hybrid | Automated scoring plus human-in-the-loop review | High-stakes decisions, annotation queues, edge cases |
For LLM-as-Judge metrics, prefer binary pass/fail criteria tied to a real failure mode over vague 1-5 scores. Before a judge becomes a CI or production gate, calibrate it against a domain expert: measure true positive and true negative rates on labeled examples, and correct for bias. An uncalibrated judge looks precise and still drifts.
Guardrails also tag an implementation approach: Non-LLM (regex, keywords, patterns, classic ML models), LLM, or Cascading (fast heuristic first, LLM only when ambiguous). If a metric says LLM-as-Judge under an input or output stage, treat it as async or review-path scoring unless it's on the critical path.
Risk changes at each stage, while system reliability affects the entire run.
Some risks appear at multiple stages. Toxicity, privacy, and profanity, for example, need separate checks at the input and output hook points.
Stage 1: Input guardrails
Hook: before the inbound message is forwarded to the agent.
Input guardrails execute on every inbound user message before it reaches the agent. They evaluate in priority order. If a guardrail triggers, the configured action fires (for example eject to a human, send a canned response, or mask content) and downstream processing may halt. High-consequence risks here usually belong on the synchronous path. We list metrics separately so you can map coverage and actions; in production, overlapping LLM checks such as Harmful Content, Off Topic, and Toxicity can share one structured model call when labels and actions stay distinct.
Guardrail checks
| Name | Detects / measures | Type of use | Evaluator type | Action on trigger |
|---|---|---|---|---|
| Harmful Content | Sexual, hateful, or violent content in inbound messages | Input | Guardrail (LLM) | Block + canned response |
| Jailbreak | Attempts to bypass safety mechanisms or alignment constraints | Input | Guardrail (Cascading) | Eject to human |
| Off Topic | Politics, sensitive issues, or illegal activities outside scope | Input | Guardrail (LLM) | Canned response + redirect |
| Vulnerability | Customer may be in a vulnerable circumstance requiring escalation | Input | Guardrail (Cascading) | Eject to human |
| Customer Complaint | Customer complaints (e.g. as defined in AU RG 271); map to your complaint rules | Input | Guardrail (Cascading) | Flag + route to complaints team |
| Profanity | Profane language in customer messages | Bidirectional | Guardrail (Non-LLM) | Mask in place |
| Privacy (Input) | PII before forwarding to the agent | Bidirectional | Guardrail (Non-LLM) | Mask PII in place |
| Competitor | Mentions of predefined competitors in customer text | Bidirectional | Guardrail (Non-LLM) | Flag for review |
| Language Check | Non-English text requiring language-specific routing | Bidirectional | Guardrail (Non-LLM) | Route to language-specific agent |
Input safety metrics
| Name | Detects / measures | Evaluator type | Required inputs |
|---|---|---|---|
| Prompt Injection Attack | Input techniques designed to override system instructions | LLM-as-Judge | Input message |
| Moderation Filter | Unsafe content categories via an external moderation API (e.g. Google, Azure, or OpenAI) | Code | Input message |
| Toxicity (Input) | Toxicity level of the inbound message on a 0 to 1 scale | LLM-as-Judge | Input message |
Stage 2: Response-level evaluation
Hook: server-side evaluator on the model event, runs after each individual LLM call.
Response-level metrics evaluate a single LLM call in isolation. They compare the generated output against ground truth, assess output quality, and verify structural correctness. These metrics run as server-side evaluators attached to the model event within the trace, so they can use that call's inputs, retrieved context, and references, not only the generated text.
Quality metrics
| Name | Detects / measures | Evaluator type | Required inputs |
|---|---|---|---|
| Answer Faithfulness | Response faithful to source documents without hallucination | LLM-as-Judge | Output, source documents |
| Answer Relevance | Response directly addresses the user query | LLM-as-Judge | Output, input query |
| Context Relevance | Retrieved context relevant to the query (RAG) | LLM-as-Judge | Input query, retrieved context |
| Context Precision | Proportion of retrieved documents that are relevant | Code | Retrieved docs, ground-truth relevant docs |
| Context Recall | Proportion of relevant documents successfully retrieved | Code | Retrieved docs, ground-truth relevant docs |
| Coherence | Logical flow and readability of the response | LLM-as-Judge | Output |
| Summary Quality | Summary captures key points without distortion | LLM-as-Judge | Output, source text |
| Format Adherence | Output follows prescribed format (JSON, markdown, template) | LLM-as-Judge | Output, format specification |
| Tone Appropriateness | Tone matches expected register (formal, empathetic, etc.) | LLM-as-Judge | Output, tone guidelines |
| G-Eval | Chain-of-thought scoring against custom criteria | LLM-as-Judge | Output, evaluation criteria |
Textual similarity metrics
| Name | Detects / measures | Evaluator type | Required inputs |
|---|---|---|---|
| Semantic Similarity | Cosine similarity between output and ground-truth embeddings (e.g. text-embedding-3-small) | Code | Output, ground truth |
| ROUGE-L | Longest-common-subsequence F1 vs reference text | Code | Output, ground truth |
| BLEU | N-gram overlap precision with brevity penalty vs reference | Code | Output, ground truth |
| Levenshtein Distance | Normalized edit distance as 0 to 1 similarity | Code | Output, ground truth |
| Response Length | Word count of model output (verbosity monitoring) | Code | Output |
| Flesch Reading Ease | Readability from sentence length and syllable count (0 to 100) | Code | Output |
| Keyword Assertion | Required or prohibited keywords present in output | Code | Output, keyword list |
Structural reliability metrics
| Name | Detects / measures | Evaluator type | Required inputs |
|---|---|---|---|
| JSON Schema Validation | Output conforms to expected JSON schema | Code | Output, JSON schema |
| SQL Parse Check | Generated SQL is syntactically valid and parseable | Code | Output (SQL string) |
| JSON Key Coverage | Proportion of expected keys present in JSON output | Code | Output, expected keys |
| Compilation Success | Generated code compiles successfully | Code | Output (code string) |
Stage 3: Output guardrails
Hook: after the agent produces a response, before delivery to the customer.
Output guardrails inspect the generated response before the user sees it. They enforce compliance, prevent hallucinated advice, and ensure grounded, safe responses. Actions below are example policy responses; adapt them to your product and jurisdiction.
Guardrail checks
| Name | Detects / measures | Type of use | Evaluator type | Action on trigger |
|---|---|---|---|---|
| Financial Advice | Financial product advice (e.g. AU Corporations Act s766B); map to your advice rules | Output | Guardrail (Cascading) | Block + canned disclaimer |
| Legal Advice | Response may constitute legal advice | Output | Guardrail (Cascading) | Block + canned disclaimer |
| Groundedness | RAG answer grounded in retrieved source documents | Output | Guardrail (Cascading) | Block + fallback response |
| Privacy (Output) | PII leaked into the agent response | Bidirectional | Guardrail (Non-LLM) | Mask PII in place |
| Profanity (Output) | Profane language in the agent response | Bidirectional | Guardrail (Non-LLM) | Mask in place |
| Competitor (Output) | Competitor names in the agent response | Bidirectional | Guardrail (Non-LLM) | Flag for review |
Output safety metrics
| Name | Detects / measures | Evaluator type | Required inputs |
|---|---|---|---|
| Toxicity (Output) | Toxicity level of the generated response on a 0 to 1 scale | LLM-as-Judge | Output |
| Moderation Filter (Output) | Unsafe categories via an external moderation API (e.g. Google, Azure, or OpenAI) | Code | Output |
| Policy Compliance | Adherence to organizational policies and guidelines | LLM-as-Judge | Output, policy document |
| Harm Avoidance | Response avoids potential harm to the customer | LLM-as-Judge | Output, context |
Groundedness here is the blocking twin of Answer Faithfulness in Stage 2: score the call asynchronously if you want; put a groundedness guard on the delivery path when an ungrounded answer is costly.
In banking, an output can be grounded and well-written while still crossing a jurisdiction- or product-specific advice boundary. Map AU examples (complaint handling, financial advice) to the laws, policies, products, and customer contexts that apply to your workflows.
Stage 4: Single-agent trajectory
Hook: after the full agent execution completes (chain / agent-run event).
Trajectory evaluation examines the observable action sequence of a single agent from intent receipt to final response: tool selection, intermediate steps, and loops. Some models or runtimes expose a reasoning trace, but current frontier APIs often return only a summary or keep the reasoning state opaque. Reasoning Trace Consistency applies only when the trace itself is exposed. Evaluating the observable trajectory still requires span-level visibility, not a score on the final message alone.
| Name | Detects / measures | Evaluator type | Required inputs |
|---|---|---|---|
| Intent Identification | Agent correctly identified customer intent from the input | LLM-as-Judge | Input, agent's classified intent, ground-truth intent |
| Correct Tool Use | Right tools selected with correct parameters | LLM-as-Judge | Agent trajectory (tool calls), expected tool calls |
| Agent Looping | Non-productive loops in tool calls or, when exposed, reasoning steps | LLM-as-Judge | Agent trajectory |
| Reasoning Trace Consistency | Whether an exposed reasoning trace supports the final answer and observed actions | LLM-as-Judge | Exposed reasoning trace, tool calls, final output |
| Plan Coverage | Execution covered all steps in the stated plan | LLM-as-Judge | Agent plan, execution trace |
| Trajectory Plan Faithfulness | Agent followed its stated plan without deviation | LLM-as-Judge | Agent plan, execution trace |
| Failure Recovery | Graceful recovery from tool errors or unexpected states | LLM-as-Judge | Agent trajectory (including error events) |
| Task Completion | Binary assessment of whether the agent resolved the stated intent | LLM-as-Judge | Agent trajectory, intent definition |
| Tool Correctness | Tool call outputs match expected schemas and return values | Code | Tool call inputs/outputs, expected schemas |
| Intent Resolution | Quality and completeness of resolving the customer intent | LLM-as-Judge | Agent trajectory, intent definition, final output |
Stage 5: Inter-agent evaluation
Hook: after agents in a thread complete, or on handoff events.
Inter-agent evaluation assesses communication, handoff quality, and coordination between multiple agents in one conversation thread. Relevant when conversations span multiple intents or an orchestrator delegates to specialists.
| Name | Detects / measures | Evaluator type | Required inputs |
|---|---|---|---|
| Handoff Quality | Context correctly preserved and communicated during agent-to-agent handoffs | LLM-as-Judge | Handoff messages, pre/post-handoff context |
| Context Preservation | Critical information from prior agent interactions retained across handoffs | LLM-as-Judge | Full thread context, individual agent contexts |
| Agent Trajectory Consistency | Multiple agents on related sub-tasks produce consistent, non-contradictory outputs | LLM-as-Judge | All agent outputs in thread |
| Escalation Appropriateness | Decision to escalate to another agent or human was appropriate | Hybrid | Escalation trigger, conversation context |
Stage 6: Conversational-level evaluation
Hook: at conversation close, or on a scheduled batch pass over the session.
Conversational-level evaluation assesses the end-to-end experience across all threads and turns. These metrics capture holistic quality that you cannot see at a single response or single-agent level.
| Name | Detects / measures | Evaluator type | Required inputs |
|---|---|---|---|
| Conversation Coherence | Logical flow and consistency across all turns | LLM-as-Judge | Full conversation transcript |
| Goal Achievement | Customer's original goal or intent ultimately achieved | LLM-as-Judge | Full conversation, original intent |
| Customer Sentiment Trend | Sentiment progression to identify satisfaction trajectory | LLM-as-Judge | Full conversation transcript |
| Conversation Efficiency | Turns-to-resolution and duration relative to intent complexity | Code | Conversation metadata |
| Context Coverage | Relevant knowledge sources consulted during conversation | LLM-as-Judge | Full conversation, available knowledge sources |
| Translation Fluency | Natural fluency of translations in multi-language conversations | LLM-as-Judge | Translated messages, source messages |
| Human Oversight Review | Structured human annotation of sampled conversations via annotation queues | Hybrid | Full conversation transcript |
System performance and reliability
Hook: OpenTelemetry auto-instrumentation on LLM and tool-call spans, captured continuously.
System performance metrics are captured automatically via OpenTelemetry. They feed real-time monitoring dashboards and CI/CD quality gates.
Performance metrics
| Name | Detects / measures | Evaluator type | Required inputs |
|---|---|---|---|
| Latency | End-to-end response time from request receipt to delivery (P50, P95, P99) | OTEL | Span timestamps |
| Time-to-First-Token (TTFT) | Time from request submission to the first generated token | Code | Streaming span events |
| Tokens per Second | LLM generation throughput in output tokens per second | Code | Token count, generation duration |
| Cost | Estimated cost per request from model, input/output token counts, and pricing tables | Code | Model name, token counts |
Reliability metrics
| Name | Detects / measures | Evaluator type | Required inputs |
|---|---|---|---|
| API Errors & Exceptions | Count and categorization of API errors, timeouts, and unhandled exceptions per span | OTEL | Span status, error attributes |
| Guardrail Trigger Rate | Percentage of messages that trigger each guardrail; use to detect drift and tune thresholds | Code | Guardrail event logs |
| Fallback Rate | Percentage of requests that fell back to a canned response or human handoff after guardrail triggers | Code | Orchestration / conversation logs |
Sustainability metrics
Hook: batch computation aggregated on a schedule from OTEL spans and infrastructure telemetry.
Sustainability metrics estimate the environmental impact of AI inference workloads. They are extrapolated from system performance data and infrastructure metadata.
| Name | Detects / measures | Evaluator type | Required inputs |
|---|---|---|---|
| Power Consumption | Estimated power draw (kWh) from GPU utilization, latency, and request volume | Code | Latency data, GPU metadata |
| Carbon Emissions | Estimated CO₂ equivalent from power consumption and regional grid carbon intensity | Code | Power consumption, grid region |
Guardrail configuration properties
Each guardrail needs a few properties in configuration:
| Property | Description |
|---|---|
| Priority | Evaluation order. Critical checks such as Jailbreak and Vulnerability should rank highest. |
| Trigger condition | Input, Output, or Bidirectional |
| Guardrail type | Non-LLM, LLM, or Cascading (fast heuristic first, LLM if ambiguous) |
| Action on trigger | Flag only, canned message, mask in place, eject to human, or terminate |
| Canned message | Pre-written, non-model text shown when the action needs a fixed reply |
Recommended guardrail priority order
The following order is a starting point for guardrail evaluation. Lower priority numbers execute first.
| Priority | Guardrail | Rationale |
|---|---|---|
| 1 | Jailbreak | Block adversarial inputs before any processing |
| 2 | Harmful Content | Block dangerous content immediately |
| 3 | Vulnerability | Detect at-risk customers for immediate escalation |
| 4 | Privacy | Mask PII before content reaches the LLM or logs |
| 5 | Profanity | Clean language before further processing |
| 6 | Customer Complaint | Route complaints to appropriate handling team |
| 7 | Off Topic | Redirect off-topic queries before agent processing |
| 8 | Language Check | Route non-English to appropriate agent |
| 9 | Competitor | Flag competitor mentions for awareness |
| 10 | Financial Advice | Block non-compliant financial advice in output |
| 11 | Legal Advice | Block non-compliant legal advice in output |
| 12 | Groundedness | Verify output grounding before delivery |
Priority alone is not enough. How a check is implemented, and whether it sits on the synchronous path, decides whether that order is usable in production.
Production, CI, drift, and human review
The same risk definitions connect production monitoring, CI, review, and the next release.
Real-time monitoring. System performance metrics (latency, TTFT, error rates) and guardrail trigger rates stream to dashboards via OpenTelemetry. Alerts fire when metrics breach defined thresholds (for example P95 latency exceeding 5 seconds, or jailbreak trigger rate exceeding 2%). Set the actual numbers from your workflow baseline and risk tolerance.
CI/CD quality gates. Before any agent, prompt, or model change is promoted to production, the evaluation pipeline runs response-level and agent-level metrics against a curated dataset of historical user journeys and synthetic test cases. Changes are blocked if any calibrated metric regresses beyond the defined tolerance.
Batch evaluation and drift detection. On a scheduled cadence (for example weekly), evaluate the full metric registry you care about against a representative sample of production conversations. Compare results to baseline benchmarks to detect quality drift, emerging safety risks, or shifts in customer intent distribution.
Annotation queues. Conversations flagged by guardrails, low-confidence evaluators, or random sampling are routed to human annotation queues. Annotators provide ground-truth labels that feed back into the evaluation dataset, closing the feedback loop for metric calibration and model improvement. Prefer binary pass/fail with short critiques before a judge becomes a CI or production gate.
Operationalizing with HoneyHive
HoneyHive is one way to wire those modes together. Use CI regression checks to catch metric degradation before release. After deployment, online evaluations score matching traces asynchronously, and alerts surface threshold breaches. Trajectory View helps you debug trajectory failures in long-running agents. Send ambiguous cases to annotation queues to calibrate judges and grow the evaluation dataset.
The HoneyHive skills and CLI support coding-agent workflows in Cursor, Claude Code, and similar tools. They can instrument the application, create evaluators, update datasets, debug alerts, and configure online scoring as part of the improvement loop.
Blocking, masking, and escalation still live in your orchestration layer. HoneyHive helps you trace, monitor, and improve your agents.
How to start
- Pick one high-risk workflow and document its lifecycle steps and map risks, available actions, and potential failures.
- Review a sample of real traces and list the observed failure modes. If traces are not yet available, use simulation to create synthetic traces.
- Map each priority risk to a lifecycle stage and metrics.
- Calibrate the first judges and guardrail thresholds against those traces. Include latency, cost, and error metrics for the controls themselves.
- Expand coverage after the first set agrees with human review on held-out data.
For runnable evaluators and monitoring patterns, see evaluator templates. To work through your agent's risk surface and connect controls to production alerts and CI gates faster, book a 30-minute session with our team.
