# How to do error analysis for LLM apps

> Error analysis is reading real LLM traces, noting what went wrong, grouping the notes into failure modes and counting them to decide what to fix.

Source: https://www.honeyhive.ai/learn/error-analysis-llm (updated 2026-11-05)

Error analysis for an LLM app means reading real traces, writing a short note on what went wrong in each, grouping the notes into named failure modes and counting how often each one happens. The counts tell you what to fix first and which [evals](/learn/llm-evaluation) are worth writing. Hamel Husain and Shreya Shankar [call it](https://hamel.dev/blog/posts/evals-faq/) "the most important activity in evals", and Husain's [field guide](https://hamel.dev/blog/posts/field-guide/) calls it "the single most valuable activity in AI development".

The method is borrowed from qualitative research. It needs no tooling beyond a place to read traces (Husain [writes](https://hamel.dev/blog/posts/field-guide/) that "a spreadsheet is better than nothing"), and it replaces guessing at metrics with a ranked list taken from your own data.

## The steps

| Step | What you do | Source guidance |
|---|---|---|
| 1. Collect traces | Gather representative traces of real use. If you have none, [generate synthetic inputs](/learn/synthetic-test-data-for-evals) to start. | [Husain and Shankar](https://hamel.dev/blog/posts/evals-faq/) |
| 2. Open code | Read each trace and write a free-form note on the first failure you see. | [Husain and Shankar](https://hamel.dev/blog/posts/evals-faq/) |
| 3. Axial code | Group the notes into a failure taxonomy and count failures per category. | [Husain and Shankar](https://hamel.dev/blog/posts/evals-faq/) |
| 4. Refine | Have a model cluster the traces and look for more instances of your categories, then accept or reject what it finds. Stop at saturation. | [Husain and Shankar](https://hamel.dev/blog/posts/evals-faq/) |

### Collect traces

Husain and Shankar [suggest](https://hamel.dev/blog/posts/evals-faq/) starting with about 100 diverse traces. A trace is the full record of one run, so you need [tracing](/learn/llm-observability) in place before you can do this on production traffic. OpenAI's guide says to [log everything](https://developers.openai.com/api/docs/guides/evaluation-best-practices) as you develop so you can mine the logs for eval cases.

When there are more traces than you can read, pick them on purpose. The FAQ [lists](https://hamel.dev/blog/posts/evals-faq/why-is-error-analysis-so-important-in-llm-evals-and-how-is-it-performed.html) clustering, sorting by user feedback and sorting by likely failure patterns as ways to sample.

### Open code: write what you see

Open coding is free-form note-taking, which the FAQ compares to [journaling](https://hamel.dev/blog/posts/evals-faq/). Write the note before you decide on a category. Husain and Shankar also warn against starting from a rubric: [reviewers can become so focused on checking each item](https://hamel.dev/blog/posts/evals-faq/) that they overlook problems outside the rubric.

When you start, note only the first failure in a trace. Errors [compound](https://hamel.dev/blog/posts/evals-faq/), and for multi-turn conversations the FAQ says to [annotate only the first failure](https://hamel.dev/blog/posts/evals-faq/) at first, because downstream failures often cascade from it.

Binary labels are the suggested start. Husain and Shankar [write](https://hamel.dev/blog/posts/evals-faq/) that "binary evaluations force clearer thinking and more consistent labeling."

### Axial code: group and count

Axial coding turns the notes into categories. The FAQ describes the result as a [failure taxonomy](https://hamel.dev/blog/posts/evals-faq/) with a count per category, and calls this the most important step. The count is the point. In Husain's field guide, the team at NurtureBoss [counted the frequency of each issue](https://hamel.dev/blog/posts/field-guide/) after labeling conversations, and found that three issues accounted for over 60% of all problems. They had started with a spreadsheet where each row was a conversation.

### Refine until saturation

Keep reviewing until, in the FAQ's words, [new traces stop revealing failure modes or changing existing ones](https://hamel.dev/blog/posts/evals-faq/). This is called theoretical saturation. If you are still learning past 100 traces, keep going.

## Who reads the traces

Husain and Shankar [recommend](https://hamel.dev/blog/posts/evals-faq/) one domain expert as a "benevolent dictator" for most small and medium teams, because a single expert eliminates annotation conflicts. Engineers and product managers can work together at the start. If you do need several reviewers, [inter-annotator agreement](/learn/inter-annotator-agreement) measures how consistently they label.

## Where a model can help

Read first, automate later. The FAQ says to [always read the raw traces yourself](https://hamel.dev/blog/posts/evals-faq/what-parts-of-evals-can-be-automated-with-llms.html) at the start, and to use an LLM for first-pass grouping only after you have open coded 30 to 50 traces. It adds that LLM-generated groupings need your review. In the main workflow, you annotate at least [30 traces yourself](https://hamel.dev/blog/posts/evals-faq/) before looking at an agent's suggestions.

## Error analysis for agents

An agent run can have many steps, so the first failure matters more. For [agentic workflows](/learn/ai-agent-evaluation) Husain and Shankar say to [record the first upstream failure](https://hamel.dev/blog/posts/evals-faq/how-do-i-evaluate-agentic-workflows.html). They describe a transition failure matrix: rows are the last successful state, columns are where the first failure occurred, and the counts show where failures cluster. After that, they move to step-level checks such as tool choice, parameter extraction, error handling and context retention.

Their advice for multi-step chains is that [early-stage improvements have more impact](https://hamel.dev/blog/posts/evals-faq/) because errors cascade. For more on what to measure per step, see [agent observability](/learn/agent-observability) and [testing AI agents](/learn/testing-ai-agents).

## Check that the failure is the app's

A failed trace is not always a model failure. Anthropic [says](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) the transcript shows whether the agent made a genuine mistake or whether the graders rejected a valid solution, and that with frontier models a 0% pass rate across many trials is most often a signal of a broken task, not an incapable agent. It lists grading bugs, harness constraints and ambiguity as causes of low scores with good agent behavior.

## Decide what to do with each failure mode

Not every category needs an evaluator. The FAQ [says](https://hamel.dev/blog/posts/evals-faq/) some issues may need an engineering or design fix instead. It also advises starting with error analysis [before considering a model switch](https://hamel.dev/blog/posts/evals-faq/).

| Failure mode looks like | Likely next step | Source |
|---|---|---|
| A clear, repeatable bug in prompts, tools or code | Fix it directly; an automated check is optional | [Husain and Shankar](https://hamel.dev/blog/posts/evals-faq/) |
| A subjective or recurring quality problem | Write an evaluator for it and calibrate it against your human labels | [OpenAI](https://developers.openai.com/api/docs/guides/evaluation-best-practices) |
| Graders reject good outputs, or the task is ambiguous | Fix the eval, not the agent | [Anthropic](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) |
| The model seems wrong for the job | Check the failure counts before switching models | [Husain and Shankar](https://hamel.dev/blog/posts/evals-faq/) |

OpenAI's guide says to [use human feedback to calibrate](https://developers.openai.com/api/docs/guides/evaluation-best-practices) automated scoring, and lists "vibe-based evals" as an anti-pattern. Your labelled traces are that human feedback.

## Keep doing it

Husain and Shankar [advise](https://hamel.dev/blog/posts/evals-faq/why-is-error-analysis-so-important-in-llm-evals-and-how-is-it-performed.html) revisiting the process regularly. Anthropic describes manual transcript review as time-intensive and not scalable, with reviewer fatigue as a risk, and suggests [sampling transcripts to read weekly](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents). A small, regular sample keeps the taxonomy current as the app and its users change.

## Doing it in HoneyHive

HoneyHive's [Traces page](https://docs.honeyhive.ai/v2/tracing/ui-flows.md) lets you inspect sessions and filter events. You can [filter sessions](https://docs.honeyhive.ai/v2/datasets/dataset-curation.md) by evaluator scores, user feedback such as thumbs down, and metadata, then add them to a dataset. [Annotation queues](https://docs.honeyhive.ai/v2/evaluation/annotation-queues.md) collect events for human review, either by hand or with automation rules, and reviewers fill in fields defined by [human evaluators](https://docs.honeyhive.ai/v2/evaluators/human.md). A spreadsheet works too; Husain's guide says it is [better than nothing](https://hamel.dev/blog/posts/field-guide/).

## FAQ

### What is error analysis for LLM apps?

It is the practice of reading real traces, writing a note on each failure, grouping the notes into categories and counting them. Hamel Husain and Shreya Shankar call it [the most important activity in evals](https://hamel.dev/blog/posts/evals-faq/) because it shows which failures are worth writing evals for.

### What is the difference between open coding and axial coding?

Open coding is writing free-form notes on traces, which Husain and Shankar compare to [journaling](https://hamel.dev/blog/posts/evals-faq/). Axial coding groups those notes into a failure taxonomy and counts the failures in each category.

### How many traces should you review?

Husain and Shankar suggest a working pool of [about 100 diverse traces](https://hamel.dev/blog/posts/evals-faq/), with at least the first 30 annotated by you. Keep going until new traces stop revealing new failure modes.

### Can an LLM do the error analysis for you?

Partly. The same FAQ says to [read raw traces yourself first](https://hamel.dev/blog/posts/evals-faq/what-parts-of-evals-can-be-automated-with-llms.html) and that LLM-generated groupings need your review, so use a model to cluster and search, not to replace the reading.