# AI evals for product managers

> A product manager's job in AI evals: define what good means, review real traces, label pass or fail, and report results. Compiled from practitioner and vendor guides.

Source: https://www.honeyhive.ai/learn/ai-evals-for-product-managers (updated 2026-11-13)

A product manager's job in AI evals is to decide what "good" means for the product, read real outputs against that standard, and turn the findings into pass/fail checks the team can run on every change. Engineers build the machinery. The judgment about whether an answer is acceptable is a product decision, and it is hard to hand off.

This guide pulls together what named practitioners and the model providers say about that work. For the engineering side, see [LLM evaluation](/learn/llm-evaluation).

## Start with success criteria

Anthropic's [guide to developing tests](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests) says success criteria should be specific, measurable, achievable and relevant. Its example of "specific" is replacing "good performance" with "accurate sentiment classification." It also notes that most use cases need evaluation along several criteria, and that relevance depends on the product: strong citation accuracy may matter for a medical app and less for a casual chatbot.

OpenAI's [evaluation guide](https://developers.openai.com/api/docs/guides/evaluation-best-practices) asks the same three questions: what is the objective, which data will test it, and how will you check that the criteria are met. It calls "vibe-based evals" an anti-pattern, defined as "waiting until you ship before implementing any evals," and recommends evaluating early and often.

## Read the outputs yourself

Hamel Husain and Shreya Shankar's [FAQ](https://hamel.dev/blog/posts/evals-faq/) calls error analysis "the most important activity in evals." Their suggested start is to annotate at least 30 traces yourself from a pool of around 100 diverse ones, reach at least 100 overall, and stop when new traces no longer reveal new failure modes. He also says outsourcing error analysis is usually a big mistake, because domain experts hold tacit knowledge a rubric cannot capture.

In a [field guide](https://hamel.dev/blog/posts/field-guide/) he adds that it is "impossible to completely determine evaluation criteria prior to human judging of LLM outputs," so criteria come out of the reading, not before it. See [error analysis for LLM apps](/learn/error-analysis-llm) for the method.

## Who decides what is good

Husain [recommends](https://hamel.dev/blog/posts/evals-faq/) "one domain expert who understands your users as your quality decision maker." A single expert removes annotation conflicts, and if you need five experts to judge one interaction, he says the product scope may be too broad. Larger or multi-domain organizations may need several annotators, in which case measure [inter-annotator agreement](/learn/inter-annotator-agreement).

His FAQ describes the PM's part this way: early on, PMs and engineers review together, with PMs catching product failures and engineers catching technical ones. Over time the team leans toward one decision maker, who may be a PM who understands user needs. He also suggests asking outcome questions, such as whether an appointment was booked, rather than whether a tool call succeeded.

## Label pass or fail

Both Husain and Eugene Yan favor binary labels. Husain [says](https://hamel.dev/blog/posts/evals-faq/) annotators on Likert scales "often default to middle values to avoid making hard decisions," and suggests separate binary sub-checks when you want finer progress. Yan [advises](https://eugeneyan.com/writing/product-evals/) focusing on pass/fail labels and recommends at least 50 to 100 failures out of 200 or more samples, so the set is not mostly passing cases.

Yan also [notes](https://eugeneyan.com/writing/product-evals/) that stakeholders sometimes ask for granular scores but eventually just ask for a threshold so they can report a pass/fail rate. A [golden dataset](/learn/golden-dataset) of labeled examples is the natural home for these labels, and a [rubric](/learn/llm-evaluation-rubric) keeps the definition of a pass written down.

## Automate with a judge, and check it

Once humans have labeled examples, an LLM judge can apply the same standard at scale. Yan [describes](https://eugeneyan.com/writing/product-evals/) splitting labeled data into development and test sets (for example 75% and 25%), using one evaluator per dimension, and measuring the evaluators on precision, recall and Cohen's Kappa. His benchmark is "human performance, not perfection." OpenAI's guide similarly says to use human feedback to calibrate automated scoring.

PMs do not have to build the judge, but they should ask how well it agrees with the labels before trusting its numbers. See [building an LLM judge](/learn/build-llm-judge-evaluator) and [LLM judge limitations](/learn/llm-judge-limitations).

## Skip generic metrics

Husain [writes](https://hamel.dev/blog/posts/evals-faq/) that ready-made generic metrics "waste time and create false confidence when you use them as quality measures." They can still help you find traces worth reading. In the field guide he calls them vanity metrics that don't correlate with real user problems. Metrics worth tracking come from your own failure categories.

## Report what changed

Husain says he has "rarely found dashboards to be helpful because they place the burden on the reader." For stakeholders he suggests an explanation of what changed, likely root causes and next steps. OpenAI's guide adds a habit that makes this possible: log everything as you develop, so you can mine logs for new eval cases. Pair offline test sets with [online evaluation](/learn/online-vs-offline-evaluation) on live traffic.

## A PM's checklist

| Step | PM's part | Source |
|---|---|---|
| Define success | Write specific, measurable criteria per dimension | [Anthropic](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests) |
| Review traces | Read 100 or more real outputs | [Husain](https://hamel.dev/blog/posts/evals-faq/) |
| Label | Pass/fail labels, enough failures to learn from | [Yan](https://eugeneyan.com/writing/product-evals/) |
| Calibrate | Ask how the judge agrees with human labels | [OpenAI](https://developers.openai.com/api/docs/guides/evaluation-best-practices) |
| Report | Narrative of changes, causes, next steps | [Husain](https://hamel.dev/blog/posts/evals-faq/) |

## FAQ

### What is a product manager's role in AI evals?

Define success, read real outputs and decide what counts as a failure. Hamel Husain's [FAQ](https://hamel.dev/blog/posts/evals-faq/) says PMs catch product failures such as unmet user expectations, while engineers catch technical issues, and recommends collaborating early before settling on one quality decision maker.

### Do product managers need to write code to run evals?

Not for the core work. Error analysis is reading outputs and labeling them, and Husain's [FAQ](https://hamel.dev/blog/posts/evals-faq/) says people without domain expertise can still help by spotting problems such as chatbots confused by conversational flow, and by collecting interactions and reviewing them regularly. Engineers typically build the harness that runs the checks.

### How many examples should a PM review first?

Husain [recommends](https://hamel.dev/blog/posts/evals-faq/) annotating at least 30 traces yourself from a pool of around 100 diverse ones, reviewing at least 100 overall, and continuing while new traces still reveal new failure modes.

### Should evals use a 1 to 5 score or pass/fail?

Both Husain and Eugene Yan favor pass/fail. Husain [notes](https://hamel.dev/blog/posts/evals-faq/) that annotators often default to middle values on Likert scales, and Yan [advises](https://eugeneyan.com/writing/product-evals/) focusing on binary labels.
