Standardizing AI Observability Before It Breaks: A Case Study on 73,000 Agent Schemas

Say you’re leading GenAI at one of the modern companies where every team is building agents. Over time, each team picks their favorite agent building framework — LangGraph, Google ADK, AWS Strands in the team that lives on Bedrock, a hand-rolled loop in the team that doesn't trust frameworks. Six months later you ask a simple question:

‍

Can we put the same evaluator, the same guardrails, and the same alerts on every agent we run?

‍

The answer, in most organizations today, is no.

‍

Not because the evaluators are hard to write, but because every framework emits telemetry in its own shape. An answer-faithfulness evaluator binds to "the last assistant message" — which lives at a different path in every framework. Span names differ: one framework calls an agent invocation invoke_agent agent_1, another just invoke_agent, so the step you want to evaluate isn't even addressable the same way twice. Trace hierarchies differ: tool calls are children of the LLM span in one framework and siblings in another. The same logical agent produces structurally different traces depending on who built it.

‍

The result is that the assets you most want to standardize — evaluator libraries, alert templates, guardrail configs, dashboards — must be rebuilt per team. The cost doesn't scale with the number of agents. It scales with the number of teams, times the number of frameworks, times the number of humans who have to agree on what a trace means. Standardization is the thread that weaves alignment across all three.

‍

This post is about what that problem looks like when you measure it. We analyzed 73,000 production agent schemas across three customers to quantify how fast agent telemetry actually changes — and what that churn means for every piece of reliability infrastructure bound to it.

‍

No consensus, three dialects

‍

The industry knows this is a problem, which is why there are now — depending on how you count — three competing attempts to solve it. The official OpenTelemetry GenAI semantic conventions, OpenInference from Arize, and OpenLLMetry from Traceloop each cover 20+ framework instrumentations. They are not interoperable without normalization processors in between.

‍

Here is the same question — where is the model's reply, and which model produced it? — answered in each dialect:

Three conventions, three shapes, and this is the easy attribute. It gets worse in the corners: the official OTel spec types its core input/output attributes as "any (structured JSON)" — so a platform team writing a mapping doesn't know whether it works until a span actually arrives.

‍

Conventions are converging, but the process could take years: The following companies all ship native support for the official OTel GenAI convention:

‍

Microsoft — Semantic Kernel, Azure OpenAI, Agent Framework
AWS — Strands, Bedrock AgentCore
Google — ADK

‍

This signal from big tech is the strongest one the industry has produced yet.

‍

But the official conventions are still marked in-development, they are not yet expressive enough to fully replace either alternative, and — a trap worth naming — "has OTel support" is not the same as "follows the GenAI conventions." Several popular frameworks (Haystack, Agno, Vercel AI SDK, LiteLLM) advertise OTel support while emitting custom attribute names, which delivers the plumbing without the standardization.

‍

Different teams emit telemetry in different shapes

‍

TL;DR there’s no consensus in the industry, but enterprises are building agents faster than conventions are stabilizing.

‍

At HoneyHive, we deal with this first hand and ingest the full spectrum of agent traces. So, we did a study on anonymyzed schemas from three of our customers to dig deeper.

‍

The study: 73,000 schemas from 3 production customers

‍

Agents emit a plethora of fields as they work — JSONs with inputs, outputs, config, metadata, user properties, etc. HoneyHive catalogs and indexes each of these fields, along with when it was first observed, when it was last observed, and how often.

‍

This catalog of an agent’s schemas represents the dictionary and vocabulary of the agent. It is the observable surface of the agent’s behavior, and thus the complete interface for monitoring and evaluating it.

‍

Alerts fire on field values. Guardrails check fields. Evaluators map fields into their inputs. Experiments compare fields across runs. When a field path changes, everything bound to it silently breaks.

‍

Methodology

We pulled the anonymized schema indices of three production customers — 73,000 discovered field schemas in total:

‍

‍A Global 2000 financial services company: dozens of projects, many teams onboarding in parallel, multiple frameworks in one organization
A browser agent startup: one product, deeply iterated for over two years
An AI-native e-commerce startup: one multilingual production agent, multiple LLM providers

‍

Raw discovered schemas are noisy, so we cleaned before measuring. We excluded the observability SDK's own system fields, collapsed array indices baked into path names, and collapsed dynamic identifiers — UUIDs, ObjectIDs, hex digests — that applications had baked into field paths. We also dropped experimentation exhaust: fields that lived less than a day, which made up 40–60% of distinct paths at every customer while carrying under half a percent of observed activity.

‍

What survives cleaning is the meaningful schema: roughly 17,000 fields across the three customers that real traffic depended on for more than a day. On those, we measured two things: Schema Lifespan (from first to last observation) and Schema Churn (fields born and dying per month, against the actively-used population).

‍

Finding 1: the median field lives two weeks to two months

Schema Lifespan Distribution for 3 Agent Profiles

‍

The median meaningful field lived 15 days at the e-commerce customer, 17 days at the browser-automation customer, and 68 days at the enterprise RAG customer. Even the long tail is short: third quartiles sit at roughly two to four months.

‍

The Enterprise RAG agent's fields live about 4× longer than the single-team products' — its first quartile (23 days) exceeds the other two customers' medians. That gap is the signature of process: multi-team production rollouts produce durable, contract-like fields, while single-team iteration produces fields that live exactly as long as the feature experiment they belong to.

‍

Every alert, evaluator mapping, and guardrail in your stack binds to specific field paths. If the median path is replaced within weeks, there is pressure to update every artifact during the same time span. Not doing this breaks your agent monitoring and evals.

‍

Finding 2: schema churn events are organizational events

Next, we wanted to understand what drives schema churn. We created a plot of 3 values:

‍

Schemas births: how many schemas showed up for the first time in that month?
Schema deaths: how many schemas never showed up again after that month?
Active schemas: how many schemas were observed that month?

‍

The plot reveals the engineering org’s story clearly.

‍

Global 2000 financial services enterprise: onboarding waves and a framework migration

Active Schemas for an Enterprise RAG Agent

‍

The enterprise chart is a story of teams arriving. Three projects start instrumenting in Month 1; around Month 9, two more teams finished their integrations in consecutive months and landed roughly a thousand fields between them, ramping the actively-used population to its peak of about 2,100 fields. Then, at Month 11, a refactor: one team retired a params object with over a thousand dynamic keys and replaced it with structured output and feedback fields — hundreds of deaths and births in the same month — while AWS Strands convention fields appeared in the index for the first time. By Month 13 the pattern inverts: new projects are still starting, but six pilot projects are decommissioned, and their entire schemas expire at once.

‍

E-commerce agent: a year of quiet, then two upheavals

Active Schemas for an E-commerce Recommender Agent

‍

The e-commerce chart shows the single-team version of the same story. For the first year (Months 1–13) the schema barely moves: quiet, escalating feature churn. Then a schema expansion adds 1,600 fields, an old product surface is refactored out, and churn accelerates until Month 25, when one massive in-place reshape lands — thousands of fields born and dying in a single month while the agent itself keeps running. Two months later, visible from metadata alone, comes a framework adoption: over 1,300 HoneyHive and OTel GenAI convention fields appearing in a single week.

‍

Browser automation agent: slow build-up, one sharp peak, wind-down

Active Schemas for a Browser Automation Agent

‍

The browser-automation customer shows the third shape. A single project instruments in Month 1, a second joins around Month 8, and the schema grows slowly for a year and a half. Then Month 19 concentrates almost all of the lifetime churn into one event: two projects onboard at the same time as a major refactor, and the births and deaths spike together before the population settles back down.

‍

Mapping schema churn to the engineering roadmap

‍

Two signatures show up in all three charts:

‍

Net growth in active schemas is a team onboarding. More active fields means the semantic range of the agents is expanding — more agents, features, or tooling being deployed.

‍

Paired births and deaths are a refactor or framework adoption. The RAG agent's Month-11 refactor and the e-commerce Month-25 reshape look identical from the outside: heavy churn while the population barely moves. This is the highest-risk event in any of these charts — every artifact bound to the old paths broke that month. And the adoption events are legible down to the week: AWS Strands fields appear at the enterprise customer in a specific month; HoneyHive conventions arrive at the e-commerce customer in one week. You can reconstruct an organization's instrumentation roadmap without reading a line of its code — and every one of those events was a schema migration some platform team paid for by hand.

‍

What this means if you run multiple teams

‍

A major schema change is a proxy for an agent re-architecture. When the telemetry reshapes, every artifact bound to the old schema — alerts, guardrails, evaluator mappings, experiment baselines — must be revisited. Miss one and it binds to dead paths, silently.

‍

Attribution is ambiguous exactly when it matters. When production evaluators start failing, was it a framework upgrade, a vendor mapping change, or a team refactor? Three schemas are in motion at all times, and the debugging happens while alerting is down.

‍

The cost multiplies across teams. Within one team, churn is an annoyance. Across ten teams with different frameworks and refactor cadences, it is a standing tax — and every team's churn is a private migration only that team can debug.

‍

Churn itself is not the problem — the median field lives weeks because refactors are how agents improve. The conclusion is not "stop changing your schema." It is: churn must happen against a stable interface.

‍

The path to standardization

‍

Adopt the OTel GenAI semantic conventions. OpenTelemetry is the standard the rest of software observability already runs on, and the official GenAI conventions inherit that ecosystem: SDKs in every language, the Collector and its processing pipeline, and native ingestion in the major backends. The agent-framework vendors are converging on it too — Microsoft (Semantic Kernel, Agent Framework), AWS (Strands, Bedrock AgentCore), and Google (ADK) all ship native support and sit in the OTel GenAI SIG. The conventions are still in development, but a dialect no one else speaks is a worse position.

‍

Treat schemas as first-class citizens. An agent's schema is its operational and observable surface, and it is coupled to everything you build on top: evaluators, instrumentation, alerts, dashboards, datasets. So manage it like an interface — version it, refactor it deliberately, and make the migration of every coupled artifact a line item in each agent refactor and framework adoption. Evaluator mappings, alert bindings, and dataset maintenance belong inside the refactor plan, not in the incident channel afterward. The data above says that bill comes due every few weeks somewhere in your org.

‍

This is also where we have put our effort at HoneyHive: audited mappings across OpenInference, OTel GenAI, and OpenLLMetry so mixed-framework estates resolve to one view; published attribute stability so teams know which paths are safe to bind to; ingestion-time transforms as an escape hatch when drift breaks a binding; and schema churn surfaced as a first-class signal instead of a silent alert.

‍

The schemas your agents emit are already churning — at the median, on a clock of two weeks to two months. Every evaluator, alert, and guardrail you deploy is bound to that moving surface, and more teams and agents only spin the clock faster. Conventions make the churn survivable; treating the schema as a first-class interface makes it manageable.

About the author:

Sunny Bakhda

Member of Technical Staff

Standardizing AI Observability Before It Breaks: A Case Study on 73,000 Agent Schemas

About the author:

Sunny Bakhda

Join our monthly newsletter