The Evolution of Observability: From Monoliths to AI Agents
.png)
Software and ML Observability struggle to scale to the world of AI agents. How did we get here? What’s the way forward?
Read Time: 7 minutes
Observability in the deterministic software era used to be easy. Remember when fixing your code meant adding a few print statements and calling it a day? Those were simpler times. Like, really simple. Fast forward to today, AI is eating software and we're in a world where systems have gotten so complex and non-deterministic that even explaining them requires a deep breath and possibly a drink. Understanding how we got here requires us to take a look at the history of observability.
The Three Ages of Observability
Software 1.0: Deterministic Software - The Era of Readable but Complex Execution
1970s to present day
In the beginning, we had monoliths. Applications were complex but readable - you could trace through the code line by line and understand exactly what was happening. The challenge wasn't understanding the code; it was knowing where to look when things went wrong.
The traditional "three pillars" approach emerged to track signals for these systems:
- Logs: For console messages
- Metrics: For system health monitoring
- Traces: For request flow tracking
But as systems became more distributed and trace volume exploded, this approach showed its limitations:
# Traditional debugging approach
log.error("API call failed") # But which service? Why?
metrics.increment("api_errors") # But what kind of error?
trace.span("api_call") # But what was the context?
The industry around 2017 responded with wide-event data models that capture data via distributed tracing. Companies like Honeycomb pioneered the Observability 2.0 approach, allowing teams to ask arbitrary questions about their system's behavior and run rapid signal correlation. These systems had to build for high cardinality (i.e. lots of structured signals) and parsing through the noise to root cause issues in production.
Sadly, the exponential curve of tech evolution doesn’t care about your observability stack. While the ecosystem was learning how to do observability 2.0, the deep neural net revolution went into full swing.
Software 2.0: Deep NNs - The Era of Unreadable but Simple Execution
2010s to present day
Then came the era of deep neural networks, what Andrej Karpathy termed "Software 2.0". These systems were fundamentally different:
# Software 1.0: Explicit logic
if user_input.contains_black_listed_words():
flag_content()
# Software 2.0: Learned behavior
model_output = flagging_model.predict(user_input)
# But why did it make this decision?
The execution was simpler (just matrix multiplications) but completely unreadable. Even when you could see something was wrong, fixing it was challenging because you couldn't simply modify an if-statement - you had to retrain the model. A lot of analysis is required after the fact to do the retraining and to validate if your retrained model indeed solved the root cause.
This era demanded new observability patterns:
- Distribution shift detection
- Robust data exporting to warehouses
- Multimodal data support
Since we relied on a single model to make a prediction, we didn’t need to worry about tracing complex executions here. Logging the inputs and outputs for the model are sufficient to get complete telemetry.
To improve, our models had to become more aware of what’s happening in the real world with constant retraining and active learning. The focus was on allowing practitioners to export their logs to data warehouses for further enrichment, analysis, and ultimately retraining. It’s an extremely iterative process. We’re still learning how to interpret and debug neural nets and fix adversarial cases.
Nonetheless, as exponentials do, the speed of tech evolution increased. While the ecosystem was finally learning how to do ML observability the right way, the Generative AI revolution went into full swing.
Software 3.0: Agents - The Era of Non-Deterministic and Complex Execution
2021 to present day
Just when we thought we had neural nets a bit under control, software 1.0 and 2.0 patterns merged to form AI agents — a new kind of semi-deterministic software that uses deep neural networks for reasoning and code for taking actions. We have entered the era of "Software 3.0". The execution is semi-readable (i.e. in natural language) but incredibly complex and non-deterministic.
The multi-step execution complexity demands a Software 1.0 approach to observability. The inherent non-determinism demands a Software 2.0 approach to observability. How do we reconcile these approaches, all while keeping analytics real-time?
Why Existing Solutions Fall Short
When I first built Codex-CLI at Microsoft, I thought pre-existing observability approaches might be sufficient. I was wrong. Three critical challenges emerged:
1. The Distributed Explosion
LLM apps went from prompts to RAG to agents to multi-agent systems in 1 year. Along side that apps went from running in a single service to being split across multiple microservices.
# 'Simple' LLM app flow
user_request ->
context_retrieval ->
embedding_generation ->
semantic_search ->
prompt_construction ->
model_call ->
response_validation ->
agent_state_update ->
potential_tool_calls ->
more_model_calls ->
response_synthesis
Traditional ML observability tools weren't built for this level of execution complexity. Each step needs monitoring, correlation, and debugging capabilities. So, just on this basis, we realized whatever the end observability system is, it will require a distributed tracing engine like OpenTelemetry to power telemetry.
2. The Unstructured Data Problem
We realized that if we are ingesting unstructured data, we also need a robust enrichment layer for extracting structured insights out of this raw data. Examples of this could include anything from measuring hallucinations in agent responses to extracting sentiment, tone, etc. This diverges from traditional software observability where logs were structured in nature and failure modes were deterministic, so enrichment tooling wasn’t ever a big need.
Moreover, context windows for models meanwhile happily scaled from 10k to 1M tokens in just 1 year. Systems engineers had sadly optimized observability systems to work with small log sizes (<1MB) containing structured signals, meanwhile each LLM request today takes up multiple MBs of storage. The Software 1.0 approach again began to fail us.
typical_model_call = {
"context": "2MB of unstructured document data",
"messages": "~10KB of thinking",
"tool_calls": "A nested JSON that goes on forever",
}
3. The Schema Flexibility Challenge
LLM interaction modalities went from completion to chat to multi-modal to realtime web sockets in 1.5 years as well.
OLAP databases don’t scale well with deeply nested data. Our customers require robust real-time aggregations to do effective analysis at scale, and no, a JSON blob column doesn’t solve that problem.
For example, if you look at OpenAI’s response schema, responses can nest up to 5 levels or more for tool calls.
{
"role": "assistant",
"content": null,
"function_call": {
"name": "get_current_weather",
"arguments": {
"request": {
"location": {
"city": {
"name": "Boston",
"details": {
"state": "MA",
"coordinates": {
"latitude": {
"degrees": 42,
"minutes": 21,
"direction": "N"
},
"longitude": {
"degrees": 71,
"minutes": 3,
"direction": "W"
}
},
"metadata": {
"timezone": {
"name": "America/New_York"
}
}
}
}
}
}
}
}
}
Imagine supporting a query that wants to find every time a sub-agent in your system asked for the weather in Boston. The nested JOIN query you would need with a OLAP db using JSON columns would give you the answer in >5 minutes. We need this to be real-time!
So, what do you need from your AI Observability?
"I need to iterate on my AI system without being constrained by rigid schemas"
- Developers need the freedom to experiment with different AI patterns and architectures without being blocked by data model limitations
- They should be able to quickly prototype new features and capture new types of data without database migrations
- The technical solution? A schema-free data model that adapts to both structured and unstructured data
"When something goes wrong, I need to see the complete picture"
- Developers need to understand exactly how their AI system arrived at a particular output
- They should be able to trace issues across the entire interaction chain, from user input through each model call and tool use
- The technical solution? A distributed tracing engine that captures the full execution path
"I need to catch AI-specific failure modes that aren't just about code errors"
- Developers need to identify semantic issues like hallucinations, incorrect reasoning, or inappropriate responses
- They should be able to evaluate the quality and correctness of AI outputs beyond just syntax
- The technical solution? An unstructured data enrichment layer that helps detect semantic issues
"I need to know immediately if something's wrong with my AI system"
- Developers need real-time visibility into their AI system's health and performance
- They should be able to quickly investigate issues and set up meaningful alerts
- The technical solution? Real-time aggregation capabilities across all system data
This infrastructure enables an Evaluation-Driven Development approach, where developers can:
- Continuously measure and improve their AI system's performance
- Build domain-specific evaluation frameworks
- Set up proactive monitoring to catch issues before users do
By focusing on these developer needs, we create an observability system that truly supports the AI development workflow while handling the unique challenges of AI systems.
The HoneyHive Solution: A New Architecture for AI Observability
We started with ClickHouse because we needed a columnar store for ingesting our wide events.
The key optimization for our ClickHouse instance was to minimize the amount of data being loaded into memory to ensure sub-second queries for millions of records.
- We optimized the database schema to segregate the heaviest fields from the lighter ones cleanly, so that whenever light queries are run, we don’t end up processing the giant blobs of text that LLMs produce.
- To ensure deeply nested fields can be aggregated easily, we made sure not to give each field its own column. We utilized a schema store (something product observability tools do well) to access just the value that is needed from each record.
- To support our auto-aggregated properties, we do both write-time and read-time aggregations to make queries for those properties fast.
Data Ingestion
OpenTelemetry acts as our distributed tracing engine for picking up all AI-related execution steps in our client’s service.
After mapping the OpenTelemetry spans to our unique data model, we pass the JSON to our writing service.
At ingestion, we also
- Populate a schema store to track the types of different fields
- Deconstruct the JSONs into a flat key-value pair list for storage.
Data Enrichment
To rapidly analyze unstructured data and ensure scalability, we made our database write-only using Merge Trees in ClickHouse.
We pair the database with a scaling compute service which runs user-defined code asynchronously to constantly update the stored records. The enrichments enable useful workflows downstream for evaluation and fine-tuning, and also speed up debugging by quickly detecting semantic issues at scale.
Bringing together these key improvements will enable rapid exploration of data on a system trajectory level. This is what’s needed for the long future of building agents.
Looking Ahead
As we stand at the frontier of AI observability, a few things are clear:
- The complexity of AI systems will continue to grow.
- Defining semantic failures is only going to become trickier and more iterative.
- The need for sophisticated, AI-aware observability will only increase.
Observability is just the beginning. Our end vision is to power self-improving AI agents. Models that excel at data science (MLEBench) and software engineering (SWEBench) are right around the corner. It is obvious that one day an AI Engineer Agent will appear that can iterate on AI applications. We need to provide that agent the deepest observability and evaluation tooling possible.
Evaluations is still a critical bottleneck in the journey to perfect observability. Evaluation best practices are still in their early days. We have talked about this issue in the past, and think scaling some of the current best practices to the world of autonomous agents requires a new set of eval tooling (more on that soon!).
A flexible observability platform to iterate on your evaluators and architectures rapidly is the best solution in the interim.
Sources
- Software 2.0 - Andrej Karpathy's seminal article on neural networks as a new programming paradigm
- https://aws.amazon.com/what-is/distributed-tracing/
- Observability 2.0 - Charity Majors' writings on modern observability
- A Practitioner's Guide to Wide Events - Deep dive into wide-event data models
- Introduction to OpenTelemetry - Introduction to OpenTelemetry
- Data Lakehouse Whitepaper
- Interpreting and Debugging Neural Networks - Current approaches to neural network interpretability
- Adversarial Cases in Neural Networks - Understanding and fixing adversarial examples in ML
- ClickHouse MergeTree Documentation - Technical details on ClickHouse's MergeTree engine
- HoneyHive Documentation - Official documentation covering our data model and implementation details
- Evaluating AI Applications Slides - HoneyHive talk on how to evaluate AI applications
- https://hamel.dev/blog/posts/llm-judge/
- https://github.com/openai/mle-bench/
- https://www.swebench.com/
