Observability Data vs Actionable Agent Signals

Observability was built around a loop that assumes a person stands at the center of every failure: an anomaly fires, an alert reaches an engineer, the engineer opens a dashboard, forms a hypothesis, adds logging, redeploys, and confirms the fix. AI agents have broken every link in that chain because the loop itself was never built for a machine to run it alone. Lightrun's 2026 analysis spells out the design intent behind each piece of that infrastructure. Dashboards are built for human reading: visual, aggregated, arranged so a person can interpret them at a glance. Alerts are tuned to human attention spans, filtered to avoid wearing out an on-call engineer and batched so they arrive in a readable rhythm. Distributed traces are laid out so a person can walk a request from one hop to the next and reason about where the latency or the error crept in.

An assumption underlies how software behaves: running the same input twice produces the same output, and a 200 response means the request did what it was supposed to do. A stack trace, under that assumption, gives an engineer enough to go on. AI agents violate both halves of that contract. The same prompt can send an agent down different tool-call paths depending on the run, and a 200 can sit on top of an answer that is confidently wrong. Confident AI's 2026 guide puts a name on the gap with a single case: a support agent nicknamed "Garry" refunded the wrong invoice, returned a clean 200, produced flat latency charts, and logged "success" on every line that mattered. The failure sat three steps deep inside one tool call that fired with the wrong argument, and it never touched a dashboard anyone on the team was watching. Observability had nothing to say about it, because nothing in the system had ever been told to watch for it. A failure that produces no alert never enters the loop. It simply does not exist to a system that was never configured to report it.

How agents fail in ways that observability's data model cannot see

Agent failures spread across a chain of decisions instead of sitting at a single point, and the data model behind observability, discrete log lines and scalar metrics, was never built to put that chain back together. Traditional application performance monitoring assumes a deterministic service: the same request produces the same response, and a failure traces back to one call. Agents don't work that way. What an agent does at step 7 depends on everything that happened at steps 1 through 6, so the behavior is state-dependent in a way no APM dashboard was designed to track. MLflow's 2026 developer guide draws the line clearly: traditional monitoring tells an engineer the service is down, while agent observability has to tell them that step 4 of a multi-step chain produced a wrong tool output. Those are different debugging problems, and they need different kinds of evidence.

The sharpest version of this problem is what might be called causal invisibility. A tool call fails silently at step 3, that failure quietly corrupts the agent's reasoning through step 8, and call-level monitoring does not record it. The only way to see it is a trace of the full session. Latitude's developer comparison from March 2026 states that the bug at turn 3 produces garbage at turn 8, and all an engineer has to work with is a timestamp and a bad final output, not a record of what happened in between. Silent tool failures, where the HTTP status comes back clean but the argument passed to the tool was wrong, rank among the most common ways agents fail in production. Telemetry only ever describes what a system was pre-configured to report. It has no way to capture a variable value, a conditional branch, or a dependency response that nobody thought to instrument in advance, so a metric that was never defined will never fire, no matter how badly the agent behaves. Lightrun's analysis names four reasons this gap keeps widening under agentic conditions: unfamiliar code ships faster than engineers can build context around it; AI-generated code carries more bugs than human-written code, many of them hitting edge cases that throw no telemetry signal at all; agents have no codebase familiarity to draw on when forming a hypothesis; and the sheer volume of code agents produce has outpaced any team's capacity to validate it by hand.

Once you see that traditional telemetry can't catch failures living inside a single multi-step session, you need to surface the full causal chain: every decision and tool output across an agent's execution. Tools built specifically for agent transcript analysis, Vex's among them, can reconstruct what happened across steps 1 through 8, revealing the silent corruptions and tool failures that call-level monitoring was never designed to catch.

What the volume of agentic output does to human-paced investigation cycles

Even a perfect data model would eventually run into the sheer volume of what agents produce, and that volume is already turning the human investigation cycle itself into the bottleneck, not whatever tooling sits behind it. An NBER study looked at a large sample of GitHub developers and found that autonomous coding agents substantially increased code volume compared to the pre-agent baseline. Stripe's internal coding agents, nicknamed Minions, are responsible for more than a thousand pull requests merged every week, written start to finish by agents, as first reported by First AI Movers. Human review capacity, not the agents' capacity to generate code, is now the limiting factor in how fast that code can ship safely.

The same volume problem occurs inside the observability stack itself. A single agent run throws off dozens of spans, and a fleet of agents throws off that volume continuously, around the clock. Pushing all of that raw telemetry into an observability backend recreates the same cost and noise problem at the infrastructure level that telemetry pipelines were built to solve. The 2026 Observability Forecast notes that if pipelines filter out low-value data and route what remains to the right storage tier, they can cut observability spend substantially. But cost isn't the only tension. The DORA 2025 report, cited by buildmvpfast.com, found that higher AI adoption correlates with increased delivery instability alongside increased throughput: more output does not automatically mean better output. Red Hat's CI/CD practice for agentic AI shows that tension at work directly. Pre-merge, it runs a set of predefined conversations plus one freshly generated conversation against the proposed upgrade before it accepts it, and at night, it runs a more thorough pass of 20 conversations across multiple configurations. Those nightly runs are what established that a shared large language model, not any single code change, was the actual source of a run of failures, a conclusion no dashboard on its own would have surfaced.

As investigation cycles become the bottleneck, the infrastructure underneath has to shift from dashboards built for human reading toward signals a machine can act on directly. Platforms built specifically for agent workflows, rather than monitoring systems adapted after the fact, can route those signals straight into the next execution step without waiting on a human to read a chart first.

What an actionable signal is

Telemetry is pre-instrumented, retrospective, and aggregated for a person to read later. It stays passive, describing only what the system happened to be configured to report. An actionable signal does the opposite job: it surfaces what the system was never configured to report, it arrives while the agent can still do something about it, and it's structured for a machine to parse rather than for a human to interpret on a screen. Lightrun's analysis calls this "runtime context," evidence pulled on demand from the live running system at the exact line where something failed, with no code change and no redeployment required to get it.

Getting agent observability right takes a layer of structure telemetry alone can't supply. Tagging each span with structured attributes, a user ID, a session ID, a workflow ID, lets an engineer query across millions of operations by tool version, model version, or user segment. Without those tags, a team can replay one failing trace and still have no way to find the same pattern repeating across thousands of others. The OpenTelemetry GenAI specification offers an experimental shared schema for these attributes, so traces generated by different frameworks can land in the same backend without custom parsing logic for each one, but every gen_ai.* attribute in that specification still sits in Development, pre-stable status as of 2026.

A signal is also a different object than a raw trace. It's a recurring failure pattern that's been clustered, named, and tracked through a lifecycle: something closer to a tracked issue, complete with example traces, a count of affected users, and a path toward resolution. Latitude's March 2026 platform comparison describes this as "Signals," the intelligence layer that sits above raw trace capture and turns a pile of recurring failures into named, tracked problems a team can actually work through. Confident AI identifies the real payoff of building this layer: every failure caught in production becomes a benchmark case, so the same bug can't ship twice. That's what closes the loop from watching a system passively to improving it actively.

Why the merge queue, not CI, is where agent signals must land

Agent-generated code needs a place where signals can still stop a bad change before it ships, and that place is the merge queue, not CI. CI checks a branch against a snapshot of the codebase at one point in time. A merge queue builds an ephemeral merge group and validates it against the latest state of the target branch, which matters a great deal when agents are merging code continuously and the target branch is moving underneath them. exe.dev's June 2026 analysis argues the merge queue is better suited to agent workflows because it validates against the latest state of the target branch as agents merge continuously. There's a technical condition attached to that, too: CI has to subscribe to the merge_group event, because if tests don't run on that event, First AI Movers puts it bluntly: "you do not have a merge queue, you have a hope."

You need to understand a limitation in the AI code review tools doing the work upstream of the queue. Claude Code Review will never formally approve a pull request, because of a deliberate vendor policy rather than a technical gap, and its check run always completes with a neutral conclusion, so it can never block a merge through branch protection rules on its own. GitHub Copilot's default behavior is similar: it always leaves a "Comment" review rather than an "Approve" or "Request changes" review, though GitHub now lets administrators turn on formal approval, a feature that has stayed off by default since September 2026. So in practice, you can't hand the merge queue off entirely to whichever AI review tool wrote the comments. It needs a purpose-built layer that can act on a signal. Elastic's own CI practice shows what that supervision looks like in production: its Claude-embedded pipeline for dependency updates across large monorepos explicitly turns off GitHub's PR auto-merge any time an AI contributed to the change, so every agent-generated update passes through a supervised gate rather than an automatic one. An empirical analysis of thousands of agent-generated pull requests that touched CI/CD YAML, measured against tens of thousands that didn't, found the PRs with CI/CD changes merged slightly less often (67.77%, against a modestly higher rate for the others), a gap that suggests signal quality at the merge gate is already separating outcomes.

Production issues, exceptions, malformed tool responses, silent confidence errors, need to feed into an agent's workflow as they happen, because the human review cycle that observability was built to support surfaces them only hours later. That's the gap Vex's Agentic Merge Queue is built to close: an agent handles merge conflicts, CI failures, and queue management on its own, so human reviewers step in only where real judgment is required. Signals route to the queue, the agent acts on them, and a person reviews what's actually worth a person's time.

From Execution Data to Actionable Skill Signals

A full-session agent trace carries a second kind of signal most teams never look at: evidence about how well the agent is using the skills it already has, which is the lever for making an agent better rather than just catching it when it breaks. If an agent drags around a pile of unused skills, it adds noise to every reasoning step it takes, so cutting that dead weight matters as much as adding new capability does.

Latitude's comparison describes a feature it calls "Behaviours," which uses machine-learning clustering to group real sessions by meaning and surface patterns a team didn't know to look for. That sits as its own intelligence layer, above raw trace capture but below a finished insight. A related gap exists in eval coverage: Braintrust reports eval pass rates clearly but doesn't tell a team which production failure patterns ought to be added to its eval dataset, and Topics, still in beta, offers clustering to categorize failure modes without measuring quality against them. If production keeps turning up failures the eval suite never anticipated, raw tracing by itself won't close that gap. The stronger approach builds evals straight out of real production failures, so that every new failure caught in the wild extends the benchmark the next version of the agent has to pass, and the same bug stops being able to ship twice.