Continuous Evaluation for AI Agents: Why Output Scoring Fails

AI agents fail between 70% and 95% of the time in production environments, and an estimated 88% of enterprise agents that work in controlled demos fail when deployed to real workflows, according to Fiddler’s analysis. That gap between demo and production isn’t a model quality problem — it’s an evaluation problem. Most teams score the final reply. The failure lives in the trajectory.

Here’s what that looks like in practice: your support agent calls lookup_customer, then get_balance, then drafts a reply. By Wednesday it’s calling get_balance first with a fabricated customer ID, getting a permission error, then apologizing fluently. The final message looks fine. The path is broken. Nobody caught it because the only thing being scored was the last message.

This is the core pattern I’ve observed across agent deployments — what I call trajectory over output. Agent failures are path-based and compound across multi-agent chains, yet most evaluation tooling still treats the final string as the signal. If you’re scoring only outputs, you’re blind to the failures that actually break deployments. Continuous evaluation for AI agents has to shift from static pre-deploy test sets to production-trajectory scoring and fork-replay debugging. That’s the entire thesis of this post, and the data backs it up.

The Production Coverage Gap

Pre-deploy and CI evaluation share the same boundary: both run before production traffic reaches the agent, and both depend on test cases someone wrote in advance. A golden dataset only covers the failure modes the team already knew to include. Once real users start interacting with an agent, input drift occurs, new intents emerge, and edge cases show up outside the original test set.

Continuous evaluation fills this gap by running scorers automatically on live traces based on trace classifications. Instead of writing a separate scorer for every possible failure mode — which doesn’t scale — a small set of high-fidelity classifiers categorizes traces first, and scorers read those classifications to detect matching failure patterns in production. A production failure becomes an ongoing quality check. Scored traces trigger alerts, review queues, and regression tests without manual trace-by-trace scoring.

The three stages of evaluation map to three different jobs:

1. Pre-deploy evaluation runs before release against a golden dataset. It gates the release.
2. CI evaluation runs on every pull request against a curated test set. It gates the merge.
3. Continuous evaluation runs on live production traffic. Scorers fire when a trace matches a classification predicate.

Most teams stop at stage two. They have a CI gate and a handful of scripted scenarios. That’s like deploying a web app without logging — you can do it, but debugging issues becomes guesswork. If you want a deeper dive into building multi-layer eval stacks with open-source tools, our guide to testing AI agents covers the architecture patterns in detail.

Why Benchmarks Understate the Problem

Public benchmarks tell you a model can complete a task. They don’t tell you whether your specific agent, with your specific tools and prompts, will complete it reliably in production. The numbers are sobering.

On AutomationBench-AA, Claude Fable 5 leads with 48.6% of objectives completed without guardrail violations. Every model breaks business rules — guardrail violations range from 0.46 to 1.26 per task.

Perplexity’s WANDR benchmark tells a similar story. It’s an open benchmark of 500 research tasks with 170,000+ source-backed records. The best model achieved a soft F1 of 0.363 and a hard F1 of 0.133. These are the best results. Most models are worse.

Here’s the tension worth noting: production failure rates cited as 70-95% conflict with benchmark task-completion rates of 27-73%. The “failure” definition shifts between the demo-to-production gap and measured task success. Fiddler reports 70-95% agent failure rates in production and 88% of demo-working agents fail in real workflows, with multi-agent chains compounding to 34% success. Yet on structured benchmarks, failures are high but not 70-95%. The gap is real, but the framing matters. Benchmarks measure whether a model can complete a task. Production measures whether your agent does complete it, consistently, across drift, edge cases, and multi-agent handoffs.

This is exactly why public AI agent benchmarks are breaking — leaderboard scores are unreliable for production decisions, and cheap proxy methods now outperform them for real-world quality assessment.

The Trajectory Corruption Problem

The dominant failure mode in production agents is not model inaccuracy. It’s trajectory corruption — invisible to output scoring. Agents produce fluent final replies while reordering tool calls, fabricating parameters, or swallowing errors mid-chain. You won’t catch these by reading the last message.

Consider what actually breaks:

• Tool call reordering: The agent calls get_balance before lookup_customer, using a fabricated ID. The error gets swallowed. The final apology looks fine.
• Parameter hallucination: The agent invents an entity ID, gets a permission error, loops twice, and gives up with a polite, useless answer.
• Multi-agent compounding: If each agent succeeds 70% of the time, a three-agent chain succeeds just 34% of the time. Failures cascade across handoffs.

Tools like AgentFlow address this with model-free unit testing — TestAgent cycles through predefined responses while exercising graph logic, routing, and tool wiring at full speed without any LLM calls. Its evaluation layer scores whether the agent called the right tool, in the right order, with the right arguments. That’s trajectory scoring, not output scoring.

SpecOps AI takes a complementary approach — a framework-agnostic, OpenTelemetry-native MIT-licensed toolkit providing deterministic replay, behavioral evaluation, self-healing policies, and root-cause analysis graphs. It’s lightweight, installs in seconds, and works across LangGraph, CrewAI, AutoGen, and Strands.

The point is the same across both: you need to score the path, not just the destination.

Fork-Replay Debugging and Deterministic Governance

When an agent fails in production, you need to reproduce the failure, fork from the point of corruption, and test fixes without re-running the entire pipeline. That’s fork-replay debugging.

Retrace is an execution replay engine that uses one decorator to capture LLM calls and enables fork-from-any-step replay. You branch from any decision point, modify the input, and watch the agent re-execute from that moment forward. Runtime guardrails halt agents before cascades — cost budgets, loop detection, context overflow enforced in real-time. CI eval gates block deploys when quality drops.

On the governance side, Exogram takes a deterministic approach. It’s an Execution Authority Layer that renders a PERMIT or DENY judgment in 0.07ms per evaluation with no LLM inference. The philosophy is simple: separate thinking from doing. Intelligence should improve; execution must be controlled. Every proposed agent action gets intercepted and adjudicated at the execution boundary before it reaches production infrastructure.

These represent two different tradeoffs:

• Deterministic execution governance (Exogram): Code-based decisions, same input same output every time. Fast, predictable, but rigid.
• Probabilistic continuous red-teaming (self-play models, adaptive guardrails): Flexible, discovers novel attack vectors, but non-deterministic and slower.

You likely need both. Deterministic governance for hard boundaries — don’t call that API, don’t write to that table. Probabilistic red-teaming for discovering failure modes you didn’t know existed.

Cost Models: BYOK vs. Platform-Billed Evaluation

Evaluation pricing splits into two opposing cost-visibility models, and the choice has real implications at scale.

Some platforms externalize all model cost to user BYOK. Prompt Assay Skills are BYOK at every tier — every LLM call from a Skill instrument bills the user’s provider account directly, and Prompt Assay never marks up inference. The per-run Behavioral Eval cap is uniform across every tier: up to 5 models, 10 trigger probes, and 6 non-trigger probes per run (max 80 cells = 160 LLM calls). The Team tier costs $99/seat/month with unlimited Skills.

Kelet takes a related approach — it fine-tunes private per-account models automatically for root-cause analysis. The Startup tier is $400/month (free during early access) and includes 5,000 sessions/month with root cause analysis and prompt patch generation.

Other platforms monetize evaluation inference as a usage-based service. Patronus AI charges $10-$20 per 1,000 evaluator calls per month, with a free Developer tier. LangWatch uses a pay-per-event model that “can become expensive at high message volumes” — the Launch plan starts at $200/month, while Developer is $0/month with full feature access for development and 14-day trace retention.

Here’s the math on a 50-developer continuous evaluation stack using the BYOK route: Prompt Assay Team at $99/seat/month × 50 = $4,950/month, plus Kelet Startup at $400/month, totals $5,350/month or $64,200/year in platform subscriptions alone — and that excludes BYOK inference billed to provider accounts. The platform cost is predictable. The inference cost is variable and depends entirely on your eval volume.

For teams already thinking about production AI agent architecture and cost drivers, this split matters: most production costs come from human oversight, not model inference. Architecture choices that reduce review steps are the fastest path to affordable deployments — and trajectory-level evaluation is what makes those reductions possible.

Tool Comparison: Pricing and Architecture Tradeoffs

Tool	Pricing	Architecture	Best For
Braintrust	Free tier; Pro: $249/mo	Unified observability + evals	Teams wanting one platform for traces and scoring
Langfuse	Free: 50K units; Core: $29/mo; Pro: $199/mo	Open-source, self-hostable	Teams needing data sovereignty and low entry cost
Patronus AI	Free Developer tier; $10-$20 per 1,000 evaluator calls	Specialized evaluator models (Lynx, Glider)	Enterprise teams needing hallucination detection and RAG grounding
LangWatch	Developer: $0/mo; Launch: $200/mo	Unified observability + evals + guardrails	Teams wanting guardrails and evals in one platform
Agenta	Hobby: Free; Pro: $49/mo; Business: $399/mo	Open-source, MIT-licensed	Small teams shipping LLM features with low-code collaboration
Future AGI	Open-source (Apache 2.0), self-hostable	Unified: tracing, evals, simulations, gateway, guardrails	Teams wanting full-stack open-source with self-hosting

The tradeoff here is between unified single-pane platforms and composable decoupled layers. Future AGI collapses tracing, evals, simulations, datasets, gateway, and guardrails into one platform — no more stitching together separate tools. Braintrust integrates evaluation directly into the observability workflow with trace classifications powering continuous scoring.

The composable alternative is something like Verifiers v1, released July 10, 2026. It decomposes agentic RL environments into taskset, harness, and runtime, with a DAG message format replacing quadratic trace growth. Each message is stored once and linked to its predecessor, keeping trace size linear in the number of turns. For long-horizon agentic training, that difference is substantial. The decoupling means any taskset can run under any compatible harness — you’re not locked into one evaluation paradigm.

Open-Source and Specialized Evaluation Layers

Several tools occupy specific niches in the evaluation stack rather than trying to be the whole platform.

Cekura provides scenario-driven multi-turn testing, persistent regression baselines, and continuous production monitoring for chatbot response consistency. It auto-generates multi-turn scenarios from prompts or knowledge bases and replays them across different models or prompt versions to surface non-deterministic variation.

CallSphere runs eval-driven fine-tuning loops where every fine-tune is gated on regression metrics via DeepEval CI. Production canary runs on 10% live traffic for 24 hours before full rollout. Every eval run pins prompt SHA + dataset SHA + model SHA, so you can answer “why did Healthcare F1 drop 0.02 last Tuesday” in 90 seconds.

On the red-teaming side, OpenAI trained GPT-Red, an automated red-teaming model, and used it to adversarially train GPT-5.6 to improve robustness to prompt injections. GPT-Red was built through a self-play loop — it attacked other models, they defended, and over many rounds it discovered new attack types that hadn’t been seen before. MIT Technology Review describes it as an “LLM super-hacker” that automates a process typically done by human testers. The risk surface grows as agents interact with files, websites, and third-party code — human red-teamers alone can’t keep up.

Skyfall AI’s MORPHEUS takes a different angle — a persistent enterprise simulation benchmark for continual reinforcement learning with failure injection at rates from 5% to 30%. It forces continual learning through persistence (past decisions compound), non-stationarity (fixed policies become suboptimal), and operational complexity (no fixed optimal policy exists). Composite reward weights default to w_f=0.5, w_l=0.25, w_p=0.25.

For governance and compliance, the LatticeFlow AI Platform connects governance frameworks with technical controls to continuously generate evidence and translate evaluation results into risk insights for agentic systems. It covers more than 40 frameworks including the EU AI Act, OWASP, NIST, ISO 42001, and FINMA. And federal experts say continuous monitoring, evaluation, and red teaming help organizations ensure AI models are safe and secure — the government is arriving at the same conclusion.

The Decision Framework

Agent reliability will not improve through larger models. It will improve through trajectory-level evaluation wired into CI and production replay loops. Teams scoring only final outputs are blind to the failures that actually break deployments.

Here’s how I’d think about the decision:

1. If you’re shipping agents to production without continuous evaluation, start with open-source. Langfuse (Free: 50K units, Core: $29/mo, Pro: $199/mo) or Future AGI (Apache 2.0, self-hostable) give you tracing and basic evals without vendor lock-in. You’ll learn what failure modes your agents actually hit before paying for specialized evaluators.

2. Pro at $249/month for unlimited spans and scores is reasonable for teams that have outgrown the free tier.

3. If you need hallucination detection and RAG grounding, Patronus AI’s specialized evaluator models (Lynx, Glider) are purpose-built. At $10-$20 per 1,000 evaluator calls, the cost scales with usage — which means you should be selective about what you score. Don’t run every trace through every evaluator.

4. If you need deterministic execution governance, Exogram’s 0.07ms PERMIT/DENY is the fastest option. No LLM inference, no probabilistic interpretation. It’s a hard boundary, not a suggestion.

5. If you need fork-replay debugging, Retrace’s one-decorator capture and fork-from-any-step replay is the lightest-weight approach. It works with any Python or TypeScript agent — no framework lock-in.

The open question I’d leave you with: if your agent produced a fluent, confident, grammatically perfect final reply right now, would you know whether the tool calls that produced it were correct? If the answer is no, you’re not doing continuous evaluation — you’re doing hope-based deployment. And the data says that fails 70-95% of the time.