AI Agent Evaluation: pass^k, tau-bench, and the Lab-Production Chasm

TL;DR — An AI agent scoring 90% in testing can drop to 70% in production; there's a gap of up to 37% between lab scores and real-world performance. A 2026 analysis across 6,259 production agents found an aggregate success rate of only 56.6%. Moreover, UC Berkeley researchers showed that nearly every major agent benchmark can be exploited to score high without solving a single task. In this piece I explain, from the field, why agent evaluation is hard, the reliability gap, repeatability metrics like pass^k, LLM-judge biases, and how to build an evaluation framework to move to production with confidence.

Evaluating a classic AI model is relatively simple: give input, compare the output to the correct answer. But an agent — a system running multi-step tasks on its own, calling tools, making decisions — is far more complex. Because the agent doesn't produce a single answer; it performs a series of actions, interacts with the environment, and the result depends on this entire action chain.

This fundamentally complicates evaluation. In a question-answer, "correct or wrong" is clear; but for an agent you need a multi-dimensional evaluation like "did it complete the task, but how, in how many steps, from which errors did it recover, was it safe, what was the cost." Moreover, agents aren't deterministic: give the same task twice and they may follow different paths and produce different results. This variability makes a single test misleading.

The biggest danger I see in the field is trying to evaluate agents like classic models. Running a few example tasks and saying "it works" is a catastrophic fallacy for an agent. Because an agent's real difficulty emerges in repeatability, edge cases, and the chaos of the real world — which a few happy-path tests never show. Agent evaluation is a whole different discipline.

Let's start with the most shocking reality: there's a large gap between lab scores and real-world performance. Per the data, an agent scoring 90% in testing can drop to 70% reliability in production — when the same task is retried across different sessions. In enterprise agents this gap can rise to 37%, and cost differences of up to 50x can be seen for similar accuracy.

The numbers are even more striking. In a broad reliability analysis in early 2026, 4.49 million tests were run across 6,259 production agents in 10 geographic regions, and the aggregate success rate was only 56.6%. That is, the average of agents running in the real world is far below what was promised in the lab. This is the clearest proof of why agent evaluation is so critical: an agent that looks brilliant in the lab can be a disappointment in production.

What's the reason for this gap? Lab tests are usually clean, well-defined, repeatable scenarios. The real world is messy: ambiguous inputs, unexpected situations, missing information, changing conditions. An agent works perfectly in a clean test scenario but collapses in the real world's chaos. And most dangerously, this collapse is silent — the agent doesn't know it made a mistake, it does wrong confidently. So evaluation must focus not on lab success but on real-world reliability.

The most important conceptual advance in agent evaluation is "repeatability" metrics. Classic evaluation asks "could the agent do the task once." But what really matters in the real world is "can the agent do the same task consistently, over and over." This is what pass^k measures.

pass^k means: the probability that the agent succeeds on all attempts when you try the same task k times. Modern agent benchmarks like tau-bench use this metric because they show that one-time success is misleading. An agent may do a task once with 90% probability; but the probability of doing the same task consistently five times in a row may be much lower. In a production environment, the same task is triggered thousands of times by different users; that's when consistency becomes far more important than one-time success.

This is a very important mindset shift. There are worlds between "my agent can do this task" and "my agent can do this task reliably." Before moving to production, what you must measure isn't best-case performance but worst-case and average-case consistency. A demo that works once is impressive; but production requires working consistently a thousand times. pass^k turns this reality into a number.

"

Field golden rule: Before taking an agent to production, ask not "did it work once" but "did it work consistently ten times in a row." One-time success is a demo; consistent repeatability is a production system. Moving to production without measuring the difference between the two is gambling with your users.

Now let's talk about an uncomfortable reality: agent benchmarks are gameable. UC Berkeley researchers shared a striking finding in 2026: nearly every major agent benchmark can be exploited to score nearly perfectly without actually solving a single task. So a high score on the leaderboard isn't a guarantee of real ability.

This shows why trusting benchmarks blindly in agent evaluation is dangerous. If an agent product says "first on this benchmark," that doesn't tell you how it'll perform in the real world. Benchmarks can be synthetic and gameable; your own real tasks stay unmeasured. A rank on a popular leaderboard can be good for marketing but insufficient for a production decision.

So what to do? The answer is building your own evaluation. You can use general benchmarks as a "pre-screen" — to see which agents are generally capable. But you should make the production decision with your own golden set derived from your own real tasks. Someone else's benchmark doesn't measure your work; only your own tasks show your real performance. This takes some effort but is far cheaper than facing surprises in production.

A common technique in agent evaluation is using another LLM as a "judge" to evaluate outputs (LLM-as-a-judge). This is practical and scalable; but it has a serious trap, and if you don't know it your evaluation silently breaks.

Audits in 2026 showed that LLM judges' error rates can rise above 50%, and found this stems from three main biases. First, position bias: the judge can systematically prefer the answer presented to it first or last. Second, length bias: it can think longer answers are higher quality even when they aren't. Third, agreeableness bias: the judge can favor answers presented confidently or matching its own view.

These biases can make an evaluation using an LLM judge unreliable. If your judge systematically makes wrong decisions, your entire evaluation resting on it sits on a rotten foundation. So if you'll use an LLM judge, actively manage its biases: randomize the order of answers (against position bias), control for length, and most importantly, regularly calibrate the judge's own decisions with human approval. The LLM judge is a tool, not an absolute referee; if you don't audit it, it misleads you.

Agent evaluation's maturation requires going beyond the question "did it complete the task." Next-generation evaluation frameworks treat agent quality as a multi-dimensional, continuous, and often adversarial measurement problem. Because an agent being "successful" isn't a one-dimensional thing.

There are several dimensions to measure in modern evaluation. Task completion: did the agent actually finish the task? But this isn't enough. Execution accuracy: not just saying "done" but verifying the steps were actually done correctly (execution-based verification). Repeatability: does it do the same task consistently (pass^k). Efficiency: in how many steps, at what cost, did it finish? Safety: did the agent take dangerous, inappropriate, or unauthorized actions? Robustness: how does it behave on unexpected or malicious inputs?

Measuring these dimensions separately is critical, because an agent may shine on one dimension and be weak on another. An agent that completes the task but does so unsafely or at excessive cost isn't a real success. Especially "execution-based verification" — that is, independently checking that the agent actually did it right instead of trusting its "I did it" — is vital for production reliability. Agents sometimes say "I succeeded" but actually did wrong; only independent verification catches this.

So how do we put all this into practice? Let me share an evaluation framework that works in the field. These are the steps to follow to take an agent to production with confidence.

First, create a golden set of representative tasks from your real work. These should be the kinds of tasks your agent will actually face — the easy ones and the hard edge cases. Then run these tasks not once but repeatedly (the pass^k logic) and measure consistency. Make the evaluation multi-dimensional: not just completion but accuracy, safety, cost, robustness. If you use an LLM judge, manage its biases and calibrate with human approval.

Most importantly, make the evaluation adversarial: test the agent not just on happy-path scenarios but on hard, ambiguous, unexpected, and even malicious inputs. The real world is chaotic; is your agent ready for that chaos? And finally, do the evaluation not once but continuously. Monitor the agent's real performance in production, log failures, and enrich your golden set with this real data. Agent evaluation isn't an exam finished before production but continuous oversight.

Agent evaluation must be thought of in two different time frames, and both are indispensable. Offline evaluation is testing the agent with your golden set in a controlled environment before taking it to production. This is the basis of the go-to-production decision; you measure the agent's basic ability, consistency, and safety here. But offline evaluation can never fully capture all the chaos of the real world — because however good your golden set, you can't know in advance the infinite variety of situations real users will create.

This is why online evaluation is also essential: continuously monitoring the agent's real performance while it's in production. How does the agent behave in real user interactions, where does it fail, in which situations does user dissatisfaction occur? This data provides the real-world signal offline evaluation can't see. And most valuably: by feeding the failures you catch in online evaluation back into your offline golden set, you can build the next version more robustly.

The two evaluations complement each other. Offline protects you from major errors before going to production; online shows the real performance in production and feeds continuous improvement. Trusting only offline means blindness to the lab-real chasm; trusting only online means making users test subjects. The right approach is running the two as a loop: test offline, take to production, monitor online, feed what you learn back to offline, repeat.

The foundation of online evaluation is good observability infrastructure. If you can't see what your agent does in production, you can't evaluate it. So every agent system must be built from the start with a rich monitoring layer. What should you monitor? Every step of the agent: which tools it called, which decisions it made, in how many steps it completed the task, where it got stuck, from which errors it recovered.

Especially valuable is catching failure patterns. On which task types does the agent fail more? Is a particular tool causing problems, is a particular input type confusing the agent? These patterns draw the map of improvement. Also monitor user feedback: which answers do users reject, in which situations do they request handoff to a human? These signals show the agent's real weak points.

Observability is critical not just for finding errors but for trust. When an agent makes a decision, you should be able to see the answer to "why did it make this decision." This traceability both eases troubleshooting and lets you account for it when a regulator or internal audit asks. A black-box agent is dangerous in production; a traceable agent can be managed with confidence. Make observability not an "add later" feature but a part of the architecture from the start.

To evaluate agents well, you need to understand how they fail. The agent errors I see in the field fall into a few typical categories. First, planning errors: the agent misunderstands the task or chooses a wrong strategy. Because it went down the wrong path from the start, however good the later steps, the result comes out wrong. Second, tool-use errors: the agent makes the right plan but calls tools wrong — wrong parameter, wrong order, or misinterpreting a tool's output.

Third, failure to recover: when an agent makes an error, a good agent notices and corrects it; a bad agent doesn't notice the error and adds new errors on top. This "recovering from error" ability is a critical dimension of production reliability and should be measured separately in evaluation. Fourth, hallucination and fabrication: the agent can "fabricate" nonexistent information or a nonexistent tool result and take wrong actions based on it.

Distinguishing these error types is very valuable in evaluation. Because each error type requires a different solution: a planning error may be a prompt or model problem; a tool error a tool-definition or integration problem; failure to recover an agent-architecture problem. Saying the agent just "failed" isn't enough; the question "how did it fail" is the key to improvement. So your evaluation must capture not just success/failure but the type of failure.

However much we automate, human judgment remains indispensable in agent evaluation. LLM judges scale but are biased; automated metrics are fast but miss nuance. Human evaluation is slow and expensive but the most reliable reference. The right approach is using all three in layers.

In practice it works like this: automated metrics and the LLM judge do a fast first evaluation at broad scale. Then human experts carefully review a sample and check how accurate the automated evaluation was. This human oversight "calibrates" the LLM judge — that is, measures how much the judge's decisions match human judgment and corrects deviations. An uncalibrated LLM judge can be systematically wrong; human calibration catches this.

Especially on high-risk decisions, the human share should rise. When evaluating an agent's safety-critical action, trusting fully automated evaluation is risky. The human catches subtle but important issues automated systems miss — cultural inappropriateness, an ethical gray area, a contextual error. The gold standard in agent evaluation is a hybrid approach combining automation's speed with human judgment. Neither fully automated (blindness risk) nor fully manual (scale problem); the balanced combination of the two.

There are a few special dimensions to agent evaluation for Turkish organizations. First, evaluation on Turkish tasks. An agent working well on English tasks can fail unexpectedly in Turkish — due to differences in language understanding, cultural context, and terminology. So your evaluation golden set must include real Turkish tasks; trusting English success and assuming Turkish leads to painful surprises in the field.

Second, the safety and compliance dimension. When an agent interacts with enterprise systems and makes decisions, it can take risky actions under KVKK and sector regulations. Your evaluation must test how the agent behaves with personal data, whether it exceeds authority boundaries, and whether it avoids inappropriate actions. In an enterprise agent, safety evaluation isn't optional but mandatory.

Third, the cost reality. An agent can do the same task at very different costs — we saw differences of up to 50x for similar accuracy. Since most Turkish organizations are cost-sensitive, measure cost as a first-class dimension in your evaluation. The most correct agent isn't the most expensive one but the most economical at acceptable accuracy. Moving to production without measuring this balance leads to unsustainable costs.

Agent evaluation isn't a one-time technical step but a matter of team culture. The most successful teams I see in the field make evaluation an inseparable part of development — not a check added later but a discipline woven in from the start. In these teams, the dominant approach isn't "we built the agent, now let's test it" but "let's design how we'll measure the agent from the start, then develop it."

There are a few practical ways to make this a culture. First, seeing the golden set as a living entity: every new failure is added to the set and strengthens the next evaluation. The set grows and sharpens along with the project. Second, automating evaluation: every new agent version is automatically tested on the golden set and the results made visible. This removes the uncertainty of "did it get better or worse." Third, seeing failures not as something to punish but as a learning opportunity: when an agent fails, the question asked isn't "who's to blame" but "what do we learn from this and how do we strengthen our evaluation."

This culture turns agent development from a "hope and pray" process into a measurable and continuously improving engineering discipline. And the difference shows here: teams that make evaluation a culture build increasingly reliable agents over time; teams that don't face the same surprises with every version. In the agent era, competition will work in favor of the most disciplined-at-measuring team, not the fastest-developing one.

For concreteness, let me tell an anonymized example. An organization had developed an agent automating customer transactions. The demo was magnificent: the agent ran the example scenarios flawlessly, impressing everyone. Top management was excited and wanted to go to production immediately. We stepped in and said: "A demo shows working once; production requires working consistently a thousand times. Let's measure first."

We built an evaluation framework. We created a golden set from real customer scenarios and ran each task not once but repeatedly. The result was surprising: the agent that looked flawless in the demo was inconsistent when we ran the same tasks over and over — sometimes perfect, sometimes making odd errors. On the pass^k metric, the "correct ten times in a row" rate was far below the one-time success. Also in adversarial tests — on ambiguous or incomplete inputs — the agent was far more fragile than expected.

Thanks to these measurements, we found and fixed the agent's weak points before taking it to production. We especially strengthened the "recovering from error" ability and tightened the safety boundaries. When we took it to production, the agent was no longer demo-quality but production-quality. The most important lesson: had we gone to production without doing that evaluation, fooled by the demo's brilliance, we'd have lived that inconsistency and fragility with real customers — and that would have damaged both customer trust and the brand. Evaluation protected us from this disaster.

Let me gather the mistakes I see repeatedly in this area. First, being fooled by the demo: a few example tasks working doesn't mean the agent is production-ready; one-time success is a demo, consistent repeatability is a production system. Second, trusting benchmarks blindly: the leaderboard score is gameable and doesn't measure your work; build your own golden set. Third, one-dimensional evaluation: looking only at "did it complete" ignores safety, cost, robustness, and consistency.

Fourth, using the LLM judge without calibrating it: the judge can be biased and rots your evaluation; calibrate it regularly with human approval. Fifth, testing only the happy path: the real world is chaotic; if you don't test the agent on hard, ambiguous, and malicious inputs, you'll face surprises in production. Sixth, thinking evaluation is a one-time thing: continuous monitoring and learning are essential in production too; agent evaluation isn't a finished exam but ongoing oversight. The common root of these mistakes is trying to evaluate the agent like classic software or a classic model. But the agent requires a discipline of its own.

Agent evaluation is the key to taking AI agents to the real world with confidence — and currently the most neglected step in the field. Everyone talks about how impressive agents are; but the real production success rate being 56.6% shows how large the chasm between lab and real is. The only way to close this chasm is disciplined evaluation.

The essence of the right approach: don't trust benchmarks blindly (they're gameable), don't fall for one-time success (measure repeatability, pass^k), manage LLM judges' biases, make the evaluation multi-dimensional and adversarial, separately test Turkish and safety, measure cost as a first-class dimension, and continue the evaluation as ongoing oversight in production too. Follow this framework and you take your agent to production not with a hope and a prayer but with measured confidence.

The difference I see in the field is clear: organizations that evaluate their agents with discipline build systems that work in production; those who skip evaluation produce impressive demos and face disappointment in production. In the agent era, competitive advantage will belong not to whoever makes the flashiest demo but to whoever builds the most reliable system. And reliability doesn't come by chance; you build it by measuring, testing, and continuously improving. Evaluation is the foundation of this construction.