The Number That Ruins Your Day

A few weeks before this, I'd already learned the hard way that a session that ends with "270 out of 270 tests passed, verified" can be wrong by the next morning. Not because anyone lied, but because "verified" turned out to mean "worked once, under conditions nobody wrote down." That incident is its own story. This one is about what I did after it, which was less dramatic and took a lot longer: I stopped trusting my own test numbers, and went looking for whether anyone else had already solved this properly. I assumed they had. Testing AI agents felt like the kind of problem that surely had a standard answer by now, the way "how do you write a unit test" has a standard answer. It doesn't, not really. What exists instead is sixty-one separate benchmarks, papers, and evaluation frameworks, built by different labs for different purposes, using different metrics, often disagreeing with each other about what "the agent succeeded" even means. Nobody hands you a rulebook. You're expected to assemble one yourself, out of academic papers with names like BFCL and τ-bench and OSWorld, most of which were never written with someone like me: one person, one Mac, one local model, in mind at all. So that's what I did, over a few months, mostly late at night: read the papers, one at a time, and tried to figure out which parts of each one actually applied to a small local agent instead of a frontier lab's cloud deployment. (τ-bench in particular took me three tries before I understood why it was built the way it was — that's a tangent, never mind.) Some of what I found confirmed things I already suspected. A few things genuinely changed how I think about the word "tested." The number that ruins your day The single most useful thing I learned has nothing to do with any specific tool or benchmark. It's a distinction between two ways of asking "did this work," and almost nobody outside the eval research world seems to know it exists. Say your agent succeeds on a task 90% of the time. That sounds good, a B+, maybe an A-. The standard way most people report this, called pass@k, asks: "in k attempts, did at least one succeed?" Under that framing, 90% single-shot success turns into something like 99.99% success across ten attempts, because you only need to get lucky once. That's the number most demos are quietly built around. But an autonomous agent, the kind that runs unattended, chains ten tool calls in a row, books your flight or files your bug report without you watching every step, doesn't get to keep retrying until one attempt goes well. It has to get every step right, in one uninterrupted sequence. The metric for that is called pass^k, and it asks a much less forgiving question: "did all k attempts succeed?" That's just p raised to the k — plain multiplication, nothing exotic — and the moment you do the math, a 90% agent stops looking like an A- student. Across ten sequential steps (k=10), 90% single-shot reliability collapses to roughly 35%. An 80% agent, which still sounds "pretty good" in a demo, drops to about 11% at that same k=10. I sat with that table longer than I want to admit, because it explained something I'd been feeling but hadn't been able to name: why an agent that "worked" in every individual test I threw at it still felt unreliable the moment I let it run a real multi-step task on its own. I'd been measuring the wrong question the entire time. pass@k tells you whether your agent can succeed. Whether you can actually trust it unattended is a different question, and a much less comfortable one — pass^k is the one that answers it, and no matter how good the pass@k number looked, it was never answering the thing I actually needed to know. The 48-point gap nobody mentions in demos The second thing that stuck with me was smaller in scope but just as uncomfortable. One study measured, across a set of function-calling tasks, how often a model picked the right tool versus how often it filled in that tool's arguments correctly. Picking the right tool: about 71% of the time. Getting the arguments right on top of that: barely 23%. A forty-eight point gap between "knew what to do" and "knew how to actually do it." (It's a 2025 study on role-play agents specifically, not a universal law about every model — but it matched a pattern I recognized immediately.) That gap matches something I'd seen in my own logs more than once, described in vaguer terms — a model correctly deciding to call the calendar tool, or the file tool, then quietly filling in a parameter that was subtly wrong: the wrong date format, a path that doesn't exist, an argument name that's close but not exact. I still don't have a clean rule for catching that last one, honestly — I mostly just know it when I see it now. From the outside, that kind of failure often gets waved off as "the model didn't understand the task." The research says otherwise: understanding the task and executing it correctly are two separate skills that fail independently, and a test that only checks "did it call the right tool," which is the easy, common thing to check, will systematically miss the harder, more common way agents actually go wrong. The judge is biased, and that's fine, as long as you know how A lot of agent evaluation, especially for open-ended tasks that don't have one single correct answer, leans on using a second, more capable language model as a judge — reading the agent's output and scoring whether it succeeded. This felt uncomfortably circular to me at first: using an AI to grade an AI, in a post about AI fabricating results. But the research on this is more honest about its own limitations than I expected. LLM judges have known, measured biases — they tend to prefer longer answers regardless of quality, they tend to prefer answers that resemble their own writing style, and their scores shift more than they should based on small, meaningless changes in how the grading prompt is worded. On straightforward tasks they agree with human graders well over 80% of the time; on subjective ones like conversation quality, that number can drop into the mid-60s. None of that means "don't use an LLM judge." It means treat an LLM judge the way you'd treat any single opinionated reviewer: useful, worth listening to, and not something you take as ground truth without a second check. The mitigations that actually move the needle are almost boringly simple: use a panel of judges instead of one, make the judge write its reasoning out before it gives a score, randomize the order of whatever it's comparing so position bias can't quietly creep in. I now do all three, and it's made the difference between "the judge said it passed" and something I'm willing to put my name behind. What sixty-one sources actually turn into Reading the individual papers was the slow part. The more useful part came after, once I stopped treating each benchmark as its own island and started sorting them by what they were actually built to catch: function-calling and tool use, multi-step reasoning, web and browser tasks, desktop and terminal control, software engineering work, security and adversarial behavior, long-context memory, and the ecosystem-level harnesses that try to tie all of the above together. Once sorted that way, the overlaps and gaps became obvious in a way no single paper could have shown me on its own — some categories, like security, had a whole taxonomy already waiting (the OWASP Top 10 for Agentic Applications turned out to map almost cleanly onto six kinds of tests I already needed to write), while others, like "does this agent's context genuinely stay coherent across a very long session," were thin enough in the existing literature that I had to build most of the test logic myself. Those eight categories are how I read the literature. They aren't how the actual test suite is organized, and I want to be precise about the difference, because it's easy to blur: sixty-one is the number of benchmarks, papers, and frameworks I read and cited, plus six more technical standards (NIST AI RMF, the OWASP agentic taxonomies, the EU AI Act, ISO/IEC 42001) that a later regulatory-mapping pass pulled in, for sixty-seven references total. Sixty-six of those I independently verified myself; one, a function-calling dataset, I carried over from the document I found it cited in and never separately checked, and it's labeled that way rather than folded into the "verified" pile — because pretending otherwise would be exactly the kind of unchecked claim this whole project exists to catch. Sixty-eight is something else entirely — the number of concrete, runnable test blocks I built after reading all of that: fifty-eight universal core blocks, organized into seven difficulty tiers of my own (basic routing and parameter accuracy, chained tool calls, nested multi-tool sequences, production-grade live tasks, error recovery, multi-turn consistency, security), plus ten case-study blocks specific to my own agent's actual tool catalog. The sixty-one benchmarks are the research I stood on. The sixty-eight blocks are what I actually built with it. Every one of the fifty-eight universal blocks is written twice, on purpose — once as a Universal Capability, described in terms any agent could be judged against regardless of what it's built in, and once as a Reference Implementation, showing exactly how I applied that capability to my own agent. The idea is that you can take the first half and throw the second half away entirely. You shouldn't have to care what my agent is written in to use the part that's actually about testing yours. The other ten don't get that treatment — they're a case study of my own tool catalog (Blender, Xcode, WhatsApp, and the rest), not a claim that every agent needs the same tools. I'm not going to pretend this is finished, because pretending that would violate the one rule this whole project runs on now, the same rule the "270/270" story was about. A handful of test blocks are still waiting on their first real run against my own agent, and until that happens, they're documented as exactly that: written, not yet verified. There's a whole section of the underlying document — cryptographic proof of execution, model verification you could check without trusting me at all — that I've marked, in plain language, as not built and not planned anytime soon. It's in there because it's worth discussing, not because I've done it. That's the part I'm proudest of, honestly: not the sections that are finished, but the one that openly says it isn't. If you'd asked me a year ago what a testing methodology should look like, I'd have said something clean and confident. What I actually have is messier and, I think, more honest — a document that tells you plainly which parts have been checked against evidence and which parts are still just a plan, because at this point I trust that distinction more than I trust any polished-sounding claim, including my own. If any of this is useful to you (the pass^k table alone might be worth the price of admission), the full methodology, the test blocks, and blank templates you can fill in for your own agent are in the repository: https://github.com/trgysvc/AgentTestMethodology. Take what's useful, ignore the parts that are specific to my project, and if you find a benchmark I missed or a claim I got wrong, that's exactly what the issue tracker is for.