{"id":1169137,"date":"2026-04-21T09:53:12","date_gmt":"2026-04-21T16:53:12","guid":{"rendered":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/?post_type=msr-blog-post&p=1169137"},"modified":"2026-05-04T09:57:18","modified_gmt":"2026-05-04T16:57:18","slug":"the-art-of-building-verifiers-for-computer-use-agents","status":"publish","type":"msr-blog-post","link":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/articles\/the-art-of-building-verifiers-for-computer-use-agents\/","title":{"rendered":"The Art of Building Verifiers for Computer Use Agents"},"content":{"rendered":"\n
\n

By Corby Rosset, Pratyusha Sharma, Andrew Zhao, Miguel Gonzalez-Fernandez, Ahmed Awadallah<\/em><\/p>\n\n\n\n

\"graphical<\/figure>\n\n\n\n

We share lessons learned from building a best-in-class verifier for computer use agent trajectories on the web, called the Universal Verifier. False positive rates drop to near zero (vs. \u226545% for WebVoyager, \u226522% for WebJudge), and agreement with humans matches human-human agreement. We open-source our Universal Verifier system along with CUAVerifierBench, a new set of CUA trajectories with both process and outcome human labels.<\/p>\n\n\n\n

Here’s what we found:<\/p>\n\n\n\n

    \n
  1. Good verifiers rely on rubric design<\/strong>\u2014and good rubrics must have specific, non-overlapping criteria, since flawed rubrics produce errors that cascade through the pipeline and can’t be corrected downstream. Good rubric design alone accounts for roughly half the gains.<\/li>\n\n\n\n
  2. Separating process from outcome and controllable from uncontrollable failures is a core design principle<\/strong>\u2014conflating process and outcome leads to reward signals that are either too lenient or too harsh. We further distinguish controllable failures (e.g., reasoning errors, hallucinations) from uncontrollable ones (e.g., CAPTCHAs, out-of-stock items).<\/li>\n\n\n\n
  3. The Universal Verifier matches human-human agreement levels<\/strong> (Cohen’s \u03ba 0.64) while cutting false positive rates to near zero\u2014outperforming WebVoyager and WebJudge by a wide margin. The advantage stems from verifier design, not just a stronger backbone model.<\/li>\n\n\n\n
  4. Verifiers deserve the same rigorous evaluation and iterative improvement we apply to models<\/strong>\u2014CUAVerifierBench makes this concrete, providing human-labeled trajectories to benchmark verifier quality and drive systematic progress.<\/li>\n\n\n\n
  5. Auto-research agents can’t fully replace human experts in verifier design yet<\/strong>\u2014but they reach ~70% of expert quality in just 5% of the time, and can even find incremental improvements on top of a human expert’s best work.<\/li>\n<\/ol>\n\n\n\n

    Full paper is available here (opens in new tab)<\/span><\/a>, and code and data are available at https:\/\/github.com\/microsoft\/fara (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n

    <\/div>\n\n\n\n

    Why is it so hard to tell whether the agent succeeded?<\/h2>\n\n\n\n

    Computer use agents \u2014 models that browse the web, click buttons, fill forms \u2014 have gotten impressively capable. But progress on training and evaluating them is bottlenecked by a deceptively simple question: did the agent actually succeed?<\/em><\/p>\n\n\n\n

    This turns out to be much harder than it sounds. Unlike text generation, where you can compare an output to a reference, computer use trajectories are long, visually rich, and interact with environments the agent does not control, inviting new categories of errors like environment blockers, out-of-stock items, and logins. A task might be partially completed. Success might arrive through an unexpected path. Failures can be subtle \u2014 e.g. mis-copying numbers from a table that appear only in a screenshot buried deep in a multi-step interaction. And the consequences of getting verification wrong compound: bad labels corrupt both your benchmarks and your training data.<\/p>\n\n\n\n

    We spent 96 experiments and several weeks building what we call the Universal Verifier \u2014 a system designed to verify agent success and score its effort against a generated rubric. What we ended up with is less a single trick and more a set of learned design principles, each addressing a failure mode we discovered. This post walks through those principles, what we tried that didn’t work, and what surprised us.<\/p>\n\n\n\n

    \"Line<\/figure>\n\n\n\n

    Figure 1: Human expert vs. auto-research agent across successive verifier design iterations. The expert iterated over 32 experiments across three weeks; the auto-research agent completed comparable iterations in roughly one day.<\/em><\/p>\n\n\n\n

    <\/div>\n\n\n\n

    How do you build a good rubric?<\/h2>\n\n\n\n

    The root of the pipeline is rubric generation, and flawed rubrics produce errors that cascade through everything downstream. We found four systematic failure modes \u2014 and rubric design alone accounted for roughly half of our total Cohen’s \u03ba gains. You can see how our rubrics evolved on WebTailBench here: https:\/\/microsoft.github.io\/fara\/docs\/webtailbench_rubric_comparison.html (opens in new tab)<\/span><\/a><\/p>\n\n\n\n

    <\/div>\n\n\n\n

    Phantom criteria<\/h3>\n\n\n\n

    This was the most insidious problem. LLM-generated rubrics frequently introduce requirements that were never stated in the task. For example in Figure 2, given a multi-step task, our early rubric added criteria for the price and address of a hotel \u2014 neither of which the user requested for the primary intent of finding a coffee shop near the hotel. The agent completed the actual task but scored 2\/8 because it “failed” those phantom criteria. After fixing the rubric to match only what was asked, the same trajectory scored 16\/18 \u2014 a success.<\/p>\n\n\n\n

    \"Side-by-side<\/figure>\n\n\n\n

    Figure 2: One way we improved rubrics is by removing “phantom” criteria and focusing only on what the task required.<\/em><\/p>\n\n\n\n

    This matters because phantom criteria inflate the denominator. An agent that did exactly what was asked gets penalized for not doing things nobody wanted.<\/p>\n\n\n\n

    <\/div>\n\n\n\n

    Cascading criteria<\/h3>\n\n\n\n

    When rubric items aren’t logically independent, a single upstream error propagates into every downstream criterion, multiplying the penalty. We learned to ensure each criterion could be evaluated on its own as demonstrated in Figure 3.<\/p>\n\n\n\n

    \"Rubric<\/figure>\n\n\n\n

    Figure 3: An example of error isolation in practice. For the task: “List all the members of the bands Nsync and BackStreet Boys. Find the net worth of the one with the longest last name.” The agent incorrectly identified “Timberlake” as the longest last name when “Kirkpatrick” is correct \u2014 but the error does not cascade to downstream criteria about reporting net worth.<\/em><\/p>\n\n\n\n

    <\/div>\n\n\n\n

    Hallucination detection<\/h3>\n\n\n\n

    Agents sometimes claim success when it contradicts evidence \u2014 they’ll confidently assert they found the right product when they did not. Or worse, they will fabricate results like stating the shopping cart has the product when it is empty. Initially, we generated and scored rubrics in one pass, but this rarely caught subtle hallucinations. So, we separate rubric generation from scoring, and decomposed scoring the rubric into two stages: with and without screenshot evidence. Discrepancies between the two stages surface hallucinations that a single-pass scorer would miss. As we explain below, we handle screenshot evidence very carefully to not miss any details.<\/p>\n\n\n\n

    \"Rubric<\/figure>\n\n\n\n

    Figure 4: A subtle hallucination caught by the Universal Verifier. The agent claimed a model exhibited “+6.2% CIDEr score” when <\/em>the actual paper<\/em> (opens in new tab)<\/span><\/a> showed “+2.8% in CIDEr” \u2014 a discrepancy even human reviewers missed.<\/em><\/p>\n\n\n\n

    We also added conditional criteria<\/strong> for tasks with contingencies \u2014 “buy organic blueberries, or if unavailable, buy non-organic.” At rubric-generation time, we mark some criteria as conditional and update them once the task is attempted, so mutually exclusive criteria don’t interfere with each other.<\/p>\n\n\n\n

    To penalize the agent for doing things that were not anticipated by the rubrics, we have a final post-hoc scoring step to identify such deviations as shown in Figure 5.<\/p>\n\n\n\n

    <\/div>\n\n\n\n

    Process and outcome rewards<\/h2>\n\n\n\n

    Not only did we generate rubrics to assign partial credit of “how well did the agent execute the task?”, but also assign a final outcome pass\/fail score answering whether the user’s goal was achieved. The reason we separate the rubrics from the outcome scores is that, in computer use settings, the environment plays an outsized role in influencing success. An agent can execute flawlessly and still fail because a CAPTCHA appeared, or a product was out of stock, or a login wall blocked the final step. From a model training perspective, it doesn’t make sense to penalize an agent for things outside its control, but from a metrics perspective, we still need to know if a task was completed.<\/p>\n\n\n\n

    The process label<\/strong> is a scored rubric \u2014 a normalized score from 0.0 to 1.0 reflecting execution quality across sub-goals, with specific justifications for why points were earned or lost. While the rubrics do penalize mistakes within the model’s control like hallucinations or incomplete executions, they do NOT penalize for uncontrollable factors. Uncontrollable factors include platform issues (CAPTCHAs, login walls without credentials), entity non-existence (discontinued products, closed businesses), availability constraints (out-of-stock items, no reservations on the requested date), and search result limitations.<\/p>\n\n\n\n

    The outcome label<\/strong> is binary: would a reasonable user consider the task done, regardless of problems stemming from the environment? It’s evaluated from the perspective of someone examining the end state.<\/p>\n\n\n\n

    \"Rubric<\/figure>\n\n\n\n

    Figure 5: An unsolicited side effect. The task was to compare shipping options between Amazon and AutoZone, but the agent added the product to the cart instead of just answering the question.<\/em><\/p>\n\n\n\n

    Why not just use one? Because conflating them leads to reward signals that are either too lenient (crediting agents for apparent effort when the user is left empty-handed) or too harsh (penalizing agents for a CAPTCHA that no model could solve). In reinforcement learning, this distinction is the difference between a training signal that teaches the model to act well and one that teaches it to be lucky.<\/p>\n\n\n\n

    <\/div>\n\n\n\n

    What do you do when the trajectory is long?<\/h2>\n\n\n\n

    The natural starting point is to hand the model a bunch of screenshots and ask: “Did the agent do the task?” Too many screenshots over-exerts the LLM by forcing it to solve a needle-in-a-haystack problem, which scales poorly with longer trajectories. Both WebVoyager and WebJudge take roughly this approach \u2014 WebVoyager includes all screenshots in one context window, WebJudge ranks and selects the top 30\u201350. Giving LLM verifiers too many instructions to look for across too many screenshots overwhelms them into trying to find a “needle-in-a-haystack” problem that scales very poorly with trajectory length. On the other hand, trying to truncate screenshots (like selecting the last one) risks missing ones where hallucinations or failure actually happened.<\/p>\n\n\n\n

    Hence, WebVoyager’s false positive rate with respect to gold human labels is at least 45%; WebJudge’s is at least 22%. That means nearly half the time WebVoyager says “the agent succeeded,” a human annotator would disagree. If you’re using these labels for training, you’re rewarding failure almost as often as success.<\/p>\n\n\n\n

    We went with a divide-and-conquer scheme. We score each screenshot against every rubric criterion to produce a relevance matrix (shown in Figure 6), then group the top-k most relevant screenshots per criterion for detailed analysis. This is both more scalable to longer trajectories and more focused \u2014 the model evaluates each criterion against only the evidence that matters most for it.<\/p>\n\n\n\n

    \"Heatmap<\/figure>\n\n\n\n

    Figure 6: A screenshot relevance matrix. Each cell scores how relevant a screenshot is to a specific rubric criterion, enabling targeted evidence retrieval rather than flooding the context window.<\/em><\/p>\n\n\n\n

    <\/div>\n\n\n\n

    Does all this show up in the numbers?<\/h2>\n\n\n\n

    The short answer: yes.<\/p>\n\n\n\n

    We validated the Universal Verifier on CUAVerifierBench \u2014 a new benchmark of 246 human-labeled CUA trajectories (140 internal, 106 from Browserbase) with both process and outcome annotations. It’s the first benchmark designed specifically to measure verifier quality on both dimensions. We wanted to validate our results with external annotators, and partnered with Browserbase (opens in new tab)<\/span><\/a> to perform a human annotation study.<\/p>\n\n\n\n

    On outcome labels, the UV achieves a Cohen’s \u03ba of 0.64 on the internal set and 0.58 on Browserbase, compared to 0.44\/0.26 for the best WebJudge configuration and 0.31\/0.13 for WebVoyager. More importantly, the UV’s false positive rate is 0.01 on the internal set and 0.08 on Browserbase \u2014 essentially zero. It almost never credits a trajectory with success when a human would call it failure.<\/p>\n\n\n\n

    You might wonder whether this is just a stronger backbone model doing the work. We tested that. Upgrading WebVoyager from GPT-4o to GPT-5.2 does drop its outcome false positive rate from 0.45 to 0.10 \u2014 but it also dramatically increases its false negative rate (0.24 to 0.44), and overall \u03ba improves only modestly. The UV’s advantage is architectural, not model-driven.<\/p>\n\n\n\n

    The UV’s agreement with humans falls within the range of human inter-annotator agreement itself: outcome \u03ba of 0.58 against a human range of 0.53\u20130.57, process \u03ba of 0.43 against a human range of 0.36\u20130.45. The verifier agrees with humans about as often as humans agree with each other.<\/p>\n\n\n\n

    Secondly, we wanted to ascertain if using the Universal Verifier as a SFT training data filter improves model performance over previous filters. In Table 1 below we show that trajectories filtered by the Universal Verifier lead to the best downstream model, especially under data-limited scenario of only training on 3k trajectories. We were somewhat surprised to observe that the process-filter outperforms outcome-filtered; we believe this is due to the process success threshold of 80% allowing some demonstrations of imperfections in the trajectory, which is ultimately beneficial to the model.<\/p>\n\n\n\n

    Experiment<\/th>Filtered by<\/th>Online Mind2Web<\/th>WebVoyager<\/th><\/tr><\/thead>
    3k traj.<\/strong><\/td>Baseline (old verifier)<\/td>0.20<\/td>0.41<\/td><\/tr>
    UV Process<\/td>0.28<\/strong><\/td>0.45<\/strong><\/td><\/tr>
    UV Outcome<\/td>0.24<\/td>0.44<\/td><\/tr>
    9k traj.<\/strong><\/td>Baseline (old verifier)<\/td>0.25<\/td>0.46<\/td><\/tr>
    UV Process<\/td>0.29<\/strong><\/td>0.52<\/strong><\/td><\/tr>
    UV Outcome<\/td>0.29<\/td>0.49<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

    Table 1: Training Qwen-3-VL-8B on insta-150k-v3 trajectories filtered by different verifiers<\/em><\/p>\n\n\n\n

    In this training experiment, we enforced compute equivalence by fixing the number of trajectories, which were sampled from the insta-150k-v3 dataset [1], after being re-solved by the FaraGen pipeline. We trained under largely the same settings as Fara-7B, the only difference being we initialized from Qwen-3-VL-Instruct. These results show that better verifiers lead to higher quality training data and hence better models.<\/p>\n\n\n\n

    <\/div>\n\n\n\n

    Can an AI build a CUA verifier on its own?<\/h2>\n\n\n\n

    The Universal Verifier is approximately 3,000 lines of code and 2,000 lines of prompts \u2014 rubric generation templates, scoring instructions, outcome verification logic, error classification rules \u2014 all designed iteratively by a human expert. Could an AI agent replicate that work?<\/p>\n\n\n\n

    We set up an auto-research experiment using Claude Code with Claude Opus 4.6, running on a 1M-token context window. We tested two settings: starting from blank prompts (all ~2,000 lines replaced with TODO placeholders, with only the code scaffold and the same design principles described above) and continuing from the human expert’s best prompts. A separate compliance agent audited each iteration to prevent the optimizer from memorizing test examples into prompts.<\/p>\n\n\n\n

    The optimization rule was simple: maximize Cohen’s \u03ba without increasing the false positive rate. Any FPR-increasing change gets automatically rolled back. The human expert iterated over 32 experiments across three weeks. The auto-research agent completed a comparable number in roughly one day. The agent reached about 70% of expert quality in 5% of the time. But it plateaued at a \u03ba around 0.55 and couldn’t close the remaining gap.<\/p>\n\n\n\n

    The most revealing part of this experiment was how the two approaches differed. The human expert’s biggest gains came from opinionated, high-level insights. After observing the verifier failing trajectories over minor issues \u2014 things like “inferring most Coursera courses can be audited for free is unsubstantiated” or “not disambiguating apartment from rental-unit” \u2014 the expert deduced general scoring rules like “separate nitpicks from critical failures.” These structural insights drove large jumps in agreement.<\/p>\n\n\n\n

    The auto-research agent tended to be conservative and incremental \u2014 adjusting thresholds, tightening rubric language for individual failure cases \u2014 rather than making the larger structural or conceptual changes that drove the human expert’s biggest gains. It was good at fine-tuning. It was not good at stepping back and asking “what category of problem am I looking at?”<\/p>\n\n\n\n

    A few things stood out watching the auto-research agent iterate. First, code changes consistently beat prompt additions<\/strong> when prompts were already long \u2014 the single most impactful change was injecting rubric scores directly into context, since it provided quantitative calibration without adding more text for the model to parse. Second, forcing explicit rule-checking helped<\/strong>: by naming rules in a mandatory output field, the LLM was far more likely to actually apply them rather than silently ignore instructions buried in a long prompt. Third, concrete tests beat abstract principles<\/strong> \u2014 “would the user say this is useful?” proved more actionable than vague guidance like “be reasonable about minor issues.”<\/p>\n\n\n\n

    <\/div>\n\n\n\n

    What did this project teach us?<\/h2>\n\n\n\n

    After 96 experiments and a few months of staring at CUA trajectories, the thing that stays with us is how much of verification is judgment \u2014 and how poorly that judgment decomposes into simple rules.<\/p>\n\n\n\n

    Each of the four principles we described \u2014 rubric design, process\/outcome separation, controllable\/uncontrollable distinction, and context management \u2014 addresses a failure mode that looks obvious in retrospect but wasn’t obvious at all in practice. Phantom criteria sound like an easy problem to fix until you realize how systematically LLMs hallucinate requirements. Separating process from outcome sounds like a clean abstraction until you’re staring at a trajectory where the agent did everything right and the website just\u2026 didn’t work.<\/p>\n\n\n\n

    The auto-research experiment sharpened this further. An AI agent can reach 70% of the quality in 5% of the time \u2014 that’s genuinely useful. But the last 30% requires the kind of opinionated, structural thinking that comes from looking at failure patterns and asking “what category of problem is this?” rather than “how do I fix this specific case?” This suggests that building reliable verifiers remains as much an art of encoding evaluative reasoning as it is an engineering problem.<\/p>\n\n\n\n

    The verifier doesn’t just tell you whether the agent succeeded. It tells you how<\/em> it failed \u2014 and whether the failure was even the agent’s fault.<\/p>\n\n\n\n

    \n\n\n\n

    Code and data: github.com\/microsoft\/fara (opens in new tab)<\/span><\/a><\/em><\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

    By Corby Rosset, Pratyusha Sharma, Andrew Zhao, Miguel Gonzalez-Fernandez, Ahmed Awadallah We share lessons learned from building a best-in-class verifier for computer use agent trajectories on the web, called the Universal Verifier. False positive rates drop to near zero (vs. \u226545% for WebVoyager, \u226522% for WebJudge), and agreement with humans matches human-human agreement. We open-source […]<\/p>\n","protected":false},"author":43341,"featured_media":1170587,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-content-parent":992148,"msr_hide_image_in_river":0,"footnotes":""},"research-area":[13556],"msr-locale":[268875],"msr-post-option":[],"class_list":["post-1169137","msr-blog-post","type-msr-blog-post","status-publish","has-post-thumbnail","hentry","msr-research-area-artificial-intelligence","msr-locale-en_us"],"msr_assoc_parent":{"id":992148,"type":"lab"},"_links":{"self":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/1169137","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post"}],"about":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-blog-post"}],"author":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/users\/43341"}],"version-history":[{"count":16,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/1169137\/revisions"}],"predecessor-version":[{"id":1170591,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/1169137\/revisions\/1170591"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media\/1170587"}],"wp:attachment":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1169137"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1169137"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1169137"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1169137"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}