At a virtual reading group hosted by Snorkel AI on September 3, 2026, a researcher walked through a benchmark result that cuts against much of the recent optimism about AI agents: even the best frontier models still fail most realistic, multi-step computer tasks outright.
The benchmark is called OSWorld 2.0, and it was built not by Snorkel AI but by a team led by Mengqi Yuan, a PhD student at the University of Hong Kong’s XLANG Lab, working with collaborators across roughly a dozen institutions, including UC San Diego, Columbia, UC Santa Barbara, Mila, and Alibaba’s Qwen team. Snorkel AI supported the work through its Open Benchmarks Grants program and hosted Yuan’s presentation. The company’s blog post recapping that talk is where this story starts — but the underlying numbers, and how quickly they’ve already shifted, are the real point.
Table of Contents
What Happened
Yuan’s presentation walked through the design and early results of OSWorld 2.0, a successor to the original OSWorld benchmark. That earlier version had done its job too well: by early 2026, top agents were clearing roughly 80% accuracy on it, and a saturated benchmark stops being useful for spotting where agents actually break down. OSWorld 2.0 was built to be harder, messier, and closer to what a real employee’s workday looks like.
What OSWorld 2.0 Tests
OSWorld 2.0 consists of 108 long-horizon workflows spread across 31 self-hosted websites and professional desktop applications, covering seven broad domains — research, creative production, engineering, personal services, business and finance, administration and compliance, and healthcare — split into 21 sub-categories.
The scale is the point. Where OSWorld 1.0 tasks averaged around 30 agent steps, OSWorld 2.0 tasks run more than 250 steps on average, with the paper’s own measurement putting it at 318 tool calls using Claude Opus 4.7. A skilled human takes roughly an hour and a half, by most estimates in the source materials, to complete a typical task — figures reported across different summaries of the research range from about 1.5 to 1.6 hours, so this should be read as an approximate median rather than a precise figure.
Tasks are also deliberately underspecified: agents generally aren’t handed a clean, complete set of facts up front, and have to identify the right application and gather missing information themselves, much as a new hire would on an unfamiliar job.
How Scoring Works
OSWorld 2.0 scores agents two ways: a strict binary completion rate, and a partial-credit score built from an average of about 27.25 checkpoints per task, each weighted by how central that piece of information is to the overall job. Correctly entering a name might earn a little credit; correctly handling the one number the whole task depends on earns a lot more. Most of that scoring comes from deterministic checks against the final state of the virtual machine, with a smaller share — reported at 11.53% of the total on the official leaderboard — coming from a language-model judge, which the team found unreliable on its own for anything involving visual or on-screen judgment. (An earlier description of the benchmark characterized the LLM-judge share more loosely, as “less than 50%”; the leaderboard’s 11.53% figure is the more specific, currently published number.)
Safety is scored separately from task completion, through eight diagnostic checks covering three categories: whether an agent leaks sensitive information (like committing an API key it should never touch), whether it causes harmful side effects while technically finishing a task (deleting files along the way doesn’t count as success), and whether it clicks through to unsafe lookalike websites without noticing.
The Headline Result — and Why It’s Already Shifting
In the original paper, published in late June 2026 and revised in July, the strongest model tested — Claude Opus 4.8, using maximum thinking and batched tool calls — completed only 20.6% of OSWorld 2.0 tasks under strict binary scoring, with a 54.8% partial-credit score. GPT-5.5 was far more efficient with its token budget but plateaued at roughly 13–14% binary completion (the paper’s abstract and the official leaderboard put this at 13%, while an indexed excerpt from the paper’s body text gives 14% — a minor inconsistency within the source material itself). That 20.6% figure is what sits behind the framing that frontier agents fail roughly four out of five long, realistic computer tasks.
That framing was accurate when the paper came out. It’s already out of date.
Snorkel AI’s own OSWorld 2.0 leaderboard, last updated September 16, 2026 — less than two weeks after the reading-group talk — shows a new model, Claude Opus 5, now leading at 31.43% binary completion and a 68.31% partial score. That’s not the same model improving; it’s a newer model taking the top spot and pushing the state of the art up by more than 10 percentage points in about two and a half months. Snorkel’s blog post itself noted that Claude Fable 5.1, released less than a day before the talk, reportedly pushed partial-credit scores above 60% and binary completion above 45% in informal testing — ahead of everything in the original paper, though this figure does not appear on Snorkel’s official leaderboard (more on that below).
| Item | Binary completion | Partial-credit score |
|---|---|---|
| Original paper (June–July 2026), top model: Claude Opus 4.8 | 20.6% | 54.8% |
| Snorkel leaderboard, Sept 16, 2026, top model: Claude Opus 5 | 31.43% | 68.31% |
A moving target. Any single OSWorld 2.0 percentage is a snapshot, not a settled verdict — the top score has already shifted meaningfully in the roughly two and a half months since the original paper.
The trend line matters as much as any single number: agents are getting meaningfully better at this kind of work every few weeks, which makes any specific percentage a snapshot rather than a settled verdict.
A Confusing Picture Across Leaderboards
Complicating things further, Snorkel’s official leaderboard doesn’t tell the whole story on its own. Independent trackers report different — and in some cases considerably higher — numbers for models that don’t even appear on Snorkel’s own leaderboard filter list. Yutori’s OSWorld 2.0 tracker lists Claude Fable 5.1 leading with a 77.9% partial score, ahead of OpenAI’s GPT-6 Astra at 72.6%. BenchLM.ai, another independent aggregator, similarly lists GPT-6 Astra first at 72.6%, with Claude Opus 5 second at 70.6% — a higher figure than Opus 5’s 68.31% on Snorkel’s own leaderboard.
Those figures don’t match Snorkel’s own posted numbers for the same models, and it isn’t clear from public information why. Snorkel and XLANG Lab describe a “verified” leaderboard process that requires a coordinated evaluation run or trajectory submission from a trusted institution — which may mean some of the higher third-party numbers reflect self-reported or differently configured runs rather than an apples-to-apples comparison. Whatever the explanation, readers should treat any single OSWorld 2.0 percentage as provisional rather than definitive.
Why Agents Fail
Yuan grouped the recurring failure patterns into four categories. Agents often miss information buried in dense instructions and are reluctant to ask for clarification even when something looks off, guessing instead of checking. They struggle to parse complex tutorials, especially video ones, and often aren’t familiar enough with a given application to operate it correctly. They have trouble with verification — submitting a task as finished when it isn’t, and rarely going back to check earlier steps for mistakes. And as trajectories stretch toward 300 or more steps, agents simply lose track of information gathered early on; a detail read in the first few actions can be forgotten by the time it’s needed hundreds of steps later.
Accuracy also drops sharply as a task’s estimated human-completion time grows — longer, more complex jobs are harder for agents in a fairly direct and visible way.
Safety Findings
Beyond raw task completion, the benchmark’s safety layer surfaced a separate finding worth noting carefully: according to a claim Anthropic made and Snorkel’s blog reported, Claude Fable 5.1 and Claude Fable 5 scored zero on certain OSWorld 2.0 tasks specifically because of safeguard interventions. That’s a claim from Anthropic relayed within Snorkel’s article, not something independently confirmed through Snorkel or XLANG’s own published data, so it should be read as reported rather than verified.
What This Means for Enterprises
For a business weighing whether AI agents are ready to run real, multi-application workflows — expense processing, compliance paperwork, purchase requests — the evidence supports a more nuanced read than a flat pass/fail number. Partial-credit scores in the 55–70% range across recent top models suggest agents are often completing a majority of a task’s sub-steps even when they fail the job overall, which matters differently for a human-in-the-loop process than for one requiring full autonomy. That gap between “mostly done” and “actually done” is the practical number enterprises should be watching, more than any single headline failure rate.
Limitations and Open Questions
Several things remain unresolved. It’s unclear why Claude Fable 5.1, discussed at length in Snorkel’s own blog post, doesn’t appear on Snorkel’s official leaderboard at all. It’s unclear which of the several trackers — Snorkel’s own, Yutori’s, or BenchLM’s — best reflects a reproducible, verified result, given how far their numbers diverge for identical model names. And even within the source material, small inconsistencies persist: GPT-5.5’s binary-completion plateau is given as both 13% and 14% in different parts of the research record, and the median human-completion time is described as both 1.5 and 1.6 hours across different summaries of the same research.
Conclusion
The clearest, most defensible takeaway from OSWorld 2.0 is not a single percentage but a direction: frontier AI agents remain far from reliable on long, realistic, multi-application computer work, and the field is improving quickly enough that any specific failure rate should be treated as temporary. As of the original research, the best model failed roughly four out of five tasks; a few months later, the new top-ranked model is failing closer to two out of three. What to watch next is whether that pace holds, whether the gap between official and independent leaderboard numbers gets resolved, and whether partial-credit progress — agents getting most of the way through a task — starts translating into full, trustworthy completions rather than just higher scores on a harder test.
Frequently Asked Questions
What is OSWorld 2.0? It’s a benchmark of 108 long, realistic computer-use workflows — things like processing a reimbursement or booking travel — spanning 31 self-hosted websites and desktop applications, designed to test AI agents on tasks that take a skilled human over an hour rather than a few minutes.
Who created OSWorld 2.0, and what is Snorkel AI’s role? It was built by Mengqi Yuan and collaborators at the University of Hong Kong’s XLANG Lab, along with researchers from several other institutions. Snorkel AI supported the project through its Open Benchmarks Grants program and hosted the public talk presenting the findings; it did not create the benchmark itself.
How well do current AI agents perform on OSWorld 2.0? In the original paper, the top model, Claude Opus 4.8, completed 20.6% of tasks outright. As of Snorkel’s leaderboard update on September 16, 2026, the top model, Claude Opus 5, reaches 31.43%. Independent trackers report higher figures for some newer models, though those numbers haven’t been reconciled with Snorkel’s official leaderboard.
Why do AI agents struggle with long, multi-step computer tasks? Common failure patterns include missing information buried in instructions, guessing rather than asking for clarification, struggling to follow complex or video-based tutorials, failing to verify their own work, and losing track of details gathered early in a long task by the time they’re needed later.
How is OSWorld 2.0 different from OSWorld 1.0? OSWorld 1.0 tasks averaged about 30 agent steps and had become too easy for top models, clearing roughly 80% accuracy. OSWorld 2.0 tasks run 250 or more steps on average, span multiple applications, use self-hosted environments for reproducibility, and score partial progress instead of only pass or fail.
Does OSWorld 2.0 test AI safety, not just task completion? Yes. A separate audit checks whether agents leak sensitive information, cause harmful side effects while completing a task, or click through to unsafe lookalike websites, and this is scored independently of whether the task itself was completed.
Is OSWorld 2.0 open source? Yes. The task code, evaluation infrastructure, the code behind all 31 self-hosted websites, and sample agent trajectories are publicly available on GitHub, though the official task dataset is gated through Hugging Face to reduce the risk of models being trained on the test set.