SWE-CoWork: The Best Agents Complete Less Than 8% of Tasks

SWE-CoWork: The Best Agents Complete Less Than 8% of Tasks

Claude-Opus-5.5 and GPT-6 Astra passed around 63% of the new-feature tests in SWE-CoWork, but fully completed only 6.93% and 7.92% of tasks, respectively. The benchmark, presented by TokenWave AI, changes the usual evaluation setup: instead of providing a fixed specification, it places agents inside a simulated software company and requires them to retrieve requirements, coordinate, and verify code.

The gap between passing tests and completing a task is at the heart of the results. In SWE-CoWork, a patch that implements part of the requested functionality is not enough: the agent must get every requested change right and preserve existing behavior. Complete delivery requires both test suites to pass in the same run.

From an isolated repository to a simulated company

The benchmark contains 101 assignments associated with 101 real repositories, with code in 11 languages and projects across 12 domains. Each task places the agent in a simulated organization with 26 coworkers, each with limited responsibilities and knowledge. Relevant information is spread across conversations, work cards, documents, and other team records.

The agent receives a brief handoff note, not a complete list of requirements. According to the SWE-CoWork repository description, 2,217 of its 2,712 requirements are known only to simulated coworkers; another 495 appear in the environment's records. The agent has tools to search those spaces, ask the people responsible, modify code, and document the solution.

The trial allows eight hours per task. Each assignment includes between 16 and 32 requirements, with a median of 28, and evaluation applies hidden tests to the change. F2P measures whether the requested new behavior is added; P2P checks whether the code preserves functions that already worked. A task counts as complete only if it passes all these checks, not merely if it achieves a good average.

This design aims to measure a different kind of work from an issue with exhaustive instructions. The agent has to find out what was decided, distinguish conflicting signals, and keep the requirements in mind while implementing and testing. The task involves coordination and follow-up as well as programming, even though the final outcome is judged on executable code.

A gap between partial tests and delivery

The top two results clearly show the gap. Claude-Opus-5.5 achieved 63.84% on F2P and 93.24% on P2P, but solved 6.93% of tasks. GPT-6 Astra scored 62.91% and 89.15% on those tests, with a complete delivery rate of 7.92%.

ModelF2PP2PCompleted tasks
Claude-Opus-5.563.84%93.24%6.93%
GPT-6 Astra62.91%89.15%7.92%
Seed Evolving39.09%94.48%0.99%
Qwen3.8-Max35.70%91.80%0.00%
GLM 5.331.27%87.72%0.99%
Qwen3.8-Flash30.70%93.40%0.00%
DeepSeek V4.1 Flash28.20%94.30%0.00%
Kimi K327.08%84.88%0.99%
GLM 5.224.40%95.20%0.00%
Seed 2.1 Pro21.01%81.85%0.00%
DeepSeek V4 Pro13.04%90.21%0.00%

The figures in the table come from the project page and correspond to the evaluation of 101 tasks. F2P and P2P are averages of per-task pass rates; the completed-tasks column counts assignments in which all tests passed at once. A high partial score is not the same as reliable delivery: even the model with the best F2P did not complete one task in ten.

The gap is not simply about preserving existing code, either. The leading models passed between 89% and 93% of P2P tests, but many runs did not cover all the new requirements. In its analysis of Claude-Opus-5.5, the team found that 31% of requirements reported as completed failed their tests. An agent may report that it finished part of the work without the verifier confirming the result.

Understanding a requirement does not guarantee implementing it

The study tracked requirements throughout each run, from discovery to final verification. For Claude-Opus-5.5, an average of 93% of its initial questions about cards it consulted were directed to the people responsible for them. However, the agent obtained only 64% of the requirements held by coworkers. Asking the right person helps, but does not ensure that all the necessary information is received.

Unresolved threads also get lost: some questions are postponed and never followed up on. And getting clarification does not complete the work. Among 976 failed F2P nodes linked to requirements in the Claude-Opus-5.5 analysis, 56% reached the implementation phase after the agent had acquired the requirement. Even so, they did not pass the associated test.

The team found a strong association between the proportion of coworker-held requirements a model manages to obtain and its F2P performance: the Spearman correlation between models was 0.94. By contrast, the relationship between the number of tool calls and F2P was weak, with ρ = 0.08. Using more tools was not associated with better results in this comparison; the observed relationship with acquiring requirements was strong, although it does not prove that one causes the other.

Providing the specification up front is not enough either

An additional test separates the difficulty of discovering requirements from the difficulty of carrying them through to completion. When Seed Evolving and GPT-6 Astra received the full specification from the start, their F2P improved by approximately 16 percentage points. For GPT-6 Astra, the result rose from 62.91% to 78.84%. But its completed-task rate remained at 7.92%.

This is a relevant result for companies expecting more detailed instructions to solve agents' problems on their own. In this test, providing the full requirements raised F2P for Seed Evolving and GPT-6 Astra, but did not change the completed-task rate. The overall analysis also records failures at different stages, from acquiring requirements to implementation and verification.

The benchmark also limits what can be concluded. The coworkers are simulated, tasks run in a prepared environment, and the documents and interactions are in English. The repository fixes the tools, containers, and tests to make the evaluation reproducible. Therefore, the results describe performance on SWE-CoWork, not a universal agent success rate across all companies or projects.

What teams deploying them should measure

For a company, the practical takeaway is not that agents cannot code: their F2P percentages show that they can handle a considerable portion of new changes. The challenge is turning that partial ability into a complete delivery, with requirements retrieved, changes tested, and existing behavior preserved. Measuring only new tests can hide incomplete tasks, while an overall success metric exposes failures that an aggregate average obscures.

For development teams, SWE-CoWork points to evaluating the entire course of an assignment: what information the agent found, which dependencies it left unresolved, which requirements it translated into code, and what results it verified before reporting that it was done. Follow-up should cover clarification, implementation, testing, and handoff, based on the problems the team observed in the benchmark.

The evaluated agents worked with an eight-hour budget and an environment with records and colleagues available. Even under those conditions, the best delivery result was about 8 out of 101 tasks for GPT-6 Astra. The question SWE-CoWork leaves for those delegating software work is not just how much code an agent can write, but how many assignments it can complete while satisfying every condition.

Sources: The Decoder, arXiv, TokenWave AI, GitHub