GPT-6 Astra completed only 26.4% of Co-Design Bench tasks, despite meeting an average of 69.9% of the acceptance criteria. The result, reported in a benchmark built from requests made during real coding sessions, describes a significant gap: an agent can handle a large part of a task and still fail to deliver a finished change.
The difference between the two figures matters more than a model ranking. In a project, it is not enough for some parts of a request to work: if a requirement remains unresolved, someone has to spot what is missing and decide how to fix it. The results show that partial satisfaction is much more common than complete delivery.
From real instructions to applications that must work
Co-Design Bench was developed by researchers from Alibaba and several Asian universities. The dataset contains 273 tasks selected from 37,463 deduplicated queries across 26,314 coding sessions, and evaluates them using 2,331 hidden acceptance criteria written by experts. Rather than asking models to solve isolated exercises, it asks them to revise existing applications based on user instructions.
The dataset divides the work into two types. Multimodal Editing includes 100 tasks involving localized changes, sometimes supported by images, audio, video, or documents. Repository Editing comprises 173 assignments that require coordinating changes across different parts of a project. Requests can range from extracting a figure from a receipt and updating dependent totals to reorganizing an application.
This approach tests more than code generation. The agent must understand what the person is asking for, connect the instruction to the available files or resources, change the interface, and ensure the resulting functionality works coherently. The benchmark groups these demands into three dimensions: grounded understanding, interface transformation, and functional implementation.
The tasks come from real sessions, and the authors favored cases in which users had to rephrase a request or revert changes. They therefore do not represent a uniform sample of all coding assignments: they concentrate on situations that had already caused problems in practice. This is a deliberate choice to measure where agents stumble, although it also limits what can be inferred about simpler tasks.
Partial success is not the same as delivery
The ten evaluated models worked in the same execution environment. GPT-6 Astra led the comparison with an average of 69.9% of criteria satisfied, but fully resolved only a little more than one in four tasks. The per-criterion score recognizes partial progress; the overall success rate requires the agent to meet every requirement in a task.
For a team using an agent, that distinction changes how to interpret the results. A high average score may suggest that the model is capable without showing how many deliverables are ready to accept as-is. If a task includes both correct changes and one outstanding item, its practical value depends on someone reviewing each requirement and checking the final behavior.
The protocol attempts to prevent an apparent implementation from passing as a functional one. An AI evaluator inspects the application in a browser and scores hidden criteria; afterward, another model reviews positive verdicts and looks for issues such as hard-coded results or interfaces that only simulate working. The evaluations combine code-based tests with browser evidence, without requiring a reference implementation or target screenshot.
This method gives more weight to what the application does than to whether the code looks plausible. However, it still depends on automated evaluators, which can make mistakes. In addition, Co-Design Bench evaluates one fixed request per task: it does not include follow-up questions or user corrections during execution. In a real collaboration, those clarifications could change the outcome, but the benchmark does not measure that exchange.
Combined assignments and the cost of more compute
The most consistent weakness appears when a request combines several kinds of changes. All ten models performed worse on repository edits combining three or four types of work than on those limited to one or two. In real projects, a revision may require simultaneous changes to design, content, layout, and interaction; the benchmark indicates that coordinating these elements is harder than handling them separately.
This helps explain why a convincing demonstration of a specific feature is not enough to predict how an agent will perform on a broad request. A visual change may look right while the related interaction fails, or a figure may be updated in one view but not in others. The dataset includes assignments of this kind, where success depends on keeping connected parts consistent.
Spending more resources did not automatically solve the problem either. According to The Decoder, GLM-5.2 used almost six times as many tokens as GPT-6 Astra on small changes and still scored significantly lower. This does not establish that more compute never helps; it does show that, in this comparison, the additional expenditure did not translate into better results.
Rankings also depend on the execution environment—the set of tools an agent uses to read, modify, and run code—not just on the model. In a separate test, Qwen3.8-27B scored 75% or 56% on broad revisions, depending on the environment used. The relative order of two models could also change. A table that lists only the model name can therefore obscure a relevant part of the evaluated setup.
For companies comparing tools, the practical implication is clear: evaluate the model and its environment as a combination. It is also more informative to test tasks representative of your own work and measure complete deliverables, rather than relying on average criterion scores.
What changes compared with saturated benchmarks
The authors justify the new test by noting that several earlier benchmarks have become too easy for agents capable of running, checking, and correcting their own code. In some, these systems score above 99%, leaving little room to distinguish between models. Co-Design Bench aims to restore that margin with existing applications, less controlled instructions, and checks on the result at runtime.
The comparison with WebCompass-edit presented by The Decoder illustrates the gap: agentic coding models approach near-perfect results on that benchmark, while Co-Design Bench leaves a gap of more than 35 percentage points. This does not mean that any one test is the definitive measure of quality; it shows that the evaluation design determines which capabilities are visible.
Caution also applies to Co-Design Bench scores. The Decoder notes that a METR study found actual acceptance of AI-generated code by project maintainers to be about 24 percentage points below SWE-bench Verified scores. This is a reference to the possible gap between passing a test and having a team accept the result, not a direct conversion of Co-Design Bench's success rate into a productivity rate.
The project's public repository also distinguishes the full experiment from a subset prepared for public use: the latter contains 96 multimodal editing tasks and 521 criteria, not the study's 273 tasks. Its scores describe a different population and should be identified as such. Keeping the evaluation criteria separate from the files the agent receives is also part of the procedure: if the system knew the rubric in advance, the test would measure something else.
The most useful takeaway for software development is not that agents cannot code, but that resolving most of the points in a request is still a long way from completing the assignment. In Co-Design Bench, even the leader left most of the evaluated tasks incomplete. For any company using these tools on existing applications, the immediate question is whether its checks verify every requirement in the final application or accept as complete a change that only seems almost ready.
