Models evaluated by METR roughly double every 3.6 months the task duration they can handle with a 50% probability of success, according to an update published on November 20. But that figure does not answer the question that concerns the organization: whether AI can accelerate research to the point that people are unable to evaluate and respond to its risks in time.
METR, an AI research organization, evaluated public models up to Opus 5.5 and GPT-6 Astra. Its conclusion combines two distinct signals: strong progress on programming and optimization tasks with clear metrics, and uncertainty about the acceleration of AI research. The organization says the available public data is not enough to rule out drastic acceleration in the coming years.
By drastic acceleration, METR means a situation in which capabilities progress so quickly that people can no longer «meaningfully understand, assess, and react to catastrophic risks posed by new AI systems». The quote expresses the risk threshold it studies; it is not a prediction that this scenario is already happening.
“to meaningfully understand, assess, and react to catastrophic risks posed by new AI systems”
— METR
Progress is clearest when there is a metric
On programming and optimization tasks with defined objectives and verifiable results, models often match or outperform skilled people. METR also found cases in which its models ranked in high percentiles against groups of more than ten specialists, though not necessarily above the best performers. The observed frontier is therefore more precise than saying that AI already outperforms experts in general.
The distinction matters: on a verifiable problem, an agent can compare its result with a score, a test, or an optimization objective. On open-ended tasks, where there is no rapid signal of progress, the organization found poorer judgment than that of human experts. In one of them, model-based evaluators overestimated the quality of an answer when it had been produced by another model of similar or greater capability.
METR divides its evaluations into three sets. Time Horizon 1.1 brings together software engineering tasks; MirrorCode focuses on replicating software from the command line; and TH2 adds both verifiable objectives and research and evaluation tasks that are difficult to score. This mix broadens the focus, but it also means that an aggregate figure depends on which tasks are included.
| Set | Tasks | Focus |
|---|---|---|
| Time Horizon 1.1 | 222 | Tasks including SWAA, HCAST, and RE-Bench. |
| MirrorCode | 75 | Software replication using command-line tools. |
| TH2 | 36, grouped into 12 families | Optimization, machine learning, games, open-ended research, and judgment. |
For the combined Time Horizon 1.1 and TH2 set, METR estimates that the time horizon doubles every approximately 3.6 months. TH2 contains more difficult tasks, so the 50% success probability for the same model corresponds to a shorter horizon in the combined set than in Time Horizon 1.1 alone. This is not a direct measure of productivity in companies or a universal timeline for AI capabilities.
A rapid trend, but no demonstrated acceleration
The 3.6-month figure measures how much longer the tasks models can complete in METR’s tests are becoming. In another measurement, the METR Capability Index, which tracks the capabilities of public models over time, the organization does not observe a marked overall acceleration or slowdown among the models tested.
METR does see uncertain indications of changes in the trends of some developers and under certain conditions, but warns that these depend on methodological choices. It also finds that, when Time Horizon 1.1 is excluded, doubling times are shorter in a verifiable portion of TH2. That could be consistent with recent acceleration that the combination of sets does not reveal, although METR notes that other explanations are possible.
The prudent interpretation is not that every model will double its usefulness each quarter. The estimate is derived from a specific set of tasks and how success on them is measured. METR leaves two central questions open: what capabilities would be needed to drastically accelerate AI research, and how quickly those capabilities are improving.

Evaluation is also falling behind
The problem is not only that capabilities are changing rapidly. METR warns that cheap, automated, and easy-to-repeat tests can overestimate real-world performance if they closely resemble the environments used to train models. When a system learns to optimize a signal that is similar to the evaluation, a high score may tell us less about its performance in different situations.
The organization calls an estimate of the cost of creating, running, and scoring an evaluation task «Lifecycle Cost». In its data, models perform somewhat worse on tasks that are more costly to turn into reinforcement training environments, even when accounting for the estimated time a person would take. Nevertheless, METR observes similar rates of improvement on high- and low-cost tasks, as well as progress on the more costly ones.
The result describes an uncomfortable trade-off for those evaluating models: simple tests provide quantitative and affordable signals, but may be less representative; more realistic tasks tend to require more work to prepare and judge. METR tried to address this problem with manually scored tasks and tests that it expects to be less like training environments, although the organization acknowledges that measuring generalizable capabilities remains difficult.
What this means for teams deploying agents
For development teams, the results support a practical distinction between tasks with clear acceptance criteria and tasks where quality depends on context or judgment. Agents seem more reliable in the first group: code with tests, numerical objectives, or results that can be checked quickly. In the second, the lack of a clear signal weakens both automated evaluation and confidence that the model itself can assess its work well.
This also affects how a company compares models or decides where to use them. A score on an inexpensive, repeatable evaluation can be useful, but it does not by itself show that the system will perform just as well in a real workflow. METR’s warning is especially relevant when testing and training share characteristics: the result may reflect familiarity with that kind of environment, not necessarily transferable capability.
The update also studies agent swarms, meaning multiple agents working in parallel. METR found them useful compared with a strategy of choosing the best result from several runs on verifiable tasks with clear metrics and rapid feedback. For other types of tasks, their effects were uncertain or negative. Parallelization, therefore, does not appear to be a general improvement independent of the problem.
The organization also changed how it builds its human baselines. It recruited more experienced evaluators, but found it difficult to get enough participants for specialized tasks. Direct human measurements account for only 9% of estimates of how long a person would take; for the rest, METR relies on methods based on public records, expert input, and extrapolations. This limitation adds uncertainty when comparing task duration with human performance.
The operational conclusion is limited: improvement on tasks with clear signals is rapid, but it does not amount to reliable autonomy in work where success takes time to emerge or cannot be easily verified. The remaining question is whether models will also improve at this kind of judgment, or whether progress will continue to concentrate on objectives that are easy to measure.
Sources: The Decoder, METR, New Runtime