mlsanityData to 14 Oct 2026

Measuring AI Ability to Complete Long Software Tasks

arXiv 2503.14499 ↗First posted 18 Mar 2025cs.AI cs.LGcited by 156

Not on the map. Why?

Cited by 73 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite rapid progress on AI benchmarks, the real-world meaning of benchmark performance remains unclear. To quantify the capabilities of AI systems in terms of human capabilities, we propose a new metric: 50%-task-completion time horizon. This is the time humans typically take to complete tasks that AI models can complete with 50% success rate. We first timed humans with relevant domain expertise on a combination of RE-Bench, HCAST, and 66 novel shorter tasks. On these tasks, current frontier AI models such as Claude 3.7 Sonnet have a 50% time horizon of around 50 minutes. Furthermore, frontier AI time horizon has been doubling approximately every seven months since 2019, though the trend may have accelerated in 2024. The increase in AI models' time horizons seems to be primarily driven by greater reliability and ability to adapt to mistakes, combined with better logical reasoning and tool use capabilities. We discuss the limitations of our results -- including their degree of external validity -- and the implications of increased autonomy for dangerous capabilities. If these results generalize to real-world software tasks, extrapolation of this trend predicts that within 5 years, AI systems will be capable of automating many software tasks that currently take humans a month.

Key facts. Cited by 156 papers in the map (data through 14 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

30 of 46 references.

References · 30

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  10. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  12. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
18 more
  1. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  2. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  3. 2021Measuring Coding Challenge Competence With APPScited by 902
  4. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 366
  5. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
  6. 2025AIDE: AI-Driven Exploration in the Space of Codecited by 226
  7. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  8. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 164
  9. 2022Compute Trends Across Three Eras of Machine Learningcited by 163
  10. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 132
  11. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 128
  12. 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?cited by 125
  13. 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 105
  14. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Modelscited by 58
  15. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Modelscited by 39
  16. 2024How predictable is language model benchmark performance?cited by 37
  17. 2025Forecasting Frontier Language Model Agent Capabilitiescited by 5
  18. 2025Mapping AI Benchmark Data to Quantitative Risk Estimates Through Expert Elicitationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 49; 2026: 107.

156 papers.

Sort:

By area:

  1. 2025Scaling Long-Horizon LLM Agent via Context-Foldingcited by 148
  2. 2025ACON: Optimizing Context Compression for Long-horizon LLM Agentscited by 136
  3. 2025xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluationscited by 120
  4. 2025Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivitycited by 101
  5. 2025Kosmos: An AI Scientist for Autonomous Discoverycited by 92
  6. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 40
  7. 2026Dive into Claude Code: The Design Space of Today's and Future AI Agent Systemscited by 38
  8. 2025A Rosetta Stone for AI Benchmarkscited by 36
  9. 2025Probing and Steering Evaluation Awareness of Language Modelscited by 35
  10. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 31
  11. 2025Characterizing AI Agents for Alignment and Governancecited by 30
  12. 2025Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agentscited by 27
38 more
  1. 2025Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competitioncited by 24
  2. 2025Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMscited by 22
  3. 2026Odysseys: Benchmarking Web Agents on Realistic Long Horizon Taskscited by 21
  4. 2025How to evaluate control measures for LLM agents? A trajectory from today to superintelligencecited by 20
  5. 2025Imitation Learning for Multi-turn LM Agents via On-policy Expert Correctionscited by 19
  6. 2026Recursive Synthesis for Long-Horizon Terminal Taskscited by 19
  7. 2025HiBayES: A Hierarchical Bayesian Modeling Framework for AI Evaluation Statisticscited by 18
  8. 2026Building Effective AI Coding Agents for the Terminal: Scaffolding, Harness, Context Engineering, and Lessons Learnedcited by 18
  9. 2025Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewritingcited by 13
  10. 2025WOLF: Werewolf-based Observations for LLM Deception and Falsehoodscited by 13
  11. 2026LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Taskscited by 13
  12. 2026Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Modelscited by 12
  13. 2025ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?cited by 11
  14. 2026Agent psychometrics: Task-level performance prediction in agentic coding benchmarkscited by 11
  15. 2026The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity?cited by 10
  16. 2026How to Interpret Agent Behaviorcited by 10
  17. 2026Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentioncited by 10
  18. 2025h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learningcited by 9
  19. 2026AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarioscited by 9
  20. 2026Agentic Uncertainty Reveals Agentic Overconfidencecited by 9
  21. 2026FACET: Preserving Source Intent and Executable State in Terminal Task Synthesiscited by 9
  22. 2025Benchmarking Misuse Mitigation Against Covert Adversariescited by 8
  23. 2025Position: The Current AI Conference Model is Unsustainable! Diagnosing the Crisis of Centralized AI Conferencecited by 8
  24. 2025Orchestrating Human-AI Teams: The Manager Agent as a Unifying Research Challengecited by 8
  25. 2025Efficient numeracy in language models through single-token number embeddingscited by 7
  26. 2025What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversitycited by 7
  27. 2026Propensity Inference: Environmental Contributors to LLM Behaviourcited by 7
  28. 2026Your Agentic LLMs Secretly Encode Indirect Prompt-Injection Exposure in Hidden Statescited by 7
  29. 2026Can Agent Memory Systems Track Evolving State?cited by 7
  30. 2025TextAtari: 100K Frames Game Playing with Language Agentscited by 6
  31. 2026Attack Selection Reduces Safety in Concentrated AI Control Settings against Trusted Monitoringcited by 6
  32. 2026BRIDGE: Predicting Human Task Completion Time From Model Performancecited by 6
  33. 2026SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitorscited by 6
  34. 2025Evaluating AI cyber capabilities with crowdsourced elicitationcited by 5
  35. 2025Breakpoint: Scalable evaluation of system-level reasoning in LLM code agentscited by 5
  36. 2025Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Frameworkcited by 5
  37. 2025Interaction as Intelligence: Deep Research With Human-AI Partnershipcited by 5
  38. 2025Leveraging LLM Agents for Automated Video Game Testingcited by 5

50 most cited of 156 All 156 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents
  2. 2025An Approach to Grounding AI Model Evaluations in Human-derived Criteria
  3. 2025The Geometry of Benchmarks: A New Path Toward AGI
  4. 2026SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?
  5. 2026Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence
  6. 2026Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields
  7. 2025General Scales Unlock AI Evaluation with Explanatory and Predictive Power
  8. 2026How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making