mlsanityData to 14 Oct 2026

HCAST: Human-Calibrated Autonomy Software Tasks

arXiv 2503.17354 ↗First posted 21 Mar 2025cs.AIcited by 24

Not on the map. Why?

Abstract (arXiv)

To understand and predict the societal impacts of highly autonomous AI systems, we need benchmarks with grounding, i.e., metrics that directly connect AI performance to real-world effects we care about. We present HCAST (Human-Calibrated Autonomy Software Tasks), a benchmark of 189 machine learning engineering, cybersecurity, software engineering, and general reasoning tasks. We collect 563 human baselines (totaling over 1500 hours) from people skilled in these domains, working under identical conditions as AI agents, which lets us estimate that HCAST tasks take humans between one minute and 8+ hours. Measuring the time tasks take for humans provides an intuitive metric for evaluating AI capabilities, helping answer the question "can an agent be trusted to complete a task that would take a human X hours?" We evaluate the success rates of AI agents built on frontier foundation models, and we find that current agents succeed 70-80% of the time on tasks that take humans less than one hour, and less than 20% of the time on tasks that take humans more than 4 hours.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 23 of its 30 references that are in the map. First posted on arXiv 21 Mar 2025.

Builds on

23 of 30 references.

References · 23

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2020Extracting Training Data from Large Language Modelscited by 2,189
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  6. 2016Concrete Problems in AI Safetycited by 1,642
  7. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  8. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  9. 2025Humanity's Last Examcited by 491
  10. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 366
  11. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
  12. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
11 more
  1. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  2. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  3. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 164
  4. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 132
  5. 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?cited by 125
  6. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 121
  7. 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 105
  8. 2024DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agentscited by 100
  9. 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 93
  10. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Modelscited by 58
  11. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 18.

24 papers.

Sort:

By area:

  1. 2026PostTrainBench: Can LLM Agents Automate LLM Post-Training?cited by 64
  2. 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systemscited by 59
  3. 2025A Rosetta Stone for AI Benchmarkscited by 36
  4. 2026Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Gradingcited by 31
  5. 2026Toward Generalist Autonomous Research via Hypothesis-Tree Refinementcited by 22
  6. 2025Remote Labor Index: Measuring AI Automation of Remote Workcited by 20
  7. 2025HiBayES: A Hierarchical Bayesian Modeling Framework for AI Evaluation Statisticscited by 18
  8. 2026Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Modelscited by 12
  9. 2025TextAtari: 100K Frames Game Playing with Language Agentscited by 6
  10. 2026BRIDGE: Predicting Human Task Completion Time From Model Performancecited by 6
  11. 2026LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoningcited by 6
  12. 2026Sandboxed Coding Agents are Competitive Omni-modal Task Solverscited by 3
12 more
  1. 2026SentinelBench: A Benchmark for Long-Running Monitoring Agentscited by 3
  2. 2025A Hierarchical Imprecise Probability Approach to Reliability Assessment of Large Language Modelscited by 2
  3. 2026Crashing Waves vs. Rising Tides: Findings on AI Automation from Thousands of Worker Evaluations of Labor Market Taskscited by 1
  4. 2026Questionnaire Responses Do not Capture the Safety of AI Agentscited by 0
  5. 2026Predicting Task Difficulty Without Rolloutscited by 0
  6. 2026TraceML: An Empirical Analysis of Human-Agent Planning in Machine Learning Developmentcited by 0
  7. 2026The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Taskscited by 0
  8. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0
  9. 2026Learning What to Investigate Next: Meta-Reasoning for Long-Horizon Research Agentscited by 0
  10. 2026Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?cited by 0
  11. 2026AgentTime: Can Agents Estimate and Control Their Own Runtime?cited by 0
  12. 2026AgentStripes: An Evidence-Graded Maturity Model for AI Agentscited by 0