mlsanityData to 7 Oct 2026

Measuring AI R&D Automation

arXiv 2603.03992 ↗First posted 4 Mar 2026cs.CY cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

The automation of AI R&D (AIRDA) could have significant implications, but its extent and ultimate effects remain uncertain. We need empirical data to resolve these uncertainties, but existing data (primarily capability benchmarks) may not reflect real-world automation or capture its broader consequences, such as whether AIRDA accelerates capabilities more than safety progress or whether our ability to oversee AI R&D can keep pace with its acceleration. To address these gaps, this work proposes metrics to track the extent of AIRDA and its effects on AI progress and oversight. The metrics span dimensions such as capital share of AI R&D spending, researcher time allocation, and AI subversion incidents, and could help decision makers understand the potential consequences of AIRDA, implement appropriate safety measures, and maintain awareness of the pace of AI development. We recommend that companies and third parties (e.g. non-profit research organisations) start to track these metrics, and that governments support these efforts.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 30 of its 72 references that are in the map. First posted on arXiv 4 Mar 2026.

Builds on

30 of 72 references.

References · 30

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  3. 2024Alignment faking in large language modelscited by 316
  4. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  5. 2024Frontier Models are Capable of In-context Schemingcited by 252
  6. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  7. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 153
  8. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 111
  9. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 104
  10. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  11. 2025Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivitycited by 93
  12. 2025Stress Testing Deliberative Alignment for Anti-Scheming Trainingcited by 71
18 more
  1. 2025Auditing language models for hidden objectivescited by 68
  2. 2020Measuring the Algorithmic Efficiency of Neural Networkscited by 51
  3. 2024Sabotage Evaluations for Frontier Modelscited by 45
  4. 2025A sketch of an AI control safety casecited by 36
  5. 2025A Rosetta Stone for AI Benchmarkscited by 29
  6. 2023AI capabilities can be significantly improved without expensive retrainingcited by 28
  7. 2024AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracycited by 21
  8. 2025Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexitycited by 16
  9. 2025Predicting Empirical AI Research Outcomes with Language Modelscited by 14
  10. 2024Scenarios for the Transition to AGIcited by 13
  11. 2025CTRL-ALT-DECEIT: Sabotage Evaluations for Automated AI R&Dcited by 11
  12. 2024Societal Adaptation to Advanced AIcited by 10
  13. 2026Frontier AI Auditing: Toward Rigorous Third-Party Assessment of Safety and Security Practices at Leading AI Companiescited by 10
  14. 2025AI Behind Closed Doors: a Primer on The Governance of Internal Deploymentcited by 8
  15. 2025On the Origin of Algorithmic Progress in AIcited by 5
  16. 2025Bare Minimum Mitigations for Autonomous AI Developmentcited by 3
  17. 2025Note on Selection Bias in Observational Estimates of Algorithmic Progresscited by 3
  18. 2025Preparing for the Intelligence Explosioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineeringcited by 7
  2. 2026Risk Reporting for Developers' Internal AI Model Usecited by 0
  3. 2026The Closing Window: How Governments Could Lose Their Ability to Restrain Advanced AIcited by 0
  4. 2026The Dynamics of Intelligence Explosionscited by 0
  5. 2026Recursive Criticality of AI Self-Improvementcited by 0
  6. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0
  7. 2026Comprehension Audits to Mitigate Risks from Automated AI Researchcited by 0