mlsanityData to 14 Oct 2026

AI for Auto-Research: Roadmap & User Guide

arXiv 2605.18661 ↗First posted 18 May 2026cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

AI-assisted research is crossing a threshold: fully automated systems can now generate research papers for as little as 15, while long-horizon agents can execute experiments, draft manuscripts, and simulate critique with minimal human input. Yet this productivity frontier exposes a deeper integrity problem: under scientific pressure, even frontier LLMs still fabricate results, miss hidden errors, and fail to judge novelty reliably. Studying developments through April 2026, we present an end-to-end analysis of AI across the complete research lifecycle, organized into four epistemological phases: Creation (idea generation, literature review, coding & experiments, tables & figures), Writing (paper writing), Validation (peer review, rebuttal & revision), and Dissemination (posters, slides, videos, social media, project pages, and interactive agents). We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy: AI excels at structured, retrieval-grounded, and tool-mediated tasks, but remains fragile for genuinely novel ideas, research-level experiments, and scientific judgment. Generated ideas often degrade after implementation, research code lags far behind pattern-matching benchmarks, and end-to-end autonomous systems have not yet consistently reached major-venue acceptance standards. We further show that greater automation can obscure rather than eliminate failure modes, making human-governed collaboration the most credible deployment paradigm. Finally, we provide a structured taxonomy, benchmark suite, and tool inventory, cross-stage design principles, and a practitioner-oriented playbook, with resources maintained at our project page.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 257 of its 338 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

257 of 338 references.

References · 257

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  10. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  11. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  12. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
38 more
  1. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 867
  2. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  3. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  4. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
  5. 2025The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Searchcited by 422
  6. 2025Accelerating scientific discovery with Co-Scientistcited by 401
  7. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  8. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 366
  9. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  10. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 356
  11. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
  12. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 279
  13. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 249
  14. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 235
  15. 2025AIDE: AI-Driven Exploration in the Space of Codecited by 226
  16. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 225
  17. 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 210
  18. 2024Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understandingcited by 188
  19. 2023SciMON: Scientific Inspiration Machines Optimized for Noveltycited by 179
  20. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 178
  21. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  22. 2025ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolutioncited by 174
  23. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 156
  24. 2024Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Modelscited by 154
  25. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 149
  26. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 148
  27. 2025Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)cited by 141
  28. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 140
  29. 2025Towards a Science of Scaling Agent Systemscited by 138
  30. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 137
  31. 2023Emergent autonomous scientific research capabilities of large language modelscited by 132
  32. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
  33. 2024DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoningcited by 125
  34. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 124
  35. 2024AutoSurvey: Large Language Models Can Automatically Write Surveyscited by 119
  36. 2024Mapping the Increasing Use of LLMs in Scientific Paperscited by 118
  37. 2026Learning to Discover at Test Timecited by 115
  38. 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Modelscited by 109

50 most cited of 257 All 257 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026When Safe Skills Collide: Measuring Compositional Risk in Agent Skill Ecosystemscited by 17
  2. 2026DanceOPD: On-Policy Generative Field Distillationcited by 14
  3. 2026How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Taskscited by 8
  4. 2026RMA: an Agentic System for Research-Level Mathematical Problemscited by 7
  5. 2026ARAC: Benchmarking Auto-Research's Alignment and Completeness on End-to-End Researchscited by 1
  6. 2026The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishingcited by 1
  7. 2026Efficiency Matters in Autonomous Researchcited by 0
  8. 2026MARS: Multi-Agent Re-ranking for Repeat-Order Food Delivery Recommendationcited by 0
  9. 2026Bioinfoysis Technical Reportcited by 0
  10. 2026Positioning manuscripts in the scientific landscape with agentic AIcited by 0
  11. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0