mlsanityData to 14 Oct 2026

How Far Are We From True Auto-Research?

arXiv 2605.19156 ↗First posted 18 May 2026cs.AI cs.CY cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Recent auto-research systems can produce complete papers, but feasibility is not the same as quality, and the field still lacks a systematic study of how good agent-generated papers actually are. We introduce ResearchArena, a minimal scaffold that lets off-the-shelf agents (Claude Code using Opus 4.6, Codex using GPT-5.4, and Kimi Code using K2.5) carry out the full research loop themselves (ideation, experimentation, paper writing, self-refinement) under only lightweight guidance. Across 13 computer science seeds and 3 trials per agent-domain pair, ResearchArena yields 117 agent-generated papers, each evaluated under three complementary lenses: a manuscript-only reviewer (SAR), an artifact-aware peer review (PR) in which agents inspect the workspace alongside the manuscript, and an human conducted meta-review. Under SAR alone the picture is optimistic: Claude Code obtains the highest score, outperforms Analemma's FARS, and matches the weighted-average human ICLR 2025 submission, suggesting that minimally scaffolded agents can produce papers that look competitive on manuscript-only review. Manual inspection, however, reveals this picture is overstated: SAR scores are poorly aligned with its actual acceptance decisions and reward plausible framing without verifying experimental substance. Under artifact-aware PR scores drop sharply, and manual auditing identifies experimental rigor as the major bottleneck, decomposing into three failure modes (fabricated results, underpowered experiments, and plan/execution mismatch) that are highly agent-dependent: Codex 5%/8% paper-vs-artifact mismatch / fabricated references versus Kimi Code 77%/72%, a ~15× spread that tracks distinct research personas the agents develop. None of the 117 agent-generated papers reaches the acceptance bar of a top-tier venue. This suggests that we are still gapped from the true auto-research.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

22 of 27 references.

References · 22

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  8. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  9. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  10. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  11. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
  12. 2025The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Searchcited by 422
10 more
  1. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 366
  2. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  3. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
  4. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 235
  5. 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 210
  6. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  7. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 123
  8. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 72
  9. 2025MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Researchcited by 64
  10. 2025MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?cited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loopscited by 35
  2. 2026NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?cited by 10
  3. 2026How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Taskscited by 8
  4. 2026ResearchStudio-Reel: Automate the Last Mile of Research from Paper to Poster, Video, and Blogcited by 2
  5. 2026Workflow Closure Is Not Scientific Closure in Auto-Research Systemscited by 1
  6. 2026One Reflection Is Not Enough: Self-Correcting Autonomous Research via Multi-Hypothesis Failure Attributioncited by 1
  7. 2026PseudoBench: Measuring How Agentic Auto-Research Fuels Pseudosciencecited by 0
  8. 2026SocioVerse2: A Longitudinal Dynamic Social Simulation Framework under a Human-AI Co-evolutionary Paradigmcited by 0
  9. 2026Large Language Models, Machine-Assisted Cultural Analytics, and the Agentic Turncited by 0
  10. 2026ResearchFlow-Eval: Tracing the Execution Gap in End-to-End Multi-Claim Autoresearchcited by 0