mlsanityData to 7 Oct 2026

AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research

arXiv 2507.08038 ↗First posted 9 Jul 2025cs.CL cs.AIcited by 2

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Language model agents are increasingly used to automate scientific research, yet evaluating their scientific contributions remains a challenge. A key mechanism to obtain such insights is through ablation experiments. To this end, we introduce AblationBench, a benchmark suite for evaluating agents on ablation planning tasks in empirical AI research. It includes two tasks: AuthorAblation, which helps authors propose ablation experiments based on a method section and contains 83 instances, and ReviewerAblation, which helps reviewers find missing ablations in a full paper and contains 350 instances. For both tasks, we develop LM-based judges that serve as an automatic evaluation framework. Our experiments with frontier LMs show that these tasks remain challenging, with the best-performing LM system identifying only 45% of the original ablations on average, below human-level performance. We observe an inverse performance trend between the author and reviewer tasks, which we attribute to differences in model grounding. Lastly, we analyze the limitations of current LMs on these tasks, and find that chain-of-thought prompting outperforms an agent-based approach. Our data is available on https://huggingface.co/collections/ai-coscientist/ablationbench, and our code is available on https://github.com/ai-scientist-bench/ablation-bench .

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 47 of its 80 references that are in the map. First posted on arXiv 9 Jul 2025.

Builds on

47 of 80 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agents20 Feb
  2. 2025CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories10 Feb
  3. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts22 Nov
  4. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering9 Oct
  5. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark17 Sep
  6. 2024Language agents achieve superhuman synthesis of scientific knowledge10 Sep
  7. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers6 Sep
  8. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation5 Oct

Other references · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  6. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  8. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  9. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  10. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  11. 2024Understanding the planning of LLM agents: A surveycited by 414
  12. 2023GPTScore: Evaluate as You Desirecited by 357
27 more
  1. 2025Accelerating scientific discovery with Co-Scientistcited by 353
  2. 2020GPT-GNN: Generative Pre-Training of Graph Neural Networkscited by 302
  3. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  4. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  5. 2020Real-Time High-Resolution Background Mattingcited by 172
  6. 2021Delving into Deep Imbalanced Regressioncited by 157
  7. 2024Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Modelscited by 150
  8. 2020Neural-Pull: Learning Signed Distance Functions from Point Clouds by Learning to Pull Space onto Surfacescited by 146
  9. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  10. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 111
  11. 2018Troubling Trends in Machine Learning Scholarshipcited by 107
  12. 2023Learning Performance-Improving Code Editscited by 103
  13. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 91
  14. 2025HalluLens: LLM Hallucination Benchmarkcited by 88
  15. 2023UniPAD: A Universal Pre-training Paradigm for Autonomous Drivingcited by 73
  16. 2024AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitionscited by 70
  17. 2021SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videoscited by 64
  18. 2019Ablation Studies in Artificial Neural Networkscited by 58
  19. 2024LitSearch: A Retrieval Benchmark for Scientific Literature Searchcited by 47
  20. 2024CiteME: Can Language Models Accurately Cite Scientific Claims?cited by 45
  21. 2024Can LLMs Replace Manual Annotation of Software Engineering Artifacts?cited by 40
  22. 2024APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inferencecited by 29
  23. 2024Accelerating Retrieval-Augmented Language Model Serving with Speculationcited by 20
  24. 2021DisCoDisCo at the DISRPT2021 Shared Task: A System for Discourse Segmentation, Classification, and Connective Detectioncited by 15
  25. 2023Preserving Commonsense Knowledge from Pre-trained Language Models via Causal Inferencecited by 13
  26. 2024Growth Inhibitors for Suppressing Inappropriate Image Concepts in Diffusion Modelscited by 10
  27. 2025Self-Composing Policies for Scalable Continual Reinforcement Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026AblateCell: A Reproduce-then-Ablate Agent for Virtual Cell Repositoriescited by 0
  2. 2026WhatWorkedBench: Benchmarking Experimental Understanding in AI Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research
  2. 2025AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite
  3. 2026AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era
  4. 2026ResearchGym: Evaluating Language Model Agents on Real-World AI Research
  5. 2026FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights
  6. 2026LABBench2: An Improved Benchmark for AI Systems Performing Biology Research
  7. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
  8. 2026ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research