mlsanityData to 7 Oct 2026

WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents

arXiv 2609.27490 ↗First posted 23 Sep 2026cs.AI cs.LGcited by 0

On the map: LLM agents and tool use → Software engineering coding agents also fits: Agent skill learning and evolution

Abstract (arXiv)

AI research agents must predict the effects of computational changes after budgeted experiments. WhatWorkedBench evaluates this experimental understanding through a delivered response surface of configuration scores. Agents inspect workflow code and buy measurements; exhaustive CPU references score conditional component effects, mean pair interactions, configuration choice, and delivery. The catalog spans 36 task conditions, 30 sources, 8 workflow families, and 1248 indexed configuration records, with 4,206 numerical controls. At eight purchased measurements plus two free anchors, pair-effect ridge selects an exact optimum on 15 of 22 sources; 13 of these cases have at least one conditional-effect error exceeding 10% of the task utility range. Shared-estimator comparisons measure acquisition and reconstruction on common observations. In a prospective typed study on 12 four-factor sources, DeepSeek Flash submits 12/12 direct tables and gains 0.147 recovery over a Gaussian process (GP) fitted to the same observations. Pro delivers 11/12 artifacts, with an all-attempt GP difference of -0.001 and a delivered-only difference of +0.063. On six prespecified new agent-evaluation sources, Flash and Pro gains are 0.149 and 0.074. In eight typed six-factor episodes, seven final tables satisfy verified code equivalences. Four fresh agents pass all six registered rules through named estimators and deliver consistent tables at 0.682 recovery versus 0.710 for separate direct-table runs. WhatWorkedBench links acquisition, inference, program structure, and delivery.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 40 of its 55 references that are in the map. First posted on arXiv 23 Sep 2026.

Builds on

40 of 55 references.

References · 40

  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  2. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  3. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 339
  4. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  5. 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 229
  6. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 208
  7. 2024Large Language Models to Enhance Bayesian Optimizationcited by 127
  8. 2023Modern Bayesian Experimental Designcited by 121
  9. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  10. 2021Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Designcited by 103
  11. 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Modelscited by 97
  12. 2024DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agentscited by 89
28 more
  1. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 47
  2. 2025AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suitecited by 45
  3. 2026SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Taskscited by 45
  4. 2025EXP-Bench: Can AI Conduct AI Research Experiments?cited by 36
  5. 2025InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Researchcited by 18
  6. 2025RExBench: Can coding agents autonomously implement AI research extensions?cited by 17
  7. 2025BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discoverycited by 15
  8. 2026ResearchGym: Evaluating Language Model Agents on Real-World AI Researchcited by 15
  9. 2025Gravity-Bench-v1: A Benchmark on Gravitational Physics Discovery for Agentscited by 13
  10. 2026FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insightscited by 13
  11. 2025AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Researchcited by 12
  12. 2026SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agentscited by 12
  13. 2025Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Labcited by 11
  14. 2026Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipescited by 10
  15. 2026Benchmarking AI Agents for Addressing Scientific Challenges Across Scalescited by 10
  16. 2026DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinkingcited by 8
  17. 2026Closed-loop Auto Research for Molecular Property Prediction: Discovering and Certifying Generalizable Improvementscited by 6
  18. 2026CausalGame: Benchmarking Causal Thinking of LLM Agents in Gamescited by 5
  19. 2026Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfercited by 5
  20. 2026AutoLLMResearch: Training Research Agents for Automating LLM Experiment Configuration - Learning from Cheap, Optimizing Expensivecited by 4
  21. 2026The Implementation Lottery: Auditing Idea Reliability in Automated Researchcited by 4
  22. 2025AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Researchcited by 2
  23. 2026Hierarchical Experimentalist Agentscited by 2
  24. 2026Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Paperscited by 2
  25. 2026CAFE: A Compound-AI Factorial Evaluation Frameworkcited by 1
  26. 2026Can LLM design high-quality experiments? A Comprehensive and Systematic Benchmark on Autonomous Experimental Designcited by 1
  27. 2026Science sandboxes measure the scientific capability of AI agentscited by 1
  28. 2026Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agentscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era
  2. 2026Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI
  3. 2026LABBench2: An Improved Benchmark for AI Systems Performing Biology Research
  4. 2024WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting
  5. 2026ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences
  6. 2026Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation
  7. 2026A case study of evaluating AI agents on a neuroscience data-to-discovery pipeline
  8. 2026Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies