# Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

arXiv: https://arxiv.org/abs/2607.17100 · HTML: https://mlsanity.com/paper/2607.17100/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 40 references that are in the map.
- First posted on arXiv 19 Jul 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 17 of 17)

- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023)
- [AIDE: AI-Driven Exploration in the Space of Code](https://mlsanity.com/paper/2502.13138/) (2025)
- [Data-centric Artificial Intelligence: A Survey](https://mlsanity.com/paper/2303.10158/) (2023)
- [Towards End-to-End Automation of AI Research](https://mlsanity.com/paper/2606.15497/) (2026)
- [AutoML-Zero: Evolving Machine Learning Algorithms From Scratch](https://mlsanity.com/paper/2003.03384/) (2020)
- [Predicting materials properties without crystal structure: Deep representation learning from stoichiometry](https://mlsanity.com/paper/1910.00617/) (2019)
- [SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks](https://mlsanity.com/paper/2604.20087/) (2026)
- [Matbench Discovery -- A framework to evaluate machine learning crystal stability predictions](https://mlsanity.com/paper/2308.14920/) (2023)
- [Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes](https://mlsanity.com/paper/2605.05724/) (2026)
- [AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentation](https://mlsanity.com/paper/2605.28655/) (2026)
- [Revision or Re-Solving? Decomposing Second-Pass Gains in Multi-LLM Pipelines](https://mlsanity.com/paper/2604.01029/) (2026)
- [Closed-loop Auto Research for Molecular Property Prediction: Discovering and Certifying Generalizable Improvements](https://mlsanity.com/paper/2606.22731/) (2026)

## Built on by (most cited first, 5 of 5)

- [The Implementation Lottery: Auditing Idea Reliability in Automated Research](https://mlsanity.com/paper/2607.26587/) (2026)
- [Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents](https://mlsanity.com/paper/2609.09219/) (2026)
- [WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents](https://mlsanity.com/paper/2609.27490/) (2026)
- [AlphaDiverse: Post-Training Local Quantitative Research Agents for Diverse Exploration in Alpha Factor Mining](https://mlsanity.com/paper/2609.29014/) (2026)
- [Rules to Tools: Executable Checks for LLM Agents in Scientific Computing](https://mlsanity.com/paper/2610.00313/) (2026)

## Abstract (arXiv)

Auto Research uses language-model agents to propose, implement, and evaluate machine-learning changes in a closed loop, but is usually judged by its terminal pipeline. A terminal score cannot reveal which technical decision produced a gain or distinguish a reusable discovery from a change adapted to development feedback. We introduce intervention-centered Auto Research, which validates research decisions rather than only final artifacts and makes their reliability measurable. Feature, Model, Representation, and Data axes are searched independently with inner five-fold feedback. Each axis winner is frozen before an outer-holdout matrix compares all alternatives on evidence the loop never sees. Across 701 agent-executed attempts spanning ten Matbench endpoints, outer evidence confirms the selected intervention on nine of ten endpoints and preserves 89.3% of non-tied intervention orderings. It also rejects an aggregate Representation gain that inner feedback endorsed. The resulting matrix reveals an information-dependent hierarchy. Composition-only tasks support several routes to improvement, whereas structure-informed tasks favor local geometry features and complementary tree ensembles. A subsequent compatibility test combines already frozen Feature and Model code without further search or tuning and raises mean outer-holdout improvement from 19.0% to 26.3%. By validating decisions rather than only artifacts, this design turns adaptive search into reusable evidence wherever agents propose executable alternatives against a fixed evaluator.