# Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes

arXiv: https://arxiv.org/abs/2605.05724 · HTML: https://mlsanity.com/paper/2605.05724/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 7 May 2026.

## Builds on (most cited first, 13 of 13)

- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [DataComp-LM: In search of the next generation of training sets for language models](https://mlsanity.com/paper/2406.11794/) (2024)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts](https://mlsanity.com/paper/2411.15114/) (2024)
- [AutoML-Zero: Evolving Machine Learning Algorithms From Scratch](https://mlsanity.com/paper/2003.03384/) (2020)

## Built on by (most cited first, 10 of 10)

- [EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learning](https://mlsanity.com/paper/2606.03108/) (2026)
- [The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement](https://mlsanity.com/paper/2609.11873/) (2026)
- [Closed-loop Auto Research for Molecular Property Prediction: Discovering and Certifying Generalizable Improvements](https://mlsanity.com/paper/2606.22731/) (2026)
- [Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer](https://mlsanity.com/paper/2607.17100/) (2026)
- [The Implementation Lottery: Auditing Idea Reliability in Automated Research](https://mlsanity.com/paper/2607.26587/) (2026)
- [Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents](https://mlsanity.com/paper/2609.09219/) (2026)
- [WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents](https://mlsanity.com/paper/2609.27490/) (2026)
- [AlphaDiverse: Post-Training Local Quantitative Research Agents for Diverse Exploration in Alpha Factor Mining](https://mlsanity.com/paper/2609.29014/) (2026)
- [Rules to Tools: Executable Checks for LLM Agents in Scientific Computing](https://mlsanity.com/paper/2610.00313/) (2026)
- [Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts](https://mlsanity.com/paper/2610.00314/) (2026)

## Abstract (arXiv)

We study auto research as a closed empirical loop driven by external measurement. Each submitted trial carries a hypothesis, an executable code edit, an evaluator-owned outcome, and feedback that shapes the next proposal. The output is not a generated paper or a single model checkpoint, but an auditable trajectory of proposals, code diffs, experiments, scores, and failure labels. We instantiate this loop with specialist agents that partition recipe surfaces and share measured lineage across trials. The central empirical finding is that lineage feedback lets agents turn evaluator outcomes, including crashes, budget overruns, size failures, and accuracy-gate misses, into later program-level recipe edits rather than one-shot suggestions. Across 1,197 headline-run trials plus 600 Parameter Golf control trials after one-time setup and launch, humans did not choose proposals, edit recipes, override scores, or repair failed trials during the search. In the three headline runs, the same submitted-trial loop reduces Parameter Golf validation bpb by 0.81%, raises NanoChat-D12 CORE by 38.7%, and reduces CIFAR-10 Airbench96 wallclock by 4.59%, with each task measured by its own external evaluator and legality checks. The trace includes a strict architecture-domain audit of 157 headline-run submissions and program rewrites such as a NanoChat attention-kernel path change. Within this scope the loop autonomously writes code, submits experiments, absorbs feedback, applies and combines known techniques inside each environment, and improves public starting recipes.