# Neural Architecture Discovery via Autonomous Evolution

arXiv: https://arxiv.org/abs/2507.18074 · HTML: https://mlsanity.com/paper/2507.18074/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 44 references that are in the map.
- First posted on arXiv 24 Jul 2025.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [Jamba: A Hybrid Transformer-Mamba Language Model](https://mlsanity.com/paper/2403.19887/) (2024)
- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents](https://mlsanity.com/paper/2505.22954/) (2025)
- [SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference](https://mlsanity.com/paper/1808.05326/) (2018)
- [cosFormer: Rethinking Softmax in Attention](https://mlsanity.com/paper/2202.08791/) (2022)
- [HGRN2: Gated Linear RNNs with State Expansion](https://mlsanity.com/paper/2404.07904/) (2024)
- [LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimization](https://mlsanity.com/paper/2306.01102/) (2023)
- [metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models](https://mlsanity.com/paper/2407.12844/) (2024)
- [Language Modeling by Language Models](https://mlsanity.com/paper/2506.20249/) (2025)

## Built on by (most cited first, 16 of 16)

- [Meta Context Engineering via Agentic Skill Evolution](https://mlsanity.com/paper/2601.21557/) (2026)
- [Towards Execution-Grounded Automated AI Research](https://mlsanity.com/paper/2601.14525/) (2026)
- [InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Research](https://mlsanity.com/paper/2510.27598/) (2025)
- [AgenticSciML: Collaborative Multi-Agent Systems for Emergent Discovery in Scientific Machine Learning](https://mlsanity.com/paper/2511.07262/) (2025)
- [Can We Predict Before Executing Machine Learning Agents?](https://mlsanity.com/paper/2601.05930/) (2026)
- [Can AI agents conduct open-ended AI research? Early evidence from two case studies](https://mlsanity.com/paper/2607.27191/) (2026)
- [ASI-Evolve: AI Accelerates AI](https://mlsanity.com/paper/2603.29640/) (2026)
- [Improving Language Agents through BREW: Bootstrapping expeRientially-learned Environmental knoWledge](https://mlsanity.com/paper/2511.20297/) (2025)
- [Data Darwinism Part II: DataEvolve -- AI can Autonomously Evolve Pretraining Data Curation](https://mlsanity.com/paper/2603.14420/) (2026)
- [From AI Assistant to AI Scientist: Autonomous Discovery of LLM-RL Algorithms with LLM Agents](https://mlsanity.com/paper/2603.23951/) (2026)
- [Agentic Discovery with Active Hypothesis Exploration for Visual Recognition](https://mlsanity.com/paper/2604.12999/) (2026)
- [Saturating Scaling Laws for Equational Discovery: A Phenomenology of Growth Dynamics in Three Toy Substrates with Two Real-World Replications](https://mlsanity.com/paper/2605.23983/) (2026)
- [AgentFold: Closed-Loop Agentic Search for Protein Folding Model Design](https://mlsanity.com/paper/2608.26747/) (2026)
- [Automated Membership Inference Attacks (AutoMIA): Discovering MIA Signal Computations using LLM Agents](https://mlsanity.com/paper/2603.19375/) (2026)
- [daVinci-LLM:Towards the Science of Pretraining](https://mlsanity.com/paper/2603.27164/) (2026)
- [Belief-Guided Decision Making with Uncertainty Gating in the Game of Go](https://mlsanity.com/paper/2607.26946/) (2026)

## Abstract (arXiv)

Recent progress in LLM agents has advanced the prospect of autonomous research. Yet whether AI can complete difficult long-horizon tasks, especially those that advance AI research itself, remains largely unexplored. We present ASI-Arch, a system for AI-driven AI research that autonomously conducts neural architecture research through a closed-loop research-experiment-analyze-update process. Applied to linear attention, ASI-Arch ran 1,773 iterative experiments and discovered 105 state-of-the-art architectures. Its best architecture improves over DeltaNet by nearly three times the gain achieved by Mamba2. Beyond the final performance gains, we analyze the contributions of different parts of the framework in this hard research setting, shedding light on what enables autonomous progress in complex AI research tasks.