mlsanityData to 7 Oct 2026

Neural Architecture Discovery via Autonomous Evolution

arXiv 2507.18074 ↗First posted 24 Jul 2025cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Recent progress in LLM agents has advanced the prospect of autonomous research. Yet whether AI can complete difficult long-horizon tasks, especially those that advance AI research itself, remains largely unexplored. We present ASI-Arch, a system for AI-driven AI research that autonomously conducts neural architecture research through a closed-loop research-experiment-analyze-update process. Applied to linear attention, ASI-Arch ran 1,773 iterative experiments and discovered 105 state-of-the-art architectures. Its best architecture improves over DeltaNet by nearly three times the gain achieved by Mamba2. Beyond the final performance gains, we analyze the contributions of different parts of the framework in this hard research setting, shedding light on what enables autonomous progress in complex AI research tasks.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 30 of its 44 references that are in the map. First posted on arXiv 24 Jul 2025.

Builds on

30 of 44 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  11. 2020Rethinking Attention with Performerscited by 1,584
  12. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
18 more
  1. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  2. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  3. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  4. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  5. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 763
  6. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  7. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  8. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  9. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  10. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  11. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  12. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  13. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 294
  14. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  15. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  16. 2023LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimizationcited by 86
  17. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Modelscited by 35
  18. 2025Language Modeling by Language Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 12.

15 papers.

Sort:

By area:

  1. 2026Meta Context Engineering via Agentic Skill Evolutioncited by 42
  2. 2026Towards Execution-Grounded Automated AI Researchcited by 19
  3. 2025InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Researchcited by 18
  4. 2025AgenticSciML: Collaborative Multi-Agent Systems for Emergent Discovery in Scientific Machine Learningcited by 13
  5. 2026Can We Predict Before Executing Machine Learning Agents?cited by 10
  6. 2026Can AI agents conduct open-ended AI research? Early evidence from two case studiescited by 7
  7. 2026ASI-Evolve: AI Accelerates AIcited by 4
  8. 2025Improving Language Agents through BREW: Bootstrapping expeRientially-learned Environmental knoWledgecited by 3
  9. 2026Data Darwinism Part II: DataEvolve -- AI can Autonomously Evolve Pretraining Data Curationcited by 3
  10. 2026From AI Assistant to AI Scientist: Autonomous Discovery of LLM-RL Algorithms with LLM Agentscited by 2
  11. 2026Saturating Scaling Laws for Equational Discovery: A Phenomenology of Growth Dynamics in Three Toy Substrates with Two Real-World Replicationscited by 1
  12. 2026Automated Membership Inference Attacks (AutoMIA): Discovering MIA Signal Computations using LLM Agentscited by 0
3 more
  1. 2026Agentic Discovery with Active Hypothesis Exploration for Visual Recognitioncited by 0
  2. 2026Belief-Guided Decision Making with Uncertainty Gating in the Game of Gocited by 0
  3. 2026AgentFold: Closed-Loop Agentic Search for Protein Folding Model Designcited by 0