mlsanityData to 14 Oct 2026

AutoML-Zero: Evolving Machine Learning Algorithms From Scratch

arXiv 2003.03384 ↗First posted 6 Mar 2020cs.LG cs.NE stat.MLcited by 143

Not on the map. Why?

Abstract (arXiv)

Machine learning research has advanced in multiple aspects, including model structures and learning methods. The effort to automate such research, known as AutoML, has also made significant progress. However, this progress has largely focused on the architecture of neural networks, where it has relied on sophisticated expert-designed layers as building blocks---or similarly restrictive search spaces. Our goal is to show that AutoML can go further: it is possible today to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks. We demonstrate this by introducing a novel framework that significantly reduces human bias through a generic search space. Despite the vastness of this space, evolutionary search can still discover two-layer neural networks trained by backpropagation. These simple neural networks can then be surpassed by evolving directly on tasks of interest, e.g. CIFAR-10 variants, where modern techniques emerge in the top algorithms, such as bilinear interactions, normalized gradients, and weight averaging. Moreover, evolution adapts algorithms to different task types: e.g., dropout-like techniques appear when little data is available. We believe these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction for the field.

Key facts. Cited by 143 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 25; 2022: 25; 2023: 25; 2024: 20; 2025: 9; 2026: 39.

143 papers.

Sort:

By area:

  1. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  2. 2021Learning to Optimize: A Primer and A Benchmarkcited by 208
  3. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
  4. 2023Neural Architecture Search: Insights from 1000 Paperscited by 139
  5. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 131
  6. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 105
  7. 2021Symbolic Regression via Neural-Guided Genetic Programming Population Seedingcited by 101
  8. 2022Tutorial on amortized optimizationcited by 83
  9. 2025ML-Master: Towards AI-for-AI via Integration of Exploration and Reasoningcited by 61
  10. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  11. 2023Interpretable Machine Learning for Science with PySR and SymbolicRegression.jlcited by 54
  12. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 51
38 more
  1. 2021Evolving Reinforcement Learning Algorithmscited by 49
  2. 2021Bootstrapped Meta-Learningcited by 45
  3. 2024Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Modelscited by 40
  4. 2025Scaling Image and Video Generation via Test-Time Evolutionary Searchcited by 40
  5. 2025MAT-Agent: Adaptive Multi-Agent Training Optimizationcited by 35
  6. 2023Cross-Modal Fine-Tuning: Align then Refinecited by 34
  7. 2021INTERN: A New Learning Paradigm Towards General Visioncited by 33
  8. 2021AlphaEvolve: A Learning Framework to Discover Novel Alphas in Quantitative Investmentcited by 32
  9. 2021NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Taskscited by 32
  10. 2026Towards Execution-Grounded Automated AI Researchcited by 28
  11. 2026DeltaEvolve: Accelerating Scientific Discovery through Momentum-Driven Evolutioncited by 26
  12. 2023EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantizationcited by 22
  13. 2021AutoBERT-Zero: Evolving BERT Backbone from Scratchcited by 21
  14. 2021PSViT: Better Vision Transformer via Token Pooling and Attention Sharingcited by 21
  15. 2024Position: Leverage Foundational Models for Black-Box Optimizationcited by 21
  16. 2021Loss Function Discovery for Object Detection via Convergence-Simulation Driven Searchcited by 20
  17. 2021PyGlove: Symbolic Programming for Automated Machine Learningcited by 19
  18. 2021AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Taskscited by 19
  19. 2022Evolving symbolic density functionalscited by 19
  20. 2021Rethinking Neural Operations for Diverse Taskscited by 18
  21. 2022On Redundancy and Diversity in Cell-based Neural Architecture Searchcited by 18
  22. 2022Efficient Architecture Search for Diverse Taskscited by 18
  23. 2022Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019cited by 16
  24. 2022Symbolic Learning to Optimize: Towards Interpretability and Scalabilitycited by 16
  25. 2022EZNAS: Evolving Zero Cost Proxies For Neural Architecture Scoringcited by 15
  26. 2024Position: A Call to Action for a Human-Centered AutoML Paradigmcited by 14
  27. 2021Automated Evolutionary Approach for the Design of Composite Machine Learning Pipelinescited by 13
  28. 2021Automated Deep Learning: Neural Architecture Search Is Not the Endcited by 13
  29. 2023Auto-Prox: Training-Free Vision Transformer Architecture Search via Automatic Proxy Discoverycited by 13
  30. 2022Selecting and Composing Learning Rate Policies for Deep Neural Networkscited by 12
  31. 2025Regression Language Models for Codecited by 11
  32. 2026AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvementcited by 11
  33. 2026Recursive self-improvement of AI research agentscited by 11
  34. 2021Meta-Learning Bidirectional Update Rulescited by 10
  35. 2022Monte Carlo Tree Search based Hybrid Optimization of Variational Quantum Circuitscited by 10
  36. 2026Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedbackcited by 10
  37. 2026Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipescited by 10
  38. 2022Learning Symbolic Model-Agnostic Loss Functions via Meta-Learningcited by 9

50 most cited of 143 All 143 (JSON)