mlsanityData to 7 Oct 2026

The BUTTER Zone: An Empirical Study of Training Dynamics in Fully Connected Neural Networks

arXiv 2207.12547 ↗First posted 25 Jul 2022cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We present an empirical dataset surveying the deep learning phenomenon on fully-connected feed-forward multilayer perceptron neural networks. The dataset, which is now freely available online, records the per-epoch training and generalization performance of 483 thousand distinct hyperparameter choices of architectures, tasks, depths, network sizes (number of parameters), learning rates, batch sizes, and regularization penalties. Repeating each experiment an average of 24 times resulted in 11 million total training runs and 40 billion epochs recorded. Accumulating this 1.7 TB dataset utilized 11 thousand CPU core-years, 72.3 GPU-years, and 163 node-years. In surveying the dataset, we observe durable patterns persisting across tasks and topologies. We aim to spark scientific study of machine learning techniques as a catalyst for the theoretical discoveries needed to progress the field beyond energy-intensive and heuristic practices.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 44 of its 67 references that are in the map. First posted on arXiv 25 Jul 2022.

Builds on

44 of 67 references.

References · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  5. 2018Datasheets for Datasetscited by 1,371
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  7. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  8. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  9. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  10. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  11. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  12. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
32 more
  1. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  2. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  3. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 340
  4. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  5. 2018An Empirical Model of Large-Batch Trainingcited by 268
  6. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  7. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  8. 2015The Power of Depth for Feedforward Neural Networkscited by 181
  9. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
  10. 2020Deep Network Approximation for Smooth Functionscited by 163
  11. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  12. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  13. 2020NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Sizecited by 98
  14. 2019Deep Neural Network Approximation Theorycited by 92
  15. 2017Implicit Regularization in Deep Learningcited by 90
  16. 2019The phase diagram of approximation rates for deep neural networkscited by 86
  17. 2013Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networkscited by 84
  18. 2020In Search of Robust Measures of Generalizationcited by 79
  19. 2018A Modern Take on the Bias-Variance Tradeoff in Neural Networkscited by 64
  20. 2020NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Searchcited by 64
  21. 2016Deep vs. shallow networks : An approximation theory perspectivecited by 63
  22. 2020PMLB v1.0: An open source dataset collection for benchmarking machine learning methodscited by 58
  23. 2019Tabular Benchmarks for Joint Architecture and Hyperparameter Optimizationcited by 57
  24. 2021TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Searchcited by 57
  25. 2022Data Scaling Laws in NMT: The Effect of Noise and Architecturecited by 55
  26. 2020Multiple Descent: Design Your Own Generalization Curvecited by 44
  27. 2022NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easycited by 36
  28. 2021Prioritized Architecture Sampling with Monto-Carlo Tree Searchcited by 32
  29. 2021Scaling Laws for Deep Learningcited by 25
  30. 2021Multi-scale Feature Learning Dynamics: Insights for Double Descentcited by 24
  31. 2021NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameterscited by 9
  32. 2021Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metricscited by 7

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025LCDB 1.1: A Database Illustrating Learning Curves Are More Ill-Behaved Than Previously Thoughtcited by 1