A framework for measuring the training efficiency of a neural architecture
Not on the map. Why?
Abstract (arXiv)
Measuring Efficiency in neural network system development is an open research problem. This paper presents an experimental framework to measure the training efficiency of a neural architecture. To demonstrate our approach, we analyze the training efficiency of Convolutional Neural Networks and Bayesian equivalents on the MNIST and CIFAR-10 tasks. Our results show that training efficiency decays as training progresses and varies across different stopping criteria for a given neural model and learning task. We also find a non-linear relationship between training stopping criteria, training Efficiency, model size, and training Efficiency. Furthermore, we illustrate the potential confounding effects of overtraining on measuring the training efficiency of a neural architecture. Regarding relative training efficiency across different architectures, our results indicate that CNNs are more efficient than BCNNs on both datasets. More generally, as a learning task becomes more complex, the relative difference in training efficiency between different architectures becomes more pronounced.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 39 of its 72 references that are in the map. First posted on arXiv 12 Sep 2024.
Builds on
39 of 72 references.
References · 39
- 2017Attention Is All You Needcited by 59,932
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
- 2020Reformer: The Efficient Transformercited by 1,658
- 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
- 2017Mixed Precision Trainingcited by 1,135
- 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
- 2020Efficient Transformers: A Surveycited by 784
- 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
27 more
- 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
- 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
- 2019Green AIcited by 620
- 2023Symbolic Discovery of Optimization Algorithmscited by 423
- 2023Data Selection for Language Models via Importance Resamplingcited by 322
- 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
- 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
- 2015Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inferencecited by 159
- 2018NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applicationscited by 146
- 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
- 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
- 2016An Analysis of Deep Neural Network Models for Practical Applicationscited by 114
- 2019The role of artificial intelligence in achieving the Sustainable Development Goalscited by 100
- 2019Accelerating Deep Learning by Focusing on the Biggest Loserscited by 94
- 2020Accelerating Training of Transformer-Based Language Models with Progressive Layer Droppingcited by 93
- 2020Practical Quasi-Newton Methods for Training Deep Neural Networkscited by 67
- 2021Compute and Energy Consumption Trends in Deep Learning Inferencecited by 61
- 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
- 2023Towards Sustainable Learning: Coresets for Data-efficient Deep Learningcited by 44
- 2022EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbonescited by 42
- 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
- 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
- 2022Neural Architecture Search as Multiobjective Optimization Benchmarks: Problem Formulation and Performance Assessmentcited by 17
- 2024EfficientTrain++: Generalized Curriculum Learning for Efficient Visual Backbone Trainingcited by 16
- 2023Reusing Pretrained Models by Multi-linear Operators for Efficient Trainingcited by 15
- 2021Profiling Neural Blocks and Design Spaces for Mobile Neural Architecture Searchcited by 4
- 2023Enhancing Deep Neural Network Training Efficiency and Performance through Linear Predictioncited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.