mlsanityData to 7 Oct 2026

The Efficiency Misnomer

arXiv 2110.12894 ↗First posted 25 Oct 2021cs.LG cs.AI cs.CLcited by 98

Not on the map. Why?

Abstract (arXiv)

Model efficiency is a critical aspect of developing and deploying machine learning models. Inference time and latency directly affect the user experience, and some applications have hard requirements. In addition to inference costs, model training also have direct financial and environmental impacts. Although there are numerous well-established metrics (cost indicators) for measuring model efficiency, researchers and practitioners often assume that these metrics are correlated with each other and report only few of them. In this paper, we thoroughly discuss common cost indicators, their advantages and disadvantages, and how they can contradict each other. We demonstrate how incomplete reporting of cost indicators can lead to partial conclusions and a blurred or incomplete picture of the practical considerations of different models. We further present suggestions to improve reporting of efficiency metrics.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Builds on 76 of its 88 references that are in the map. First posted on arXiv 25 Oct 2021.

Builds on

76 of 88 references.

References · 76

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2016Densely Connected Convolutional Networkscited by 6,691
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2014Going Deeper with Convolutionscited by 4,231
38 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  6. 2018SlowFast Networks for Video Recognitioncited by 1,862
  7. 2021ViViT: A Video Vision Transformercited by 1,610
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  9. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  10. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  11. 2021Scaling Vision Transformerscited by 993
  12. 2020Designing Network Design Spacescited by 954
  13. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  14. 2021Multiscale Vision Transformerscited by 896
  15. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  16. 2020Efficient Transformers: A Surveycited by 784
  17. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  18. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  19. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  20. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  21. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  22. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  23. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  24. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  25. 2019Model-Based Reinforcement Learning for Ataricited by 495
  26. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  27. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  28. 2019The State of Sparsity in Deep Neural Networkscited by 471
  29. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  30. 2019AutoML: A Survey of the State-of-the-Artcited by 388
  31. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  32. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  33. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  34. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 333
  35. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  36. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  37. 2021Scaling Laws for Transfercited by 265
  38. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 24; 2023: 29; 2024: 19; 2025: 11; 2026: 10.

98 papers.

Sort:

By area:

  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  2. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  3. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  4. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  5. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  6. 2022Confident Adaptive Language Modelingcited by 297
  7. 2022UL2: Unifying Language Learning Paradigmscited by 292
  8. 2023Challenges and Applications of Large Language Modelscited by 251
  9. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  10. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  11. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  12. 2022FlexiViT: One Model for All Patch Sizescited by 151
38 more
  1. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  2. 2022MobileOne: An Improved One millisecond Mobile Backbonecited by 109
  3. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  4. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 95
  5. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 87
  6. 2022ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similaritycited by 82
  7. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  8. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  9. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  10. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  11. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  12. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  13. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformerscited by 59
  14. 2025Distillation Scaling Lawscited by 59
  15. 2021PolyViT: Co-training Vision Transformers on Images, Videos and Audiocited by 58
  16. 2022Transcending Scaling Laws with 0.1% Extra Computecited by 55
  17. 2023Intriguing Properties of Quantization at Scalecited by 48
  18. 2023How Does Generative Retrieval Scale to Millions of Passages?cited by 47
  19. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  20. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  21. 2024A Simple Model of Inference Scaling Lawscited by 36
  22. 2023Vision Transformers with Mixed-Resolution Tokenizationcited by 35
  23. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 34
  24. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  25. 2021Augmenting Convolutional networks with attention-based aggregationcited by 29
  26. 2024Time-, Memory- and Parameter-Efficient Visual Adaptationcited by 28
  27. 2021Learned Queries for Efficient Local Attentioncited by 27
  28. 2023Skip-Attention: Improving Vision Transformers by Paying Less Attentioncited by 23
  29. 2023Adaptive Computation with Elastic Input Sequencecited by 22
  30. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  31. 2023How can objects help action recognition?cited by 21
  32. 2024On the Limitations of Compute Thresholds as a Governance Strategycited by 17
  33. 2022Understanding Gradient Regularization in Deep Learning: Efficient Finite-Difference Computation and Implicit Biascited by 16
  34. 2023Watt For What: Rethinking Deep Learning's Energy-Performance Relationshipcited by 16
  35. 2023Towards quantitative precision for ECG analysis: Leveraging state space models, self-supervision and patient metadatacited by 15
  36. 2022RepGhost: A Hardware-Efficient Ghost Module via Re-parameterizationcited by 13
  37. 2025Edge-Cloud Collaborative Computing on Distributed Intelligence and Model Optimization: A Surveycited by 13
  38. 2022A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Modelscited by 12

50 most cited of 98 All 98 (JSON)