mlsanityData to 7 Oct 2026

Compute and Energy Consumption Trends in Deep Learning Inference

arXiv 2109.05472 ↗First posted 12 Sep 2021cs.LG cs.AIcited by 61

Not on the map. Why?

Abstract (arXiv)

The progress of some AI paradigms such as deep learning is said to be linked to an exponential growth in the number of parameters. There are many studies corroborating these trends, but does this translate into an exponential increase in energy consumption? In order to answer this question we focus on inference costs rather than training costs, as the former account for most of the computing effort, solely because of the multiplicative factors. Also, apart from algorithmic innovations, we account for more specific and powerful hardware (leading to higher FLOPS) that is usually accompanied with important energy efficiency optimisations. We also move the focus from the first implementation of a breakthrough paper towards the consolidated version of the techniques one or two year later. Under this distinctive and comprehensive perspective, we study relevant models in the areas of computer vision and natural language processing: for a sustained increase in performance we see a much softer growth in energy consumption than previously anticipated. The only caveat is, yet again, the multiplicative factor, as future AI increases penetration and becomes more pervasive.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 58 of its 98 references that are in the map. First posted on arXiv 12 Sep 2021.

Builds on

58 of 98 references.

References · 58

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  9. 2016Densely Connected Convolutional Networkscited by 6,691
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
38 more
  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  3. 2017Squeeze-and-Excitation Networkscited by 4,763
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2014Going Deeper with Convolutionscited by 4,231
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  10. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  13. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  14. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  15. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  16. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  17. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  18. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  19. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  20. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  21. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  22. 2021Scaling Vision Transformerscited by 993
  23. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  24. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  25. 2021Going deeper with Image Transformerscited by 759
  26. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  27. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  28. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  29. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  30. 2020ResNeSt: Split-Attention Networkscited by 445
  31. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  32. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  33. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  34. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 246
  35. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  36. 2020The Computational Limits of Deep Learningcited by 188
  37. 2020Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Modelscited by 168
  38. 2017Dual Path Networkscited by 119

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 15; 2024: 18; 2025: 13; 2026: 5.

61 papers.

Sort:

By area:

  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  2. 2023Power Hungry Processing: Watts Driving the Cost of AI Deployment?cited by 186
  3. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  4. 2024Convolutional Differentiable Logic Gate Networkscited by 37
  5. 2024Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systemscited by 27
  6. 2023Brain-inspired learning in artificial neural networks: a reviewcited by 26
  7. 2023Federated Fine-Tuning of LLMs on the Very Edge: The Good, the Bad, the Uglycited by 17
  8. 2024Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloadscited by 17
  9. 2022Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Modelscited by 16
  10. 2023Watt For What: Rethinking Deep Learning's Energy-Performance Relationshipcited by 16
  11. 2023Unveiling Energy Efficiency in Deep Learning: Measurement, Prediction, and Scoring across Edge Devicescited by 16
  12. 2025Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraintscited by 16
38 more
  1. 2024MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AIcited by 15
  2. 2023MACP: Efficient Model Adaptation for Cooperative Perceptioncited by 14
  3. 2022Don't Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systemscited by 12
  4. 2022The Future of Consumer Edge-AI Computingcited by 12
  5. 2022The European AI Liability Directives -- Critique of a Half-Hearted Approach and Lessons for the Futurecited by 12
  6. 2023Efficiency is Not Enough: A Critical Perspective of Environmentally Sustainable AIcited by 12
  7. 2024TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguesecited by 12
  8. 2024Tucano: Advancing Neural Text Generation for Portuguesecited by 12
  9. 2024Green AI: A Preliminary Empirical Study on Energy Consumption in DL Models Across Different Runtime Infrastructurescited by 11
  10. 2024Measuring the Energy Consumption and Efficiency of Deep Neural Networks: An Empirical Analysis and Design Recommendationscited by 11
  11. 2025Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Modelscited by 11
  12. 2024HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Modelscited by 10
  13. 2024Verification methods for international AI agreementscited by 9
  14. 2023Chat2Map: Efficient Scene Mapping from Multi-Ego Conversationscited by 8
  15. 2025Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architecturescited by 8
  16. 2024Hyperdimensional Representation Learning for Node Classification and Link Predictioncited by 5
  17. 2022Neural Networks with Quantization Constraintscited by 4
  18. 2024Dynamic Semantic Compression for CNN Inference in Multi-access Edge Computing: A Graph Reinforcement Learning-based Autoencodercited by 4
  19. 2024SFPrompt: Communication-Efficient Split Federated Fine-Tuning for Large Pre-Trained Models over Resource-Limited Devicescited by 4
  20. 2023FLEdge: Benchmarking Federated Machine Learning Applications in Edge Computing Systemscited by 3
  21. 2026This Is Taking Too Long -- Investigating Time as a Proxy for Energy Consumption of LLMscited by 3
  22. 2022A Transistor Operations Model for Deep Learning Energy Consumption Scaling Lawcited by 2
  23. 2022AnalogVNN: A fully modular framework for modeling and optimizing photonic neural networkscited by 2
  24. 2023Sustainable Edge Intelligence Through Energy-Aware Early Exitingcited by 2
  25. 2023Feature Imitating Networks Enhance The Performance, Reliability And Speed Of Deep Learning On Biomedical Image Processing Taskscited by 2
  26. 2024Identifying architectural design decisions for achieving green ML servingcited by 2
  27. 2024A framework for measuring the training efficiency of a neural architecturecited by 2
  28. 2025OscNet: Machine Learning on CMOS Oscillator Networkscited by 2
  29. 2023ISimDL: Importance Sampling-Driven Acceleration of Fault Injection Simulations for Evaluating the Robustness of Deep Learningcited by 1
  30. 2023Neural Architecture Transfer 2: A Paradigm for Improving Efficiency in Multi-Objective Neural Architecture Searchcited by 1
  31. 2023Bitformer: An efficient Transformer with bitwise operation-based attention for Big Data Analytics at low-cost low-precision devicescited by 1
  32. 2024A Training Rate and Survival Heuristic for Inference and Robustness Evaluation (TRASHFIRE)cited by 1
  33. 2025Leveraging time and parameters for nonlinear model reduction methodscited by 1
  34. 2025acoupi: An Open-Source Python Framework for Deploying Bioacoustic AI Models on Edge Devicescited by 1
  35. 2025OscNet v1.5: Energy Efficient Hopfield Network on CMOS Oscillators for Image Classificationcited by 1
  36. 2025Energy-Aware Code Generation with LLMs: Benchmarking Small vs. Large Language Models for Sustainable AI Programmingcited by 1
  37. 2025Characterizing and Understanding Energy Footprint and Efficiency of Small Language Model on Edgescited by 1
  38. 2022Improving the Performance of DNN-based Software Services using Automated Layer Cachingcited by 0

50 most cited of 61 All 61 (JSON)