# Green AI: A Preliminary Empirical Study on Energy Consumption in DL Models Across Different Runtime Infrastructures

arXiv: https://arxiv.org/abs/2402.13640 · HTML: https://mlsanity.com/paper/2402.13640/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 33 references that are in the map.
- First posted on arXiv 21 Feb 2024.

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning](https://mlsanity.com/paper/2002.05651/) (2020)
- [Compute and Energy Consumption Trends in Deep Learning Inference](https://mlsanity.com/paper/2109.05472/) (2021)
- [A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research](https://mlsanity.com/paper/2009.06520/) (2020)
- [Data-Centric Green AI: An Exploratory Empirical Study](https://mlsanity.com/paper/2204.02766/) (2022)
- [Uncovering Energy-Efficient Practices in Deep Learning Training: Preliminary Steps Towards Green AI](https://mlsanity.com/paper/2303.13972/) (2023)
- [Energy Efficiency of Training Neural Network Architectures: An Empirical Study](https://mlsanity.com/paper/2302.00967/) (2023)
- [Batching for Green AI -- An Exploratory Study on Inference](https://mlsanity.com/paper/2307.11434/) (2023)
- [Energy Efficiency Considerations for Popular AI Benchmarks](https://mlsanity.com/paper/2304.08359/) (2023)

## Built on by (most cited first, 11 of 11)

- [A Survey on Inference Optimization Techniques for Mixture of Experts Models](https://mlsanity.com/paper/2412.14219/) (2024)
- [Greening AI-enabled Systems with Software Engineering: A Research Agenda for Environmentally Sustainable AI Practices](https://mlsanity.com/paper/2506.01774/) (2025)
- [The Impact of Hyperparameters on Large Language Model Inference Performance: An Evaluation of vLLM and HuggingFace Pipelines](https://mlsanity.com/paper/2408.01050/) (2024)
- [DEUCE: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learning](https://mlsanity.com/paper/2502.00305/) (2025)
- [How Do Model Export Formats Impact the Development of ML-Enabled Systems? A Case Study on Model Integration](https://mlsanity.com/paper/2502.00429/) (2025)
- [Dissecting Transformers: A CLEAR Perspective towards Green AI](https://mlsanity.com/paper/2510.02810/) (2025)
- [Event-driven Robust Fitting on Neuromorphic Hardware](https://mlsanity.com/paper/2508.09466/) (2025)
- [The Energy Impact of Domain Model Design in Classical Planning](https://mlsanity.com/paper/2601.21967/) (2026)
- [Pruning Extensions and Efficiency Trade-Offs for Sustainable Time Series Classification](https://mlsanity.com/paper/2604.07953/) (2026)
- [Watts and Debts of Agentic Frameworks: An Empirical Study (Registered Report)](https://mlsanity.com/paper/2606.10702/) (2026)
- [Gradient Under Microscope: Benchmarking Resource Utilization of Memory-Efficient Gradient Computation Methods](https://mlsanity.com/paper/2608.08961/) (2026)

## Abstract (arXiv)

Deep Learning (DL) frameworks such as PyTorch and TensorFlow include runtime infrastructures responsible for executing trained models on target hardware, managing memory, data transfers, and multi-accelerator execution, if applicable. Additionally, it is a common practice to deploy pre-trained models on environments distinct from their native development settings. This led to the introduction of interchange formats such as ONNX, which includes its runtime infrastructure, and ONNX Runtime, which work as standard formats that can be used across diverse DL frameworks and languages. Even though these runtime infrastructures have a great impact on inference performance, no previous paper has investigated their energy efficiency. In this study, we monitor the energy consumption and inference time in the runtime infrastructures of three well-known DL frameworks as well as ONNX, using three various DL models. To have nuance in our investigation, we also examine the impact of using different execution providers. We find out that the performance and energy efficiency of DL are difficult to predict. One framework, MXNet, outperforms both PyTorch and TensorFlow for the computer vision models using batch size 1, due to efficient GPU usage and thus low CPU usage. However, batch size 64 makes PyTorch and MXNet practically indistinguishable, while TensorFlow is outperformed consistently. For BERT, PyTorch exhibits the best performance. Converting the models to ONNX yields significant performance improvements in the majority of cases. Finally, in our preliminary investigation of execution providers, we observe that TensorRT always outperforms CUDA.