# Towards Accurate and Reliable Energy Measurement of NLP Models

arXiv: https://arxiv.org/abs/2010.05248 · HTML: https://mlsanity.com/paper/2010.05248/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2020.

## Built on by (most cited first, 16 of 16)

- [Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training](https://mlsanity.com/paper/2208.06102/) (2022)
- [Efficient Methods for Natural Language Processing: A Survey](https://mlsanity.com/paper/2209.00099/) (2022)
- [A Systematic Review of Green AI](https://mlsanity.com/paper/2301.11047/) (2023)
- [IrEne: Interpretable Energy Prediction for Transformers](https://mlsanity.com/paper/2106.01199/) (2021)
- [Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models](https://mlsanity.com/paper/2502.05610/) (2025)
- [Dissecting FLOPs along input dimensions for GreenAI cost estimations](https://mlsanity.com/paper/2107.11949/) (2021)
- [Bridging Fairness and Environmental Sustainability in Natural Language Processing](https://mlsanity.com/paper/2211.04256/) (2022)
- [Counter-GAP: Counterfactual Bias Evaluation through Gendered Ambiguous Pronouns](https://mlsanity.com/paper/2302.05674/) (2023)
- [PolyThrottle: Energy-efficient Neural Network Inference on Edge Devices](https://mlsanity.com/paper/2310.19991/) (2023)
- [Panacea: Novel DNN Accelerator using Accuracy-Preserving Asymmetric Quantization and Energy-Saving Bit-Slice Sparsity](https://mlsanity.com/paper/2412.10059/) (2024)
- [EPAM: A Predictive Energy Model for Mobile AI](https://mlsanity.com/paper/2303.01509/) (2023)
- [Estimating Deep Learning energy consumption based on model architecture and training environment](https://mlsanity.com/paper/2307.05520/) (2023)
- [An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2](https://mlsanity.com/paper/2411.12758/) (2024)
- [NL2CMD: An Updated Workflow for Natural Language to Bash Commands Translation](https://mlsanity.com/paper/2302.07845/) (2023)
- [Cheaply Evaluating Inference Efficiency Metrics for Autoregressive Transformer APIs](https://mlsanity.com/paper/2305.02440/) (2023)
- [Phoeni6: a Systematic Approach for Evaluating the Energy Consumption of Neural Networks](https://mlsanity.com/paper/2502.17734/) (2025)

## Abstract (arXiv)

Accurate and reliable measurement of energy consumption is critical for making well-informed design choices when choosing and training large scale NLP models. In this work, we show that existing software-based energy measurements are not accurate because they do not take into account hardware differences and how resource utilization affects energy consumption. We conduct energy measurement experiments with four different models for a question answering task. We quantify the error of existing software-based energy measurements by using a hardware power meter that provides highly accurate energy measurements. Our key takeaway is the need for a more accurate energy estimation model that takes into account hardware variabilities and the non-linear relationship between resource utilization and energy consumption. We release the code and data at https://github.com/csarron/sustainlp2020-energy.