Data to 14 Oct 2026

From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference

arXiv 2310.03003 ↗First posted 4 Oct 2023cs.CL cs.DCcited by 170

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have exploded in popularity due to their new generative capabilities that go far beyond prior state-of-the-art. These technologies are increasingly being leveraged in various domains such as law, finance, and medicine. However, these models carry significant computational challenges, especially the compute and energy costs required for inference. Inference energy costs already receive less attention than the energy costs of training LLMs -- despite how often these large models are called on to conduct inference in reality (e.g., ChatGPT). As these state-of-the-art LLMs see increasing usage and deployment in various domains, a better understanding of their resource utilization is crucial for cost-savings, scaling performance, efficient hardware usage, and optimal inference strategies. In this paper, we describe experiments conducted to study the computational and energy utilization of inference with LLMs. We benchmark and conduct a preliminary analysis of the inference performance and inference energy costs of different sizes of LLaMA -- a recent state-of-the-art LLM -- developed by Meta AI on two generations of popular GPUs (NVIDIA V100 & A100) and two datasets (Alpaca and GSM8K) to reflect the diverse set of tasks/benchmarks for LLMs in research and practice. We present the results of multi-node, multi-GPU inference using model sharding across up to 32 GPUs. To our knowledge, our work is the one of the first to study LLM inference performance from the perspective of computational and energy resources at this scale.

Key facts. Cited by 170 papers in the map (data through 14 Oct 2026). Builds on 14 of its 28 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

14 of 28 references.

References · 14

2017Attention Is All You Needcited by 60,361
2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
2021Training Verifiers to Solve Math Word Problemscited by 10,109
2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
2022Compute Trends Across Three Eras of Machine Learningcited by 163
2018Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysiscited by 110
2023Art and the science of generative AI: A deeper divecited by 95
2023ChatGPT is not all you need. A State of the Art Review of large Generative AI modelscited by 86
2022Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Modelscited by 21

2 more

2022MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant Systems for Machine Learningcited by 15
2023A Green(er) World for A.Icited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 55; 2025: 66; 2026: 45.

170 papers.

Sort:

By area:

2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 117
2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
2025OverThink: Slowdown Attacks on Reasoning LLMscited by 96
2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
2025How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inferencecited by 76
2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 61
2025Energy Considerations of Large Language Model Inference and Efficiency Optimizationscited by 55
2024Everybody Prune Now: Structured Pruning of LLMs with only Forward Passescited by 43
2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 39
2025Measuring the environmental impact of delivering AI at Google Scalecited by 32

38 more

2024A Review of Multi-Modal Large Language and Vision Modelscited by 30
2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 29
2024Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systemscited by 28
2024Bias patterns in the application of LLMs for clinical decision support: A comprehensive studycited by 27
2024The Future of AI-Driven Software Engineeringcited by 23
2024The Price of Prompting: Profiling Energy Use in Large Language Models Inferencecited by 23
2025TokenPowerBench: Benchmarking the Power Consumption of LLM Inferencecited by 22
2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 19
2024A Survey of Sustainability in Large Language Models: Applications, Economics, and Challengescited by 18
2024Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloadscited by 17
2024MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AIcited by 16
2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 16
2024Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inferencecited by 15
2024LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Servicescited by 14
2025Intelligence per Watt: Measuring Intelligence Efficiency of Local AIcited by 14
2026Beyond Max Tokens: Stealthy Resource Amplification via Tool Calling Chains in LLM Agentscited by 14
2025Semantic Caching for Low-Cost LLM Serving: From Offline Learning to Online Adaptationcited by 13
2025Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scalingcited by 13
2024AI "News" Content Farms Are Easy to Make and Hard to Detect: A Case Study in Italiancited by 12
2024Persistent Topological Features in Large Language Modelscited by 12
2025From Prompts to Power: Measuring the Energy Footprint of LLM Inferencecited by 12
2026Mano: Restriking Manifold Optimization for LLM Trainingcited by 12
2024Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inferencecited by 11
2025Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Modelscited by 11
2025RARE: Retrieval-Augmented Reasoning Modelingcited by 11
2025Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exitscited by 11
2025SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decouplingcited by 11
2024Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitationscited by 10
2024Incentivizing Quality Text Generation via Statistical Contractscited by 10
2024The Unseen AI Disruptions for Power Grids: LLM-Induced Transientscited by 10
2025Does quantization affect models' performance on long-context tasks?cited by 10
2025LOST: Low-rank and Sparse Pre-training for Large Language Modelscited by 10
2024Derailer-Rerailer: Adaptive Verification for Efficient and Reliable Language Model Reasoningcited by 9
2024TinyLLM: A Framework for Training and Deploying Language Models at the Edge Computerscited by 9
2025MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guaranteescited by 9
2025Large Language Models, Knowledge Graphs and Search Engines: A Crossroads for Answering Users' Questionscited by 8
2025Language Models Can Predict Their Own Behaviorcited by 8
2025Learning to Insert [PAUSE] Tokens for Better Reasoningcited by 8

50 most cited of 170 All 170 (JSON)