mlsanityData to 7 Oct 2026

Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency

arXiv 2504.03360 ↗First posted 4 Apr 2025cs.CY cs.AI cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

Deploying Large Language Models (LLMs) on edge devices presents significant challenges due to computational constraints, memory limitations, inference speed, and energy consumption. Model quantization has emerged as a key technique to enable efficient LLM inference by reducing model size and computational overhead. In this study, we conduct a comprehensive analysis of 28 quantized LLMs from the Ollama library, which applies by default Post-Training Quantization (PTQ) and weight-only quantization techniques, deployed on an edge device (Raspberry Pi 4 with 4GB RAM). We evaluate energy efficiency, inference performance, and output accuracy across multiple quantization levels and task types. Models are benchmarked on five standardized datasets (CommonsenseQA, BIG-Bench Hard, TruthfulQA, GSM8K, and HumanEval), and we employ a high-resolution, hardware-based energy measurement tool to capture real-world power consumption. Our findings reveal the trade-offs between energy efficiency, inference speed, and accuracy in different quantization settings, highlighting configurations that optimize LLM deployment for resource-constrained environments. By integrating hardware-level energy profiling with LLM benchmarking, this study provides actionable insights for sustainable AI, bridging a critical gap in existing research on energy-aware LLM deployment.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 42 of its 86 references that are in the map. First posted on arXiv 4 Apr 2025.

Builds on

42 of 86 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  7. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  8. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  9. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  10. 2019MLPerf Inference Benchmarkcited by 195
  11. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  12. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
30 more
  1. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 137
  2. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  3. 2021MLPerf Tiny Benchmarkcited by 118
  4. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  5. 2019AI Benchmark: All About Deep Learning on Smartphones in 2019cited by 76
  6. 2023Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Servicescited by 76
  7. 2024Evaluating Quantized Large Language Modelscited by 75
  8. 2024Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Modelscited by 64
  9. 2024An empirical study of LLaMA3 quantization: from LLMs to MLLMscited by 63
  10. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
  11. 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 55
  12. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 47
  13. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Studycited by 43
  14. 2024Accurate LoRA-Finetuning Quantization of LLMs via Information Retentioncited by 40
  15. 2023Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edgecited by 33
  16. 2024EDGE-LLM: Enabling Efficient Large Language Model Adaptation on Edge Devices via Layerwise Unified Compression and Adaptive Layer Tuning and Votingcited by 26
  17. 2024The Price of Prompting: Profiling Energy Use in Large Language Models Inferencecited by 22
  18. 2023Greening Large Language Models of Codecited by 19
  19. 2024Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devicescited by 17
  20. 2024MobileQuant: Mobile-friendly Quantization for On-device Language Modelscited by 16
  21. 2024Designing Efficient LLM Accelerators for Edge Devicescited by 13
  22. 2024Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolboxcited by 11
  23. 2024Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniquescited by 10
  24. 2023Quantized Transformer Language Model Implementations on Edge Devicescited by 9
  25. 2024TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devicescited by 9
  26. 2023A Performance Evaluation of a Quantized Large Language Model on Various Smartphonescited by 7
  27. 2024PalmBench: A Comprehensive Benchmark of Compressed Large Language Models on Mobile Platformscited by 7
  28. 2024Decentralized LLM Inference over Edge Networks with Energy Harvestingcited by 5
  29. 2024A Comparative Study on Large Language Models for Log Parsingcited by 5
  30. 2025FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devicescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 22.

32 papers.

Sort:

By area:

  1. 2025SLED: A Speculative LLM Decoding Framework for Efficient Edge Servingcited by 13
  2. 2025Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMscited by 7
  3. 2026Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Usecited by 4
  4. 2025Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendationcited by 3
  5. 2026The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMscited by 3
  6. 2025A Survey on Foundation Models for Personalized Federated Intelligencecited by 2
  7. 2025Edge Artificial Intelligence: A Systematic Review of Evolution, Taxonomic Frameworks, and Future Horizonscited by 2
  8. 2025Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implicationscited by 2
  9. 2026The Quantization Trap: Breaking Linear Scaling Laws in Multi-Hop Reasoningcited by 2
  10. 2026Weight Pruning Amplifies Bias: A Multi-Method Study of Compressed LLMs for Edge AIcited by 2
  11. 2026Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPUcited by 2
  12. 2025Vocabulary Customization for Efficient Domain-Specific LLM Deploymentcited by 1
20 more
  1. 2026SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Modelscited by 1
  2. 2026Retrieval-Feedback-Driven Distillation and Preference Alignment for Efficient LLM-based Query Expansioncited by 1
  3. 2026Multilinguality at the Edge: Developing Language Models for the Global Southcited by 1
  4. 2026Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurationscited by 1
  5. 2026Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levelscited by 1
  6. 2025A Systematic Evaluation of On-Device LLMs: Quantization, Performance, and Resourcescited by 0
  7. 2025Resource-Aware LLM Reasoning for Mobile Edge General Intelligencecited by 0
  8. 2025Metrics and evaluations for computational and sustainable AI efficiencycited by 0
  9. 2026Energy Efficient Software Hardware CoDesign for Machine Learning: From TinyML to Large Language Modelscited by 0
  10. 2026Networking-Aware Energy Efficiency in Agentic AI Inference: A Surveycited by 0
  11. 2026Benchmarking Local Language Models for Social Robots using Edge Devicescited by 0
  12. 2026Online LLM Selection via Constrained Bandits with Time-Varying Demandcited by 0
  13. 2026Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Studycited by 0
  14. 2026Learning-Based Collaborative MEC for LLM Inference with Soft-Deadline Awareness via Transformer-Enhanced PPOcited by 0
  15. 2026Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIscited by 0
  16. 2026GreenBench: Benchmarking Energy Efficiency and Carbon Footprint of Open-Source LLM Inference on Apple Siliconcited by 0
  17. 2026Triple-Bottom-Line Sustainability of Language Models for Edge AI: A Comparison Between SLMs and Quantized LLMscited by 0
  18. 2026Broken on Arrival: Silently Defective LLM Artifacts in Public Model Registries and How to Catch Themcited by 0
  19. 2026The Battery Price of edge AI: A study of the Environmental Impact of LLM Inference on Mobile Devicescited by 0
  20. 2026Not All 4-bit Quantizers Are Equal: Deployment-Time Mitigation of PII Leakage in Fine-Tuned Small Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025EdgeProfiler: A Fast Profiling Framework for Lightweight LLMs on Edge Using Analytical Model
  2. 2023Efficient LLM Inference on CPUs
  3. 2025MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices
  4. 2024EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
  5. 2024Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems
  6. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
  7. 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
  8. 2024LLM Inference Unveiled: Survey and Roofline Model Insights