mlsanityData to 14 Oct 2026

Energy Efficiency of Locally Deployed LLMs: A Preliminary Quantitative GPU Power Benchmark on Consumer Hardware

arXiv 2608.00008 ↗First posted 12 Jun 2026cs.AI cs.ET cs.PFcited by 0

Not on the map. Why?

Abstract (arXiv)

The local deployment of large language models (LLMs) is gaining traction due to privacy concerns and the desire for on-premise inference. However, the energy costs on consumer hardware remain poorly characterized, as most benchmarks focus solely on accuracy. This paper presents a reproducible, hardware-level energy benchmark of 18 open-source LLMs (0.5B to 7B parameters) executed on a single consumer GPU (RTX 4060ti 16GB). Using the Ollama inference engine, GPU power draw was sampled at 2hz via nvidia-smi across a fixed prompt set. We evaluate mean/peak power, total energy per prompt (J/prompt), energy per output token (J/tok), and throughput (tok/s). Our findings suggest that factors beyond raw parameter count, including model architecture and quantization strategy, drive energy efficiency. Specifically, qwen2.5:0.5b and tinyllama:1.1b achieve the lowest energy cost (0.2747 J/tok and 0.3234 J/tok) and the highest throughput (>325 tok/s). In contrast, the 7B-Mistral model consumes up to 8.6x more energy per token than the most efficient model. Notably, qwen3.5:0.8b(on) exhibits anomalously high per-prompt energy due to extended internal reasoning, highlighting the need to distinguish between token generation modes in efficiency metrics.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 8 of its 20 references that are in the map. First posted on arXiv 12 Jun 2026.

Builds on

8 of 20 references.

References · 8

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  4. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  5. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  6. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  7. 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scalingcited by 19
  8. 2025Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Modelscited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures
  2. 2026Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use
  3. 2026Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms
  4. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
  5. 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference
  6. 2026The 1/W Law: An Analytical Study of Context-Length Routing Topology and GPU Generation Gains for LLM Inference Energy Efficiency
  7. 2026Silicon Showdown: Performance, Efficiency, and Ecosystem Barriers in Consumer-Grade LLM Inference
  8. 2026EnergyLens: Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimization