mlsanityData to 14 Oct 2026

PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU

arXiv 2312.12456 ↗First posted 16 Dec 2023cs.LG cs.OScited by 217

Not on the map. Why?

Cited by 72 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper introduces PowerInfer, a high-speed Large Language Model (LLM) inference engine on a personal computer (PC) equipped with a single consumer-grade GPU. The key principle underlying the design of PowerInfer is exploiting the high locality inherent in LLM inference, characterized by a power-law distribution in neuron activation. This distribution indicates that a small subset of neurons, termed hot neurons, are consistently activated across inputs, while the majority, cold neurons, vary based on specific inputs. PowerInfer exploits such an insight to design a GPU-CPU hybrid inference engine: hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons are computed on the CPU, thus significantly reducing GPU memory demands and CPU-GPU data transfers. PowerInfer further integrates adaptive predictors and neuron-aware sparse operators, optimizing the efficiency of neuron activation and computational sparsity. The evaluation shows that PowerInfer significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU. For the OPT-30B model, PowerInfer achieves performance comparable to that of a high-end server-grade A100 GPU, reaching 82% of its token generation rate on a single consumer-grade RTX 4090 GPU.

Key facts. Cited by 217 papers in the map (data through 14 Oct 2026). Builds on 30 of its 68 references that are in the map. First posted on arXiv 16 Dec 2023.

Builds on

30 of 68 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
18 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  2. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  3. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  5. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  6. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  7. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 204
  8. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
  9. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  10. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  11. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
  12. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 94
  13. 2023LLM-Rec: Personalized Recommendation via Prompting Large Language Modelscited by 77
  14. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  15. 2023Distilling Script Knowledge from Large Language Models for Constrained Language Planningcited by 45
  16. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Studycited by 43
  17. 2023PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformationcited by 25
  18. 2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 63; 2025: 77; 2026: 76.

217 papers.

Sort:

By area:

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  3. 2025Small Language Models are the Future of Agentic AIcited by 270
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  5. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  6. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  7. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  8. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  9. 2024Small Language Models: Survey, Measurements, and Insightscited by 112
  10. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  11. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  12. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
38 more
  1. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  2. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 71
  3. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 67
  4. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  5. 2024Fast On-device LLM Inference with NPUscited by 62
  6. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 56
  7. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  8. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 55
  9. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  10. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 47
  11. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 45
  12. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  13. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Cachingcited by 40
  14. 2024Densing Law of LLMscited by 40
  15. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameterscited by 38
  16. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 38
  17. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  18. 2025Delta Decompression for MoE-based LLMs Compressioncited by 33
  19. 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Mergingcited by 28
  20. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decodingcited by 25
  21. 2024BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devicescited by 25
  22. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 25
  23. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniquescited by 24
  24. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 23
  25. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exitingcited by 23
  26. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  27. 2024Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycited by 22
  28. 2025Bitnet.cpp: Efficient Edge Inference for Ternary LLMscited by 22
  29. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forwardcited by 21
  30. 2024Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuningcited by 21
  31. 2025R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inferencecited by 21
  32. 2025D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Servingcited by 20
  33. 2024SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Servingcited by 19
  34. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Schedulingcited by 19
  35. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  36. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 19
  37. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 17
  38. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inferencecited by 16

50 most cited of 217 All 217 (JSON)