mlsanityData to 7 Oct 2026

PowerInfer-2: Fast Large Language Model Inference on a Smartphone

arXiv 2406.06282 ↗First posted 10 Jun 2024cs.LGcited by 101

Not on the map. Why?

Cited by 24 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) on smartphones enable real-time AI assistance and privacy-preserving, offline operation. However, resource constraints of smartphones limit current deployments to small language models (SLMs), significantly compromising their capabilities. This paper introduces PowerInfer-2, a smartphone-based framework that enables fast inference for LLMs exceeding the memory capacity. The key insight is decomposing matrix operations into neuron clusters as the basic processing unit, which enables flexible scheduling and efficient I/O-computation pipelining. PowerInfer-2 leverages this neuron-cluster-based design in both computation and storage. For computation, neuron clusters with dense activations are processed on NPU, while sparse clusters use CPU. The storage engine provides a fine-grained pipeline mechanism that coordinates cluster-level computation and I/O operations, enhanced by a segmented neuron cache to reduce I/O activities. PowerInfer-2 achieves up to a 27.8x speed increase compared to state-of-the-art frameworks. PowerInfer-2 is the first system to serve a 47B LLM on a smartphone, achieving 11.68 tokens/s. Notably, these performance improvements preserve model quality with negligible accuracy degradation.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Builds on 24 of its 54 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

24 of 54 references.

References · 24

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2020GLU Variants Improve Transformercited by 1,724
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  10. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  11. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  12. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
12 more
  1. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  2. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  3. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  4. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
  5. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  6. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  7. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  8. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 43
  9. 2023The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Mattercited by 42
  10. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  11. 2024Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUscited by 17
  12. 2022STI: Turbocharge NLP Inference at the Edge via Elastic Pipeliningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 52; 2026: 29.

101 papers.

Sort:

By area:

  1. 2025Small Language Models are the Future of Agentic AIcited by 254
  2. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  3. 2024Small Language Models: Survey, Measurements, and Insightscited by 109
  4. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  5. 2024Fast On-device LLM Inference with NPUscited by 58
  6. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 53
  7. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  8. 2024Densing Law of LLMscited by 39
  9. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  10. 2024BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devicescited by 25
  11. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  12. 2024Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycited by 22
38 more
  1. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  2. 2025Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inferencecited by 20
  3. 2025Bitnet.cpp: Efficient Edge Inference for Ternary LLMscited by 17
  4. 2024CoreInfer: Accelerating Large Language Model Inference with Semantics-Inspired Adaptive Sparse Activationcited by 14
  5. 2024TroL: Traversal of Layers for Large Language and Vision Modelscited by 13
  6. 2025IC-Cache: Efficient Large Language Model Serving via In-context Cachingcited by 13
  7. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  8. 2025MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Expertscited by 12
  9. 2025Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoCcited by 11
  10. 2024Understanding Large Language Models in Your Pockets: Performance Study on COTS Mobile Devicescited by 10
  11. 2024Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Maskingcited by 10
  12. 2025Cognitive Edge Computing: A Comprehensive Survey on Optimizing Large Models and AI Agents for Pervasive Deploymentcited by 10
  13. 2025SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMscited by 10
  14. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPUcited by 9
  15. 2025SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsitycited by 9
  16. 2025Scaling LLM Test-Time Compute with Mobile NPU on Smartphonescited by 9
  17. 2025CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Modelscited by 8
  18. 2025DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Expertscited by 7
  19. 2025MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devicescited by 7
  20. 2025BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsitycited by 6
  21. 2025KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inferencecited by 5
  22. 2024Neuralink: Fast LLM Inference on Smartphones with Neuron Co-Activation Linkingcited by 4
  23. 2025FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inferencecited by 4
  24. 2025PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computingcited by 4
  25. 2025Token Level Routing Inference System for Edge Devicescited by 4
  26. 2025LEANN: A Low-Storage Vector Indexcited by 4
  27. 2025SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deploymentcited by 4
  28. 2025ShadowNPU: System and Algorithm Co-design for NPU-Centric On-Device LLM Inferencecited by 4
  29. 2025Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a surveycited by 4
  30. 2025Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Executioncited by 4
  31. 2025DynaKV: Enabling Accurate and Efficient Long-Sequence LLM Decoding on Smartphonescited by 4
  32. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Servingcited by 4
  33. 2024Large Language Models for Cuffless Blood Pressure Measurement From Wearable Biosignalscited by 3
  34. 2024Elastic On-Device LLM Servicecited by 3
  35. 2024Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Cachingcited by 3
  36. 2025TranSQL+: Serving Large Language Models with SQL on Low-Resource Hardwarecited by 3
  37. 2025GenieBlue: Integrating both Linguistic and Multimodal Capabilities for Large Language Models on Mobile Devicescited by 3
  38. 2025Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flashcited by 3

50 most cited of 101 All 101 (JSON)