mlsanityData to 7 Oct 2026

PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing System

arXiv 2502.15470 ↗First posted 21 Feb 2025cs.AR cs.AI cs.DCcited by 19

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Large language models (LLMs) are widely used for natural language understanding and text generation. An LLM model relies on a time-consuming step called LLM decoding to generate output tokens. Several prior works focus on improving the performance of LLM decoding using parallelism techniques, such as batching and speculative decoding. State-of-the-art LLM decoding has both compute-bound and memory-bound kernels. Some prior works statically identify and map these different kernels to a heterogeneous architecture consisting of both processing-in-memory (PIM) units and computation-centric accelerators. We observe that characteristics of LLM decoding kernels (e.g., whether or not a kernel is memory-bound) can change dynamically due to parameter changes to meet user and/or system demands, making (1) static kernel mapping to PIM units and computation-centric accelerators suboptimal, and (2) one-size-fits-all approach of designing PIM units inefficient due to a large degree of heterogeneity even in memory-bound kernels. In this paper, we aim to accelerate LLM decoding while considering the dynamically changing characteristics of the kernels involved. We propose PAPI (PArallel Decoding with PIM), a PIM-enabled heterogeneous architecture that exploits dynamic scheduling of compute-bound or memory-bound kernels to suitable hardware units. PAPI has two key mechanisms: (1) online kernel characterization to dynamically schedule kernels to the most suitable hardware units at runtime and (2) a PIM-enabled heterogeneous computing system that harmoniously orchestrates both computation-centric processing units and hybrid PIM units with different computing capabilities. Our experimental results on three broadly-used LLMs show that PAPI achieves 1.8× and 11.1× speedups over a state-of-the-art heterogeneous LLM accelerator and a state-of-the-art PIM-only LLM accelerator, respectively.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 45 of its 102 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

45 of 102 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Models28 Oct
  2. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  3. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  4. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
29 more
  1. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  3. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  4. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  5. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  6. 2020A3: Accelerating Attention Mechanisms in Neural Networks with Approximationcited by 97
  7. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 83
  8. 2024NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencingcited by 48
  9. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 46
  10. 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformerscited by 46
  11. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 43
  12. 2022ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Designcited by 41
  13. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 38
  14. 2020A Modern Primer on Processing in Memorycited by 30
  15. 2022ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attentioncited by 29
  16. 2023Ramulator 2.0: A Modern, Modular, and Extensible DRAM Simulatorcited by 26
  17. 2023Efficient LLM Inference on CPUscited by 24
  18. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
  19. 2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 14
  20. 2024Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real Systemcited by 14
  21. 2023Pathfinding Future PIM Architectures by Demystifying a Commercial PIM Technologycited by 13
  22. 2021Benchmarking Memory-Centric Computing Systems: Analysis of Real Processing-in-Memory Hardwarecited by 12
  23. 2022Sparse Attention Acceleration with Synergistic In-Memory Pruning and On-Chip Recomputationcited by 12
  24. 2024IANUS: Integrated Accelerator based on NPU-PIM Unified Memory Systemcited by 11
  25. 2023An Introduction to the Compute Express Link (CXL) Interconnectcited by 9
  26. 2024PIM-Opt: Demystifying Distributed Optimization Algorithms on a Real-World Processing-In-Memory Systemcited by 5
  27. 2021pLUTo: Enabling Massively Parallel Computation in DRAM via Lookup Tablescited by 3
  28. 2024PIM-AI: A Novel Architecture for High-Efficiency LLM Inferencecited by 2
  29. 2024Memory-Centric Computing: Recent Advances in Processing-in-DRAMcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 9.

19 papers.

Sort:

By area:

  1. 2025LP-Spec: Leveraging LPDDR PIM for Efficient LLM Mobile Speculative Inference with Architecture-Dataflow Co-Optimizationcited by 9
  2. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructurecited by 8
  3. 2025CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIMcited by 5
  4. 2025Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIMcited by 3
  5. 2025P3-LLM: An Integrated NPU-PIM Accelerator for Edge LLM Inference Using Hybrid Numerical Formatscited by 3
  6. 2026Bole: Efficient Tree Speculation for Hybrid-Attention Language Modelscited by 3
  7. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  8. 2025Harmonia: Enhancing Data Placement and Migration in Hybrid Storage Systems via Multi-Agent Reinforcement Learningcited by 1
  9. 2025Hardwired-Neurons Language Processing Units as General-Purpose Cognitive Substratescited by 1
  10. 2025Cocoon: A System Architecture for Differentially Private Training with Correlated Noisescited by 1
  11. 2025KVNAND: Efficient On-Device Large Language Model Inference Using DRAM-Free In-Flash Computingcited by 1
  12. 2025Accelerating Sparse Transformer Inference on GPUcited by 0
7 more
  1. 2025DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architecturescited by 0
  2. 2026Joint Hardware-Workload Co-Optimization for In-Memory Computing Acceleratorscited by 0
  3. 2026Hardware-Software Co-design for 3D-DRAM-based LLM Serving Acceleratorcited by 0
  4. 2026Machine Learning-Driven Intelligent Memory System Design: From On-Chip Caches to Storagecited by 0
  5. 2026TokenStack: A Heterogeneous HBM-PIM Architecture and Runtime for Efficient LLM Inferencecited by 0
  6. 2026Beyond Prefill-Decode Disaggregation: Dissecting LLM Inference for Heterogeneous Platforms via Dynamic Operator Schedulingcited by 0
  7. 2026MoE Expert Execution in Disaggregated LLM Serving with a High-Bandwidth ReRAM Near-Memory Architecturecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024HADES: Hardware Accelerated Decoding for Efficient Speculation in Large Language Models
  2. 2024Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference
  3. 2023Efficient LLM inference solution on Intel GPU
  4. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
  5. 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
  6. 2025ASPD: Unlocking Adaptive Serial-Parallel Decoding by Exploring Intrinsic Parallelism in LLMs
  7. 2025AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism
  8. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding