mlsanityData to 7 Oct 2026

NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing

arXiv 2403.00579 ↗First posted 1 Mar 2024cs.ARcited by 48

Not on the map. Why?

Cited by 28 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Modern transformer-based Large Language Models (LLMs) are constructed with a series of decoder blocks. Each block comprises three key components: (1) QKV generation, (2) multi-head attention, and (3) feed-forward networks. In batched processing, QKV generation and feed-forward networks involve compute-intensive matrix-matrix multiplications (GEMM), while multi-head attention requires bandwidth-heavy matrix-vector multiplications (GEMV). Machine learning accelerators like TPUs or NPUs are proficient in handling GEMM but are less efficient for GEMV computations. Conversely, Processing-in-Memory (PIM) technology is tailored for efficient GEMV computation, while it lacks the computational power to handle GEMM effectively. Inspired by this insight, we propose NeuPIMs, a heterogeneous acceleration system that jointly exploits a conventional GEMM-focused NPU and GEMV-optimized PIM devices. The main challenge in efficiently integrating NPU and PIM lies in enabling concurrent operations on both platforms, each addressing a specific kernel type. First, existing PIMs typically operate in a "blocked" mode, allowing only either NPU or PIM to be active at any given time. Second, the inherent dependencies between GEMM and GEMV in LLMs restrict their parallel processing. To tackle these challenges, NeuPIMs is equipped with dual row buffers in each bank, facilitating the simultaneous management of memory read/write operations and PIM commands. Further, NeuPIMs employs a runtime sub-batch interleaving technique to maximize concurrent execution, leveraging batch parallelism to allow two independent sub-batches to be pipelined within a single NeuPIMs device. Our evaluation demonstrates that compared to GPU-only, NPU-only, and a na"ive NPU+PIM integrated acceleration approaches, NeuPIMs achieves 3×, 2.4× and 1.6× throughput improvement, respectively.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Mar 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 23; 2026: 18.

48 papers.

Sort:

By area:

  1. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  2. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batchingcited by 33
  3. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 30
  4. 2025PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inferencecited by 22
  5. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  6. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
  7. 2025PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing Systemcited by 19
  8. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 17
  9. 2025Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantizationcited by 13
  10. 2025Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoCcited by 11
  11. 2024Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environmentscited by 9
  12. 2025LP-Spec: Leveraging LPDDR PIM for Efficient LLM Mobile Speculative Inference with Architecture-Dataflow Co-Optimizationcited by 9
36 more
  1. 2025Stratum: System-Hardware Co-Design with Tiered Monolithic 3D-Stackable DRAM for Efficient MoE Servingcited by 9
  2. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructurecited by 8
  3. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Servingcited by 7
  4. 2025HPU: High-Bandwidth Processing Unit for Scalable, Cost-effective LLM Inference via GPU Co-processingcited by 5
  5. 2025CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIMcited by 5
  6. 2024PIMphony: Overcoming Bandwidth and Capacity Inefficiency in PIM-based Long-Context LLM Inference Systemcited by 4
  7. 2025A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMscited by 4
  8. 2025ADOR: A Design Exploration Framework for LLM Serving with Enhanced Latency and Throughputcited by 4
  9. 2025Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Expertscited by 4
  10. 2025Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIMcited by 3
  11. 2025P3-LLM: An Integrated NPU-PIM Accelerator for Edge LLM Inference Using Hybrid Numerical Formatscited by 3
  12. 2025A3D-MoE: Acceleration of Large Language Models with Mixture of Experts via 3D Heterogeneous Integrationcited by 2
  13. 2025Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristicscited by 2
  14. 2026AQPIM: Breaking the PIM Capacity Wall for LLMs with In-Memory Activation Quantizationcited by 2
  15. 2025HERA: Hybrid Edge-cloud Resource Allocation for Cost-Efficient AI Agentscited by 1
  16. 2025Exploring the Dynamic Scheduling Space of Real-Time Generative AI Applications on Emerging Heterogeneous Systemscited by 1
  17. 2025KVNAND: Efficient On-Device Large Language Model Inference Using DRAM-Free In-Flash Computingcited by 1
  18. 2026RPU -- A Reasoning Processing Unitcited by 1
  19. 2026AHASD: Asynchronous Heterogeneous Architecture for LLM Adaptive Drafting Speculative Decoding on Mobile Devicescited by 1
  20. 2025HALO: Memory-Centric Heterogeneous Accelerator with 2.5D Integration for Low-Batch LLM Inferencecited by 0
  21. 2025Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIMcited by 0
  22. 2025DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architecturescited by 0
  23. 2026Hardware-Software Co-design for 3D-DRAM-based LLM Serving Acceleratorcited by 0
  24. 2026ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Servingcited by 0
  25. 2026AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Servingcited by 0
  26. 2026TokenStack: A Heterogeneous HBM-PIM Architecture and Runtime for Efficient LLM Inferencecited by 0
  27. 2026Sieve: Dynamic Expert-Aware PIM Acceleration for Evolving Mixture-of-Experts Modelscited by 0
  28. 2026RED-PIM: Reducing Data Movement for Transformers using Processing-in-Memorycited by 0
  29. 2026Beyond Prefill-Decode Disaggregation: Dissecting LLM Inference for Heterogeneous Platforms via Dynamic Operator Schedulingcited by 0
  30. 2026Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attentioncited by 0
  31. 2026MoE Expert Execution in Disaggregated LLM Serving with a High-Bandwidth ReRAM Near-Memory Architecturecited by 0
  32. 2026NOVA: Technology-Architecture Co-Design of Near-Memory Processing for Attention-SSM-MoE Hybrid LLM Inferencecited by 0
  33. 2026CHIPSMORE: Compute-in-Interconnect and -Memory Chiplets for Multi-Mode Multi-Request LLM Inference Accelerationcited by 0
  34. 2026DynaNDE: Dynamic Near-Data Expert Scheduling for Batched MoE Inferencecited by 0
  35. 2026A Multi-Engine Dataflow for MoE Decoding on Scratchpad-Based Tensor Acceleratorscited by 0
  36. 2026AIR-LLM: Broadcasting AI Weights over Radio for Memory-Free Edge LLM Inference via RF Computingcited by 0