mlsanityData to 7 Oct 2026

PIMphony: Overcoming Bandwidth and Capacity Inefficiency in PIM-based Long-Context LLM Inference System

arXiv 2412.20166 ↗First posted 28 Dec 2024cs.AR cs.AIcited by 4

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The expansion of long-context Large Language Models (LLMs) creates significant memory system challenges. While Processing-in-Memory (PIM) is a promising accelerator, we identify that it suffers from critical inefficiencies when scaled to long contexts: severe channel underutilization, performance-limiting I/O bottlenecks, and massive memory waste from static KV cache management. In this work, we propose PIMphony, a PIM orchestrator that systematically resolves these issues with three co-designed techniques. First, Token-Centric PIM Partitioning (TCP) ensures high channel utilization regardless of batch size. Second, Dynamic PIM Command Scheduling (DCS) mitigates the I/O bottleneck by overlapping data movement and computation. Finally, a Dynamic PIM Access (DPA) controller enables dynamic memory management to eliminate static memory waste. Implemented via an MLIR-based compiler and evaluated on a cycle-accurate simulator, PIMphony significantly improves throughput for long-context LLM inference (up to 72B parameters and 1M context length). Our evaluations show performance boosts of up to 11.3x on PIM-only systems and 8.4x on xPU+PIM systems, enabling more efficient deployment of LLMs in real-world long-context applications.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 71 references that are in the map. First posted on arXiv 28 Dec 2024.

Builds on

21 of 71 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines18 Mar

Other references · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2023Mistral 7Bcited by 3,065
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  11. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  12. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
8 more
  1. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
  2. 2024NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencingcited by 48
  3. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 46
  4. 2020A Modern Primer on Processing in Memorycited by 30
  5. 2023Pathfinding Future PIM Architectures by Demystifying a Commercial PIM Technologycited by 13
  6. 2024IANUS: Integrated Accelerator based on NPU-PIM Unified Memory Systemcited by 11
  7. 2024PIM-MMU: A Memory Management Unit for Accelerating Data Transfers in Commercial PIM Systemscited by 4
  8. 2024Toleo: Scaling Freshness to Tera-scale Memory using CXL and PIMcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026LoopQ: Quantization for Recursive Transformerscited by 4
  2. 2025P3-LLM: An Integrated NPU-PIM Accelerator for Edge LLM Inference Using Hybrid Numerical Formatscited by 3
  3. 2026TokenStack: A Heterogeneous HBM-PIM Architecture and Runtime for Efficient LLM Inferencecited by 0
  4. 2026PATTON: Enabling Commodity PIM for Production LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  2. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  3. 2024Multi-Bin Batching for Increasing LLM Inference Throughput
  4. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving
  5. 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
  6. 2026Flow-Controlled Scheduling for LLM Inference with Provable Stability Guarantees
  7. 2023Efficient LLM Inference on CPUs
  8. 2025LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling