# PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inference arXiv: https://arxiv.org/abs/2502.07578 · HTML: https://mlsanity.com/paper/2502.07578/ ## Key facts - Cited by 22 papers in the map (data through 14 Oct 2026). - Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. - Builds on 28 of its 110 references that are in the map. - First posted on arXiv 11 Feb 2025. ## Placed in - Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/) ## Builds on (most cited first, 28 of 28) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025) - [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023) - [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021) - [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016) - [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019) - [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020) - [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019) - [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023) - [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021) - [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017) - [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024) - [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024) - [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024) - [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023) - [LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models](https://mlsanity.com/paper/2309.12307/) (2023) - [SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning](https://mlsanity.com/paper/2012.09852/) (2020) - [TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings](https://mlsanity.com/paper/2304.01433/) (2023) - [OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization](https://mlsanity.com/paper/2304.07493/) (2023) - [NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing](https://mlsanity.com/paper/2403.00579/) (2024) - [RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing](https://mlsanity.com/paper/1912.12953/) (2019) - [Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-design](https://mlsanity.com/paper/2209.09570/) (2022) - [Ramulator 2.0: A Modern, Modular, and Extensible DRAM Simulator](https://mlsanity.com/paper/2308.11030/) (2023) - [Sparse Attention Acceleration with Synergistic In-Memory Pruning and On-Chip Recomputation](https://mlsanity.com/paper/2209.00606/) (2022) - [Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices](https://mlsanity.com/paper/2303.15375/) (2023) - [Accelerating Neural Network Inference with Processing-in-DRAM: From the Edge to the Cloud](https://mlsanity.com/paper/2209.08938/) (2022) ## Built on by (most cited first, 22 of 22) - [CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIM](https://mlsanity.com/paper/2504.17584/) (2025) - [Agent-X: Full Pipeline Acceleration of On-device AI Agents](https://mlsanity.com/paper/2605.10380/) (2026) - [Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Experts](https://mlsanity.com/paper/2507.15465/) (2025) - [Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits](https://mlsanity.com/paper/2511.00321/) (2025) - [Cosmos: A CXL-Based Full In-Memory System for Approximate Nearest Neighbor Search](https://mlsanity.com/paper/2505.16096/) (2025) - [Harmonia: Enhancing Data Placement and Migration in Hybrid Storage Systems via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2503.20507/) (2025) - [Hardwired-Neurons Language Processing Units as General-Purpose Cognitive Substrates](https://mlsanity.com/paper/2508.16151/) (2025) - [Cocoon: A System Architecture for Differentially Private Training with Correlated Noises](https://mlsanity.com/paper/2510.07304/) (2025) - [Automating Hardware Design and Verification from Architectural Papers via a Neural-Symbolic Graph Framework](https://mlsanity.com/paper/2511.06067/) (2025) - [KVNAND: Efficient On-Device Large Language Model Inference Using DRAM-Free In-Flash Computing](https://mlsanity.com/paper/2512.03608/) (2025) - [LLM Inference in a Flash!](https://mlsanity.com/paper/2609.16161/) (2026) - [HALO: Memory-Centric Heterogeneous Accelerator with 2.5D Integration for Low-Batch LLM Inference](https://mlsanity.com/paper/2510.02675/) (2025) - [DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architectures](https://mlsanity.com/paper/2511.15503/) (2025) - [Hardware-Software Co-design for 3D-DRAM-based LLM Serving Accelerator](https://mlsanity.com/paper/2603.04797/) (2026) - [TokenStack: A Heterogeneous HBM-PIM Architecture and Runtime for Efficient LLM Inference](https://mlsanity.com/paper/2605.05639/) (2026) - [SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Serving](https://mlsanity.com/paper/2606.11244/) (2026) - [Beyond Prefill-Decode Disaggregation: Dissecting LLM Inference for Heterogeneous Platforms via Dynamic Operator Scheduling](https://mlsanity.com/paper/2607.25498/) (2026) - [Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention](https://mlsanity.com/paper/2608.03555/) (2026) - [MoE Expert Execution in Disaggregated LLM Serving with a High-Bandwidth ReRAM Near-Memory Architecture](https://mlsanity.com/paper/2608.13962/) (2026) - [CHIPSMORE: Compute-in-Interconnect and -Memory Chiplets for Multi-Mode Multi-Request LLM Inference Acceleration](https://mlsanity.com/paper/2608.30509/) (2026) - [DynaNDE: Dynamic Near-Data Expert Scheduling for Batched MoE Inference](https://mlsanity.com/paper/2609.00407/) (2026) - [PATTON: Enabling Commodity PIM for Production LLM Serving](https://mlsanity.com/paper/2609.11392/) (2026) ## Abstract (arXiv) Large Language Model (LLM) inference uses an autoregressive manner to generate one token at a time, which exhibits notably lower operational intensity compared to earlier Machine Learning (ML) models such as encoder-only transformers and Convolutional Neural Networks. At the same time, LLMs possess large parameter sizes and use key-value caches to store context information. Modern LLMs support context windows with up to 1 million tokens to generate versatile text, audio, and video content. A large key-value cache unique to each prompt requires a large memory capacity, limiting the inference batch size. Both low operational intensity and limited batch size necessitate a high memory bandwidth. However, contemporary hardware systems for ML model deployment, such as GPUs and TPUs, are primarily optimized for compute throughput. This mismatch challenges the efficient deployment of advanced LLMs and makes users pay for expensive compute resources that are poorly utilized for the memory-bound LLM inference tasks. We propose CENT, a CXL-ENabled GPU-Free sysTem for LLM inference, which harnesses CXL memory expansion capabilities to accommodate substantial LLM sizes, and utilizes near-bank processing units to deliver high memory bandwidth, eliminating the need for expensive GPUs. CENT exploits a scalable CXL network to support peer-to-peer and collective communication primitives across CXL devices. We implement various parallelism strategies to distribute LLMs across these devices. Compared to GPU baselines with maximum supported batch sizes and similar average power, CENT achieves 2.3× higher throughput and consumes 2.9× less energy. CENT enhances the Total Cost of Ownership (TCO), generating 5.2× more tokens per dollar than GPUs.