# NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference

arXiv: https://arxiv.org/abs/2411.01142 · HTML: https://mlsanity.com/paper/2411.01142/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 2 Nov 2024.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT](https://mlsanity.com/paper/1909.05840/) (2019)
- [Atom: Low-bit Quantization for Efficient and Accurate LLM Serving](https://mlsanity.com/paper/2310.19102/) (2023)
- [PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU](https://mlsanity.com/paper/2312.12456/) (2023)
- [Fast Distributed Inference Serving for Large Language Models](https://mlsanity.com/paper/2305.05920/) (2023)
- [Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads](https://mlsanity.com/paper/2401.11181/) (2024)
- [NanoFlow: Towards Optimal Large Language Model Serving Throughput](https://mlsanity.com/paper/2408.12757/) (2024)
- [FlashDecoding++: Faster Large Language Model Inference on GPUs](https://mlsanity.com/paper/2311.01282/) (2023)
- [FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines](https://mlsanity.com/paper/2403.11421/) (2024)
- [D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving](https://mlsanity.com/paper/2403.01876/) (2024)
- [HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices](https://mlsanity.com/paper/2403.01164/) (2024)

## Built on by (most cited first, 30 of 37)

- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation](https://mlsanity.com/paper/2412.10543/) (2024)
- [TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications](https://mlsanity.com/paper/2510.18586/) (2025)
- [Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation](https://mlsanity.com/paper/2503.20552/) (2025)
- [KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation](https://mlsanity.com/paper/2411.17089/) (2024)
- [CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents](https://mlsanity.com/paper/2606.16824/) (2026)
- [HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading](https://mlsanity.com/paper/2502.12574/) (2025)
- [MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching](https://mlsanity.com/paper/2503.09716/) (2025)
- [MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing](https://mlsanity.com/paper/2506.02006/) (2025)
- [OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration](https://mlsanity.com/paper/2601.10729/) (2026)
- [CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIM](https://mlsanity.com/paper/2504.17584/) (2025)
- [TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving](https://mlsanity.com/paper/2508.17219/) (2025)
- [FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Management](https://mlsanity.com/paper/2511.00868/) (2025)
- [Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits](https://mlsanity.com/paper/2511.00321/) (2025)
- [Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization](https://mlsanity.com/paper/2607.08057/) (2026)
- [HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs](https://mlsanity.com/paper/2507.19823/) (2025)
- [SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution](https://mlsanity.com/paper/2508.18983/) (2025)
- [An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference](https://mlsanity.com/paper/2605.07719/) (2026)
- [ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference](https://mlsanity.com/paper/2605.27081/) (2026)
- [HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management](https://mlsanity.com/paper/2608.07009/) (2026)
- [RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU](https://mlsanity.com/paper/2504.15302/) (2025)
- [Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Survey](https://mlsanity.com/paper/2510.00078/) (2025)
- [SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips](https://mlsanity.com/paper/2601.20309/) (2026)
- [DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs](https://mlsanity.com/paper/2602.03495/) (2026)
- [Learn from the Past: Fast Sparse Indexing for Large Language Model Decoding](https://mlsanity.com/paper/2506.15704/) (2025)
- [Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding](https://mlsanity.com/paper/2509.18344/) (2025)
- [APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs](https://mlsanity.com/paper/2506.03296/) (2025)
- [MLP-Offload: Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wall](https://mlsanity.com/paper/2509.02480/) (2025)
- [DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architectures](https://mlsanity.com/paper/2511.15503/) (2025)
- [MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inference](https://mlsanity.com/paper/2604.21026/) (2026)

## Abstract (arXiv)

Online LLM inference powers many exciting applications such as intelligent chatbots and autonomous agents. Modern LLM inference engines widely rely on request batching to improve inference throughput, aiming to make it cost-efficient when running on expensive GPU accelerators. However, the limited GPU memory has largely limited the batch size achieved in practice, leaving significant GPU compute resources wasted. We present NEO, an online LLM inference system that offloads part of attention compute and KV cache states from the GPU to the local host CPU, effectively increasing the GPU batch size and thus inference throughput. To this end, NEO proposes asymmetric GPU-CPU pipelining and load-aware scheduling to balance GPU and CPU loads and fully utilize their compute and memory resources. We evaluate NEO on a wide range of workloads (i.e., code generation, text summarization), GPUs (i.e., T4, A10G, H100), and LLM models (i.e., 7B, 8B, 70B). NEO achieves up to 7.5×, 26%, and 14% higher throughput compared to GPU-only approach on T4, A10G, and H100 GPUs, respectively, while maintaining the same latency; with more powerful CPUs, NEO achieves up to 79.3% throughput gain on A10G GPU.