mlsanityData to 7 Oct 2026

NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference

arXiv 2411.01142 ↗First posted 2 Nov 2024cs.DC cs.AI cs.LGcited by 37

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Online LLM inference powers many exciting applications such as intelligent chatbots and autonomous agents. Modern LLM inference engines widely rely on request batching to improve inference throughput, aiming to make it cost-efficient when running on expensive GPU accelerators. However, the limited GPU memory has largely limited the batch size achieved in practice, leaving significant GPU compute resources wasted. We present NEO, an online LLM inference system that offloads part of attention compute and KV cache states from the GPU to the local host CPU, effectively increasing the GPU batch size and thus inference throughput. To this end, NEO proposes asymmetric GPU-CPU pipelining and load-aware scheduling to balance GPU and CPU loads and fully utilize their compute and memory resources. We evaluate NEO on a wide range of workloads (i.e., code generation, text summarization), GPUs (i.e., T4, A10G, H100), and LLM models (i.e., 7B, 8B, 70B). NEO achieves up to 7.5×, 26%, and 14% higher throughput compared to GPU-only approach on T4, A10G, and H100 GPUs, respectively, while maintaining the same latency; with more powerful CPUs, NEO achieves up to 79.3% throughput gain on A10G GPU.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 44 references that are in the map. First posted on arXiv 2 Nov 2024.

Builds on

28 of 44 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughput22 Aug
  2. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines18 Mar
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  5. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs2 Nov
  6. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
10 more
  1. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  2. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  3. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  4. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  5. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  6. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  7. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  8. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  9. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  10. 2024HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devicescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 18; 2026: 16.

37 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  2. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 16
  3. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  4. 2025Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregationcited by 14
  5. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputationcited by 12
  6. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloadingcited by 9
  7. 2026CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agentscited by 8
  8. 2025MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batchingcited by 6
  9. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfigurationcited by 6
  10. 2025CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIMcited by 5
  11. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
  12. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizingcited by 4
25 more
  1. 2025Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limitscited by 4
  2. 2025FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Managementcited by 4
  3. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMscited by 3
  4. 2025SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitutioncited by 3
  5. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inferencecited by 3
  6. 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inferencecited by 3
  7. 2025RAGDoll: Efficient Offloading-based Online RAG System on a Single GPUcited by 2
  8. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  9. 2026SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchipscited by 2
  10. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCscited by 2
  11. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  12. 2026HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Managementcited by 2
  13. 2025Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decodingcited by 1
  14. 2025APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUscited by 0
  15. 2025Learn from the Past: Fast Sparse Indexing for Large Language Model Decodingcited by 0
  16. 2025MLP-Offload: Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wallcited by 0
  17. 2025DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architecturescited by 0
  18. 2026MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inferencecited by 0
  19. 2026OOM-Free Alpamayo via CPU-GPU Memory Swapping for Vision-Language-Action Modelscited by 0
  20. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemptioncited by 0
  21. 2026HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrievalcited by 0
  22. 2026BatchGen: An Architecture for Scalable and Efficient Batch Inferencecited by 0
  23. 2026Anatomy of a Quantized Agent: VRAM Stability and Forecasting in Code-Synthesis Agentic Workloadscited by 0
  24. 2026BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inferencecited by 0
  25. 2026Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Managementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains
  2. 2024Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies
  3. 2025SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices
  4. 2026PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
  5. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
  6. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  7. 2026Lodestar: An Online-Learning LLM Inference Router
  8. 2025Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints