mlsanityData to 14 Oct 2026

POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference

arXiv 2410.18038 ↗First posted 23 Oct 2024cs.LG cs.DCcited by 47

Not on the map. Why?

Cited by 23 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Each request in LLM inference goes through two phases: compute-bound prefill and memory-bandwidth-bound decode. To improve GPU utilization, recent systems use hybrid batching that combines the prefill and decode phases of different requests into the same batch. This approach optimizes linear operations but remains inefficient for attention computation because existing attention kernels specialize execution independently for the prefill and decode phases. In this paper, we present POD-Attention - the first GPU kernel that efficiently computes attention for hybrid batches. POD-Attention aims to maximize the utilization of both compute and memory bandwidth by carefully allocating the GPU's resources such that prefill and decode operations happen concurrently on the same multiprocessor. POD-Attention speeds up attention computation by up to 59% (mean 28%), enabling higher throughput and lower latency LLM inference compared to the use of independently optimized prefill and decode attention kernels.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Builds on 28 of its 68 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

28 of 68 references.

References · 28

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  9. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  10. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  11. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  12. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
16 more
  1. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  2. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  3. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  4. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  5. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  6. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 98
  7. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 90
  8. 2023Fairness in Serving Large Language Modelscited by 78
  9. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 71
  10. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 57
  11. 2021Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloadscited by 40
  12. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 37
  13. 2023Stream-K: Work-centric Parallel Decomposition for Dense Matrix-Matrix Multiplication on the GPUcited by 23
  14. 2020Automatic Horizontal Fusion for GPU Kernelscited by 17
  15. 2024Lean Attention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformerscited by 16
  16. 2023A Framework for Fine-Grained Synchronization of Dependent GPU Kernelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 17; 2026: 28.

47 papers.

Sort:

By area:

  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  2. 2025ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Servingcited by 18
  3. 2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agentscited by 18
  4. 2026LLM-42: Enabling Determinism in LLM Inference with Verified Speculationcited by 16
  5. 2025Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Servingcited by 13
  6. 2024KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Servingcited by 11
  7. 2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspectivecited by 11
  8. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Servingcited by 6
  9. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practicecited by 6
  10. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
  11. 2025Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inferencecited by 5
  12. 2025DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurancecited by 5
35 more
  1. 2025Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokenscited by 5
  2. 2025From Tokens to Layers: Redefining Stall-Free Scheduling for MoE Serving with Layered Prefillcited by 4
  3. 2025DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexingcited by 4
  4. 2026DASH: Deterministic Attention Scheduling for High-throughput Reproducible LLM Trainingcited by 4
  5. 2025PyGraph: Robust Compiler Support for CUDA Graphs in PyTorchcited by 3
  6. 2026QUOKA: Query-Oriented KV Selection For Efficient LLM Prefillcited by 3
  7. 2026Bole: Efficient Tree Speculation for Hybrid-Attention Language Modelscited by 3
  8. 2025AGFT: An Adaptive GPU Frequency Tuner for Real-Time LLM Inference Optimizationcited by 2
  9. 2025Astrolabe: Balancing Load in LLM Serving with Randomized Prediction-Guided Schedulingcited by 2
  10. 2025P/D-Device: Disaggregated Large Language Model between Cloud and Devicescited by 2
  11. 2026Simple is Better: Multiplication May Be All You Need for LLM Request Schedulingcited by 2
  12. 2026Towards Load-Aware Prefill Deflection for Disaggregated LLM Servingcited by 2
  13. 2026SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Schedulingcited by 2
  14. 2026SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfercited by 2
  15. 2024Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernelscited by 1
  16. 2026PASCAL: A Phase-Aware Scheduling Algorithm for Serving Reasoning-based Large Language Modelscited by 1
  17. 2026HyFunc: Accelerating LLM-based Function Calls for Agentic AI through Hybrid-Model Cascade and Dynamic Templatingcited by 1
  18. 2026Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottleneckscited by 1
  19. 2026Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelinescited by 1
  20. 2026ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reusecited by 1
  21. 2026Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inferencecited by 1
  22. 2026ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernelscited by 1
  23. 2026AInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE Rolloutscited by 1
  24. 2026Taking Tokenization off the Host for Agentic LLM Servingcited by 1
  25. 2025DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architecturescited by 0
  26. 2026SwiftKV: An Edge-Oriented Attention Algorithm and Multi-Head Accelerator for Fast, Efficient LLM Decodingcited by 0
  27. 2026RAPID-Serve: Resource-efficient and Accelerated P/D Intra-GPU Disaggregationcited by 0
  28. 2026Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decodingcited by 0
  29. 2026Requests of a Feather Must Flock Together: Batch Size vs. Prefix Homogeneity in LLM Inferencecited by 0
  30. 2026OpScale: Operator-level Provisioning and Autoscaling for LLM Servingcited by 0
  31. 2026Simthesizer: An Agent-Driven Simulation Framework for LLM Serving Systemscited by 0
  32. 2026NeuronFuzz: Safety Neuron Guided Fuzzing for LLM Safety Evaluationcited by 0
  33. 2026PipeSwift: Revisiting Pipeline Parallelism for Large-Scale Completion-Oriented Agentic LLM Servingcited by 0
  34. 2026Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Managementcited by 0
  35. 2026FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Servingcited by 0