mlsanityData to 7 Oct 2026

PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference

arXiv 2602.06072 ↗First posted 3 Feb 2026cs.DC cs.LGcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Attention efficiency is critical to large language model (LLM) inference. While prior advances optimize attention execution for individual requests (e.g., FlashAttention), production LLM serving relies on batching requests with highly heterogeneous sequence lengths for high serving throughput. This mismatch induces severe computation and I/O imbalance, exacerbates stragglers, and underutilizes GPU resources. We present PackInfer, a kernel-level attention framework that enables compute- and I/O-aware execution for heterogeneous batched inference. PackInfer orchestrates batched requests into load-balanced execution groups, effectively saturating GPU utilization by packing multiple requests into unified kernel launches. By constructing attention kernels directly over packed query-key regions, PackInfer eliminates redundant computation and balances thread-block execution. It then incorporates I/O-aware grouping that co-locates shared-prefix requests and reorganizes KV caches into group-contiguous layouts, reducing memory fragmentation and redundant data movement as generation evolves. Evaluations on real-world workloads show that PackInfer reduces inference latency by 13.0-20.1%, and improves throughput by 20% compared to the state-of-the-art FlashAttention.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 33 references that are in the map. First posted on arXiv 3 Feb 2026.

Builds on

26 of 33 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  2. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration3 Oct
  3. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  4. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  5. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  12. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
7 more
  1. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  2. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  3. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 204
  4. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  5. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  6. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Informationcited by 17
  7. 2024Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Runtime-Certified Bounded-Error Quantized Attentioncited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
  2. 2025HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference
  3. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
  4. 2025MoSKA: Mixture of Shared KV Attention for Efficient Long-Sequence LLM Inference
  5. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
  6. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
  7. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
  8. 2026KernelSight-LM: A Kernel-Level LLM Inference Simulator