mlsanityData to 7 Oct 2026

FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving

arXiv 2501.01005 ↗First posted 2 Jan 2025cs.DC cs.AI cs.LGcited by 248

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Transformers, driven by attention mechanisms, form the foundation of large language models (LLMs). As these models scale up, efficient GPU attention kernels become essential for high-throughput and low-latency inference. Diverse LLM applications demand flexible and high-performance attention solutions. We present FlashInfer: a customizable and efficient attention engine for LLM serving. FlashInfer tackles KV-cache storage heterogeneity using block-sparse format and composable formats to optimize memory access and reduce redundancy. It also offers a customizable attention template, enabling adaptation to various settings through Just-In-Time (JIT) compilation. Additionally, FlashInfer's load-balanced scheduling algorithm adjusts to dynamism of user requests while maintaining compatibility with CUDAGraph which requires static configuration. FlashInfer have been integrated into leading LLM serving frameworks like SGLang, vLLM and MLC-Engine. Comprehensive kernel-level and end-to-end evaluations demonstrate FlashInfer's ability to significantly boost kernel performance across diverse inference scenarios: compared to state-of-the-art LLM serving solutions, FlashInfer achieve 29-69% inter-token-latency reduction compared to compiler backends for LLM serving benchmark, 28-30% latency reduction for long-context inference, and 13-17% speedup for LLM serving with parallel generation.

Key facts. Cited by 248 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 69 references that are in the map. First posted on arXiv 2 Jan 2025.

Builds on

39 of 69 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  2. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition23 Feb
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  11. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  12. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
24 more
  1. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  2. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  3. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  4. 2022FP8 Formats for Deep Learningcited by 187
  5. 2018Online normalizer calculation for softmaxcited by 154
  6. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  7. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  8. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  9. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 85
  10. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71
  11. 2024Theory, Analysis, and Best Practices for Sigmoid Self-Attentioncited by 62
  12. 2022SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learningcited by 47
  13. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  14. 2017Block-Sparse Recurrent Neural Networkscited by 38
  15. 2022ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputscited by 32
  16. 2021Understanding GNN Computational Graph: A Coordinated Computation, IO, and Memory Perspectivecited by 28
  17. 2020FusedMM: A Unified SDDMM-SpMM Kernel for Graph Embedding and Graph Neural Networkscited by 26
  18. 2023Stream-K: Work-centric Parallel Decomposition for Dense Matrix-Matrix Multiplication on the GPUcited by 22
  19. 2021Sequential Aggregation and Rematerialization: Distributed Full-batch Training of Graph Neural Networks on Large Graphscited by 18
  20. 2021TC-GNN: Bridging Sparse GNN Computation and Dense Tensor Cores on GPUscited by 17
  21. 2022Efficient Quantized Sparse Matrix Operations on Tensor Corescited by 16
  22. 2024Lean Attention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformerscited by 16
  23. 2024RelayAttention for Efficient Large Language Model Serving with Long System Promptscited by 13
  24. 2023Relax: Composable Abstractions for End-to-End Dynamic Machine Learningcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 90; 2026: 158.

248 papers.

Sort:

By area:

  1. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  2. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 204
  3. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 100
  4. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  5. 2025Kevin: Multi-Turn RL for Generating CUDA Kernelscited by 73
  6. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 64
  7. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  8. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 43
  9. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compressioncited by 41
  10. 2025MultiKernelBench: A Multi-Platform Benchmark for Kernel Generationcited by 35
  11. 2026LingBot-Map: Geometric Context Transformer for Streaming 3D Reconstructioncited by 32
  12. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantizationcited by 30
38 more
  1. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMscited by 24
  2. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  3. 2025AccelOpt: A Self-Improving LLM Agentic System for AI Accelerator Kernel Optimizationcited by 21
  4. 2026Dr. Kernel: Reinforcement Learning Done Right for Triton Kernel Generationscited by 21
  5. 2026FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systemscited by 20
  6. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 20
  7. 2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agentscited by 17
  8. 2026K-Search: LLM Kernel Generation via Co-Evolving Intrinsic World Modelcited by 17
  9. 2025Adaptive Self-improvement LLM Agentic System for ML Library Developmentcited by 16
  10. 2026SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decodingcited by 15
  11. 2025Scaling Inference-Efficient Language Modelscited by 14
  12. 2025Optimizing Sequential Multi-Step Tasks with Parallel LLM Agentscited by 14
  13. 2025MPK: A Compiler and Runtime for Mega-Kernelizing Tensor Programscited by 14
  14. 2026Thinking in Streaming Videocited by 14
  15. 2025FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Accelerationcited by 13
  16. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  17. 2025Speculative Decoding: Performance or Illusion?cited by 13
  18. 2026Speculative Speculative Decodingcited by 13
  19. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  20. 2025Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Servingcited by 12
  21. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 11
  22. 2026Fast Autoregressive Video Diffusion and World Models with Temporal Cache Compression and Sparse Attentioncited by 11
  23. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  24. 2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspectivecited by 10
  25. 2025Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decodingcited by 9
  26. 2025MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigmcited by 9
  27. 2026Towards Automated Kernel Generation in the Era of LLMscited by 9
  28. 2026Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attentioncited by 9
  29. 2025LLM Applications: Current Paradigms and the Next Frontiercited by 8
  30. 2025Optimizing Mixture of Block Attentioncited by 8
  31. 2026TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharingcited by 8
  32. 2025AttentionPredictor: Temporal Patterns Matter for KV Cache Compressioncited by 7
  33. 2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
  34. 2025Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltascited by 7
  35. 2025A3: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Servingcited by 7
  36. 2025Accelerate Parallelizable Reasoning via Parallel Decoding within One Sequencecited by 6
  37. 2025HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inferencecited by 6
  38. 2025SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inferencecited by 6

50 most cited of 248 All 248 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference
  2. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
  3. 2025HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference
  4. 2026FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference
  5. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memory
  6. 2026HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing
  7. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs
  8. 2026Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU