mlsanityData to 14 Oct 2026

BitDecoding: Unlocking Tensor Cores for Long-Context LLMs with Low-Bit KV Cache

arXiv 2503.18773 ↗First posted 24 Mar 2025cs.AR cs.AI cs.CLcited by 8

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The growth of long-context Large Language Models (LLMs) significantly increases memory and bandwidth pressure during autoregressive decoding due to the expanding Key-Value (KV) cache. While accuracy-preserving KV-cache quantization (e.g., 4-bit or 2-bit) reduces memory footprint, existing systems decode inefficiently by relying solely on CUDA cores, underutilizing Tensor Cores-the dominant compute resource on GPUs. We present BitDecoding, the first inference system to efficiently decode low-bit KV caches by cooperatively leveraging CUDA cores and Tensor Cores. BitDecoding smartly induces Tensor-Core-friendly layouts, introduces warp-level dequantization parallelism, and provides unified system support through query transformation, high-performance tensor- and channel-wise quantization, and a software-pipelined dequantization kernel enabling mixed-precision execution. Architecture-aware optimizations further leverage Hopper's warpgroup tensor instructions and Blackwell's NVFP4 (MXFP4) tensor formats. Evaluated on Blackwell, Hopper, and Ampere GPUs, BitDecoding achieves an average 7.5x decoding speedup over FP16 FlashDecoding-v2, up to 8.6x on Blackwell with NVFP4, and up to 4.3x over state-of-the-art approaches. On LLaMA-3.1-8B with a 128K context, BitDecoding reduces single-batch decoding latency by 3x. BitDecoding is open-sourced at https://github.com/OpenBitSys/BitDecoding.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 40 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

24 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  8. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  9. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  11. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  12. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
10 more
  1. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  2. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  3. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  4. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 76
  5. 2024MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Modelscited by 52
  6. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 46
  7. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Productioncited by 27
  8. 2024Benchmarking and Dissecting the Nvidia Hopper GPU Architecturecited by 24
  9. 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurationscited by 19
  10. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Taskscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
  2. 2025Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boostcited by 10
  3. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  4. 2025VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantizationcited by 4
  5. 2026PEEK: Context Map as an Orientation Cache for Long-Context LLM Agentscited by 4
  6. 2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
  7. 2026SPHERICAL KV: Angle-Domain Attention and Rate-Distortion Retention for Efficient Long-Context Inferencecited by 2
  8. 2026BatchGen: An Architecture for Scalable and Efficient Batch Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores
  2. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache
  3. 2024QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference
  4. 2026GPU-Accelerated INT8 Quantization for KV Cache Compression in Large Language Models
  5. 2024Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression
  6. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression
  7. 2025Striking the Right Balance between Compute and Copy: Improving LLM Inferencing Under Speculative Decoding
  8. 2025APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration