mlsanityData to 14 Oct 2026

BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation

arXiv 2610.17966 ↗First posted 13 Oct 2026cs.LG cs.AIcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context inference makes the key-value (KV) cache a major memory bottleneck for large language models. We propose BELT, a low-rank KV cache compression framework. Its key-side objective targets attention distributions rather than raw keys or pre-softmax logits, yielding a quadratic surrogate with a closed-form low-rank solution. A learned global rank allocator uses differentiable soft selection during training and hard selection at deployment to meet a user-specified component budget. Experiments demonstrate improved model quality under aggressive compression and increased decoding throughput at comparable accuracy. Evaluated across three LLMs and multiple benchmarks, BELT achieves up to 70% low-rank KV cache compression while outperforming the strongest baseline by up to 23.3 points on LongBench, and up to 17.8x overall KV cache reduction when combined with quantization. With Triton-based GPU kernels and a quantized cache, BELT delivers up to 2.7x speedup for the attention module and 3.3x end-to-end decoding throughput.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 43 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

31 of 43 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization9 Jul
  2. 2026STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control7 Jun
  3. 2026EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction24 Mar
  4. 2025SALS: Sparse Attention in Latent Space for KV cache Compression28 Oct
  5. 2025A3 : an Analytical Low-Rank Approximation Framework for Attention19 May
  6. 2025xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction24 Mar
  7. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy4 Oct
  8. 2024Palu: Compressing KV-Cache with Low-Rank Projection30 Jul
  9. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  12. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 19

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2016Pointer Sentinel Mixture Modelscited by 3,096
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  7. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  9. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  11. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  12. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
7 more
  1. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  2. 2024Transformers are Multi-State RNNscited by 120
  3. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  4. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
  5. 2024Scaling Granite Code Models to 128K Contextcited by 11
  6. 2026Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compressioncited by 8
  7. 2025ERC-SVD: Error-Controlled SVD for Large Language Model Compressioncited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
  2. 2026PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
  3. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  4. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  5. 2026Behavior-Preserving KV Cache Compression
  6. 2026DynaCalKV: Key-Value Cache Compression via Head Grouping and Adaptive Rank Allocation
  7. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs
  8. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression