mlsanityData to 14 Oct 2026

A Method for Building Large Language Models with Predefined KV Cache Capacity

arXiv 2411.15785 ↗First posted 24 Nov 2024cs.CLcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

This paper introduces a novel approach, the Bounded-Cache Transformer (BCT), for building large language models with a predefined Key-Value (KV) cache capacity. The BCT addresses the excessive memory consumption issue in traditional KV caches by implementing a bounded-length KV cache, which is particularly suitable for the attention layers in Transformer decode-only architectures. By dynamically updating the key-value vector sequences, the BCT achieves efficient inference within limited cache capacity, significantly reducing memory usage while maintaining model performance and system throughput. Experimental results demonstrate that the BCT significantly reduces memory usage while maintaining the model's inference quality, offering a new solution for efficient inference in large language models.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 6 of its 10 references that are in the map. First posted on arXiv 24 Nov 2024.

Builds on

6 of 10 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb

Other references · 5

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2014Neural Turing Machinescited by 644

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Gated Differentiable Working Memory for Long-Context Language Modelingcited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Models
  2. 2024CORM: Cache Optimization with Recent Message for Large Language Model Inference
  3. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  4. 2024A Survey on Large Language Model Acceleration based on KV Cache Management
  5. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management
  6. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
  7. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  8. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing