mlsanityData to 7 Oct 2026

Inference-Friendly Models With MixAttention

arXiv 2409.15012 ↗First posted 23 Sep 2024cs.CL cs.AIcited by 4

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The size of the key-value (KV) cache plays a critical role in determining both the maximum context length and the number of concurrent requests supported during inference in modern language models. The KV cache size grows proportionally with the number of attention heads and the tokens processed, leading to increased memory consumption and slower inference for long inputs. In this work, we explore the use of MixAttention, a model architecture modification closely related to a blog published by Character.AI. MixAttention combines sliding window attention, where only a small subset of recent tokens is stored in the KV cache, with KV cache sharing across layers. Our experiments demonstrate that MixAttention significantly reduces memory usage and improves inference speed without sacrificing model performance in both short and long-context tasks. We also explore various configurations of this architecture, identifying those that maintain quality across evaluation metrics while optimizing resource efficiency.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 34 references that are in the map. First posted on arXiv 23 Sep 2024.

Builds on

28 of 34 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time7 Aug
  2. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May
  3. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Models17 May
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2023Mistral 7Bcited by 3,065
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
12 more
  1. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  7. 2018Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Networkcited by 415
  8. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  9. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  10. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  11. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  12. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 155

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  2. 2025Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?cited by 10
  3. 2025TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inferencecited by 1
  4. 2026Shared Global KV with Layer-Specific Local Historycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Paged Attention Meets FlexAttention: Unlocking Long-Context Efficiency in Deployed Inference
  2. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  3. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
  4. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  5. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
  6. 2025Task-KV: Task-aware KV Cache Optimization via Semantic Differentiation of Attention Heads
  7. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
  8. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens