Slim attention: cut your context memory in half without loss -- K-cache is all you need for MHA
Not on the map. Why?
Abstract (arXiv)
Slim attention shrinks the context memory size by 2x for transformer models with MHA (multi-head attention), which can speed up inference by up to 2x for large context windows. Slim attention is an exact, mathematically identical implementation of the standard attention mechanism and therefore doesn't compromise model accuracy. In other words, slim attention losslessly compresses the context memory by a factor of 2. For encoder-decoder transformers, the context memory size can be reduced even further: For the Whisper models for example, slim attention reduces the context memory by 8x, which can speed up token generation by 5x for batch size 64 for example. And for the T5-11B model for example, the memory can be reduced by 32x because its MHA projection dimension is larger than the embedding dimension. See https://github.com/OpenMachine-ai/transformer-tricks for code and more transformer tricks, and https://www.youtube.com/watch?v=uVtk3B6YO4Y for this paper's YouTube video.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 31 of its 50 references that are in the map. First posted on arXiv 7 Mar 2025.
Builds on
31 of 50 references.
References · 31
- 2017Attention Is All You Needcited by 60,361
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
- 2023Code Llama: Open Foundation Models for Codecited by 2,139
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
- 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
19 more
- 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
- 2023A decoder-only foundation model for time-series forecastingcited by 692
- 2024OLMo: Accelerating the Science of Language Modelscited by 613
- 2024Qwen2-Audio Technical Reportcited by 560
- 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
- 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
- 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
- 2024CodeGemma: Open Code Models Based on Gemmacited by 150
- 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
- 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
- 2024MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Taskcited by 93
- 2024Stable LM 2 1.6B Technical Reportcited by 90
- 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 86
- 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Accelerationcited by 43
- 2024Meta Large Language Model Compiler: Foundation Models of Compiler Optimizationcited by 39
- 2024CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptionscited by 34
- 2024Moonshine: Speech Recognition for Live Transcription and Voice Commandscited by 19
- 2025DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrieverscited by 19
- 2025METAGENE-1: Metagenomic Foundation Model for Pandemic Monitoringcited by 5
Built on by
Citing papers per year: 2025: 1.
1 paper.