Attention Is All You Need But You Don't Need All Of It For Inference of Large Language Models
Not on the map. Why?
Abstract (arXiv)
The inference demand for LLMs has skyrocketed in recent months, and serving models with low latencies remains challenging due to the quadratic input length complexity of the attention layers. In this work, we investigate the effect of dropping MLP and attention layers at inference time on the performance of Llama-v2 models. We find that dropping dreeper attention layers only marginally decreases performance but leads to the best speedups alongside dropping entire layers. For example, removing 33% of attention layers in a 13B Llama2 model results in a 1.8% drop in average performance over the OpenLLM benchmark. We also observe that skipping layers except the latter layers reduces performances for more layers skipped, except for skipping the attention layers.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 22 Jul 2024.
Builds on
30 of 41 references.
References · 30
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
- 2023Mistral 7Bcited by 3,065
- 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
- 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
- 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
- 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
- 2021Carbon Emissions and Large Neural Network Trainingcited by 561
18 more
- 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
- 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
- 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
- 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
- 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
- 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
- 2022Confident Adaptive Language Modelingcited by 297
- 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
- 2023Challenges and Applications of Large Language Modelscited by 251
- 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
- 2019Depth-Adaptive Transformercited by 227
- 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
- 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
- 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
- 2020Accelerating Training of Transformer-Based Language Models with Progressive Layer Droppingcited by 93
- 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 89
- 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 58
- 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.