# What Matters in Transformers? Not All Attention is Needed

arXiv: https://arxiv.org/abs/2406.15786 · HTML: https://mlsanity.com/paper/2406.15786/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jun 2024.

## Built on by (most cited first, 30 of 84)

- [SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression](https://mlsanity.com/paper/2503.12340/) (2025)
- [Adaptive Layer-skipping in Pre-trained LLMs](https://mlsanity.com/paper/2503.23798/) (2025)
- [TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Training](https://mlsanity.com/paper/2501.04765/) (2025)
- [Puzzle: Distillation-Based NAS for Inference-Optimized LLMs](https://mlsanity.com/paper/2411.19146/) (2024)
- [Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers](https://mlsanity.com/paper/2505.13737/) (2025)
- [ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layers](https://mlsanity.com/paper/2504.00502/) (2025)
- [Persistent Topological Features in Large Language Models](https://mlsanity.com/paper/2410.11042/) (2024)
- [Looking Beyond The Top-1: Transformers Determine Top Tokens In Order](https://mlsanity.com/paper/2410.20210/) (2024)
- [LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models](https://mlsanity.com/paper/2503.16843/) (2025)
- [LittleBit: Ultra Low-Bit Quantization via Latent Factorization](https://mlsanity.com/paper/2506.13771/) (2025)
- [ROCKET: Residual-Oriented Multi-Layer Alignment for Spatially-Aware Vision-Language-Action Models](https://mlsanity.com/paper/2602.17951/) (2026)
- [Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformers](https://mlsanity.com/paper/2410.13184/) (2024)
- [Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference](https://mlsanity.com/paper/2512.16391/) (2025)
- [From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Models](https://mlsanity.com/paper/2601.18091/) (2026)
- [Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Models](https://mlsanity.com/paper/2509.17588/) (2025)
- [RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decoding](https://mlsanity.com/paper/2505.22135/) (2025)
- [Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning](https://mlsanity.com/paper/2601.06787/) (2026)
- [SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference](https://mlsanity.com/paper/2508.02751/) (2025)
- [Emergent Extreme-View Geometry in 3D Foundation Models](https://mlsanity.com/paper/2511.22686/) (2025)
- [Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques](https://mlsanity.com/paper/2503.11816/) (2025)
- [On Relation-Specific Neurons in Large Language Models](https://mlsanity.com/paper/2502.17355/) (2025)
- [Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferability](https://mlsanity.com/paper/2504.10804/) (2025)
- [The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference](https://mlsanity.com/paper/2603.19664/) (2026)
- [Rethinking Layer Relevance in Large Language Models Beyond Cosine Similarity](https://mlsanity.com/paper/2605.14075/) (2026)
- [Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy](https://mlsanity.com/paper/2606.09080/) (2026)
- [Elastic On-Device LLM Service](https://mlsanity.com/paper/2409.09071/) (2024)
- [DeltaLLM: Compress LLMs with Low-Rank Deltas between Shared Weights](https://mlsanity.com/paper/2501.18596/) (2025)
- [UniAttn: Reducing Inference Costs via Softmax Unification for Post-Training LLMs](https://mlsanity.com/paper/2502.00439/) (2025)
- [Reasoning Bias of Next Token Prediction Training](https://mlsanity.com/paper/2502.02007/) (2025)
- [Prompt-based Depth Pruning of Large Language Models](https://mlsanity.com/paper/2502.04348/) (2025)

## Abstract (arXiv)

While scaling Transformer-based large language models (LLMs) has demonstrated promising performance across various tasks, it also introduces redundant architectures, posing efficiency challenges for real-world deployment. Despite some recognition of redundancy in LLMs, the variability of redundancy across different architectures in transformers, such as MLP and Attention layers, is under-explored. In this work, we investigate redundancy across different modules within Transformers, including Blocks, MLP, and Attention layers, using a similarity-based metric. Surprisingly, despite the critical role of attention layers in distinguishing transformers from other architectures, we found that a large portion of these layers exhibit excessively high similarity and can be pruned without degrading performance. For instance, Llama-2-70B achieved a 48.4% speedup with only a 2.4% performance drop by pruning half of the attention layers. Furthermore, by tracing model checkpoints throughout the training process, we observed that attention layer redundancy is inherent and consistent across training stages. Additionally, we further propose a method that jointly drops Attention and MLP layers, allowing us to more aggressively drop additional layers. For instance, when dropping 31 layers (Attention + MLP), Llama-2-13B still retains 90% of the performance on the MMLU task. Our work provides valuable insights for future network architecture design. The code is released at: urlhttps://github.com/Shwai-He/LLM-Drop.