mlsanityData to 7 Oct 2026

What Matters in Transformers? Not All Attention is Needed

arXiv 2406.15786 ↗First posted 22 Jun 2024cs.LG cs.AI cs.CLcited by 83

Not on the map. Why?

Abstract (arXiv)

While scaling Transformer-based large language models (LLMs) has demonstrated promising performance across various tasks, it also introduces redundant architectures, posing efficiency challenges for real-world deployment. Despite some recognition of redundancy in LLMs, the variability of redundancy across different architectures in transformers, such as MLP and Attention layers, is under-explored. In this work, we investigate redundancy across different modules within Transformers, including Blocks, MLP, and Attention layers, using a similarity-based metric. Surprisingly, despite the critical role of attention layers in distinguishing transformers from other architectures, we found that a large portion of these layers exhibit excessively high similarity and can be pruned without degrading performance. For instance, Llama-2-70B achieved a 48.4% speedup with only a 2.4% performance drop by pruning half of the attention layers. Furthermore, by tracing model checkpoints throughout the training process, we observed that attention layer redundancy is inherent and consistent across training stages. Additionally, we further propose a method that jointly drops Attention and MLP layers, allowing us to more aggressively drop additional layers. For instance, when dropping 31 layers (Attention + MLP), Llama-2-13B still retains 90% of the performance on the MMLU task. Our work provides valuable insights for future network architecture design. The code is released at: urlhttps://github.com/Shwai-He/LLM-Drop.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Jun 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 46; 2026: 30.

83 papers.

Sort:

By area:

  1. 2025SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compressioncited by 63
  2. 2025Adaptive Layer-skipping in Pre-trained LLMscited by 24
  3. 2025TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Trainingcited by 21
  4. 2024Puzzle: Distillation-Based NAS for Inference-Optimized LLMscited by 17
  5. 2025Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformerscited by 16
  6. 2025ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layerscited by 13
  7. 2024Looking Beyond The Top-1: Transformers Determine Top Tokens In Ordercited by 12
  8. 2024Persistent Topological Features in Large Language Modelscited by 11
  9. 2025LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Modelscited by 11
  10. 2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11
  11. 2026ROCKET: Residual-Oriented Multi-Layer Alignment for Spatially-Aware Vision-Language-Action Modelscited by 11
  12. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inferencecited by 9
38 more
  1. 2026From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Modelscited by 9
  2. 2024Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformerscited by 8
  3. 2025RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decodingcited by 8
  4. 2025Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Modelscited by 7
  5. 2026Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruningcited by 7
  6. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inferencecited by 5
  7. 2025Emergent Extreme-View Geometry in 3D Foundation Modelscited by 5
  8. 2025On Relation-Specific Neurons in Large Language Modelscited by 4
  9. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniquescited by 4
  10. 2025Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferabilitycited by 4
  11. 2026The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inferencecited by 4
  12. 2026Rethinking Layer Relevance in Large Language Models Beyond Cosine Similaritycited by 4
  13. 2024Elastic On-Device LLM Servicecited by 3
  14. 2025UniAttn: Reducing Inference Costs via Softmax Unification for Post-Training LLMscited by 3
  15. 2025Reasoning Bias of Next Token Prediction Trainingcited by 3
  16. 2025Prompt-based Depth Pruning of Large Language Modelscited by 3
  17. 2025RAP: Runtime Adaptive Pruning for LLM Inferencecited by 3
  18. 2025Recurrence Meets Transformers for Universal Multimodal Retrievalcited by 3
  19. 2025Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguitycited by 3
  20. 2026GradPruner: Gradient-Guided Layer Pruning Enabling Efficient Fine-Tuning and Inference for LLMscited by 3
  21. 2026Drop-Then-Recovery: How Redundant Are Vision-Language-Action Models?cited by 3
  22. 2024EchoAtt: Attend, Copy, then Adjust for More Efficient Large Language Modelscited by 2
  23. 2025DeltaLLM: Compress LLMs with Low-Rank Deltas between Shared Weightscited by 2
  24. 2025When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspectivecited by 2
  25. 2025Prada: Black-Box LLM Adaptation with Private Data on Resource-Constrained Devicescited by 2
  26. 2025Entropy-Based Block Pruning for Efficient Large Language Modelscited by 2
  27. 2025Semantic Retention and Extreme Compression in LLMs: Can We Have Both?cited by 2
  28. 2025Exploring Federated Pruning for Large Language Modelscited by 2
  29. 2025ALPS: Attention Localization and Pruning Strategy for Efficient Alignment of Large Language Modelscited by 2
  30. 2025Diversity of Transformer Layers: One Aspect of Parameter Scaling Lawscited by 2
  31. 2025Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helpscited by 2
  32. 2025AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cachecited by 2
  33. 2025Making Large Language Models Efficient Dense Retrieverscited by 2
  34. 2025Data-Free Pruning of Self-Attention Layers in LLMscited by 2
  35. 2026SpikeMLLM: Spike-based Multimodal Large Language Models via Modality-Specific Temporal Scales and Temporal Compressioncited by 2
  36. 2024ElastiFormer: Learned Redundancy Reduction in Transformer via Self-Distillationcited by 1
  37. 2025Network Dynamics-Based Framework for Understanding Deep Neural Networkscited by 1
  38. 2025FFN Fusion: Rethinking Sequential Computation in Large Language Modelscited by 1

50 most cited of 83 All 83 (JSON)