mlsanityData to 14 Oct 2026

When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models

arXiv 2404.08634 ↗First posted 12 Apr 2024cs.CL cs.AI cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are known for their performance, but we uncover a significant structural inefficiency: a phenomenon we term attention collapse. In many pre-trained decoder-style LLMs, the attention matrices in deeper layers degenerate, collapsing to near rank-one structures. These underutilized layers, which we call lazy layers, are redundant and impair model efficiency. To address this, we introduce Inheritune, a simple yet powerful training recipe designed to build smaller, stronger language models. Inheritune initializes a compact model by inheriting the potent early layers from a larger pre-trained model and then progressively trains and expands it. Our experiments on various models, including the GPT-2 family, demonstrate that models trained with Inheritune can match or even surpass the performance of their larger counterparts, despite having significantly fewer layers. This work presents a novel path toward model compression by design, enabling the creation of compact, yet highly performant language models. Code is available at https://github.com/sanyalsunny111/LLM-Inheritune.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 36 of its 55 references that are in the map. First posted on arXiv 12 Apr 2024.

Builds on

36 of 55 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2020Longformer: The Long-Document Transformercited by 3,413
24 more
  1. 2016Pointer Sentinel Mixture Modelscited by 3,096
  2. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  10. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  11. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  12. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  13. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  14. 2023Scaling Data-Constrained Language Modelscited by 403
  15. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  16. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
  17. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
  18. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  19. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 76
  20. 2023Mimetic Initialization of Self-Attention Layerscited by 54
  21. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 45
  22. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 37
  23. 2020A Review of Winograd Schema Challenge Datasets and Approachescited by 27
  24. 2024Stacking as Accelerated Gradient Descentcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 6.

12 papers.

  1. 2025A Survey on Latent Reasoningcited by 81
  2. 2025The Curse of Depth in Large Language Modelscited by 62
  3. 2024InkubaLM: A small language model for low-resource African languagescited by 18
  4. 2024Puzzle: Distillation-Based NAS for Inference-Optimized LLMscited by 17
  5. 2025Tokens for Learning, Tokens for Unlearning: Mitigating Membership Inference Attacks in Large Language Models via Dual-Purpose Trainingcited by 15
  6. 2026Inverse Depth Scaling From Most Layers Being Similarcited by 9
  7. 2026Universal Redundancies in Time Series Foundation Modelscited by 5
  8. 2026Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficientscited by 5
  9. 2024Context is the Key: Backdoor Attacks for In-Context Learning with Vision Transformerscited by 2
  10. 2026Can Scale Save Us From Plasticity Loss in Large Language Models?cited by 1
  11. 2026Performance of Small Language Model Pretraining on FABRIC: An Empirical Studycited by 0
  12. 2026Unveiling the Depth-Performance Dilemma in Split-Federated Fine-tuning of LLMscited by 0