# Understanding Performance Collapse in Layer-Pruned Large Language Models via Decision Representation Transitions

arXiv: https://arxiv.org/abs/2605.07271 · HTML: https://mlsanity.com/paper/2605.07271/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 33 references that are in the map.
- First posted on arXiv 8 May 2026.

## Builds on (most cited first, 28 of 28)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://mlsanity.com/paper/2401.15024/) (2024)
- [ShortGPT: Layers in Large Language Models are More Redundant Than You Expect](https://mlsanity.com/paper/2403.03853/) (2024)
- [The Unreasonable Ineffectiveness of the Deeper Layers](https://mlsanity.com/paper/2403.17887/) (2024)
- [LaCo: Large Language Model Pruning via Layer Collapse](https://mlsanity.com/paper/2402.11187/) (2024)
- [LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation](https://mlsanity.com/paper/2306.11222/) (2023)
- [Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods](https://mlsanity.com/paper/2402.02834/) (2024)
- [Learngene: From Open-World to Your Learning Task](https://mlsanity.com/paper/2106.06788/) (2021)
- [Learngene: Inheriting Condensed Knowledge from the Ancestry Model to Descendant Models](https://mlsanity.com/paper/2305.02279/) (2023)
- [LogitLens4LLMs: Extending Logit Lens Analysis to Modern Large Language Models](https://mlsanity.com/paper/2503.11667/) (2025)
- [Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging](https://mlsanity.com/paper/2406.16330/) (2024)
- [Transformer as Linear Expansion of Learngene](https://mlsanity.com/paper/2312.05614/) (2023)
- [Exploring Learngene via Stage-wise Weight Sharing for Initializing Variable-sized Models](https://mlsanity.com/paper/2404.16897/) (2024)
- [Building Variable-sized Models via Learngene Pool](https://mlsanity.com/paper/2312.05743/) (2023)
- [Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation](https://mlsanity.com/paper/2510.15304/) (2025)
- [IG-Pruning: Input-Guided Block Pruning for Large Language Models](https://mlsanity.com/paper/2511.02213/) (2025)
- [PocketLLM: Ultimate Compression of Large Language Models via Meta Networks](https://mlsanity.com/paper/2511.17637/) (2025)

## Built on by (most cited first, 2 of 2)

- [Beyond Depth Truncation: Controlled Evaluation of Depth Utilization in Recursive Language Models](https://mlsanity.com/paper/2609.19934/) (2026)
- [Weight-Adjusted Gradients Reveal Parameter Importance and Failure Modes in LLMs](https://mlsanity.com/paper/2607.10803/) (2026)

## Abstract (arXiv)

Layer pruning efficiently reduces Large Language Model (LLM) computational costs but often triggers sudden performance collapse. Existing representation-based analyses struggle to explain this mechanism. We propose studying pruning through decision representation. Focusing on multiple-choice tasks, we introduce two metrics, Decision Margin and Option Frequency, and an Iterative Pruning method to analyze layer-wise decision dynamics. Our findings reveal a sharp decision transition that partitions the network into two stages: a Silent Phase, where the model cannot yet predict the correct answer, and a Decisive Phase, where the correct prediction emerges. We also find that pruning the Decisive Phase has minimal impact, whereas pruning the Silent Phase triggers immediate performance collapse, highlighting its extreme sensitivity to structural changes. Therefore, we conclude that pruning-induced collapse stems from disrupting the Silent Phase, which prevents the critical decision transition from occurring.