# What Affects the Effective Depth of Large Language Models?

arXiv: https://arxiv.org/abs/2512.14064 · HTML: https://mlsanity.com/paper/2512.14064/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 12 references that are in the map.
- First posted on arXiv 16 Dec 2025.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Do Language Models Use Their Depth Efficiently?](https://mlsanity.com/paper/2505.13898/) (2025)

## Built on by (most cited first, 4 of 4)

- [Inverse Depth Scaling From Most Layers Being Similar](https://mlsanity.com/paper/2602.05970/) (2026)
- [HRM-Text: Efficient Pretraining Beyond Scaling](https://mlsanity.com/paper/2605.20613/) (2026)
- [Do Transformers Use their Depth Adaptively? Evidence from a Relational Reasoning Task](https://mlsanity.com/paper/2604.12426/) (2026)
- [Increasing Skill Level Recruits Deeper Attention Layers in a Frozen Chess Transformer](https://mlsanity.com/paper/2609.23917/) (2026)

## Abstract (arXiv)

The scaling of large language models (LLMs) emphasizes increasing depth, yet performance gains diminish with added layers. Prior work introduces the concept of "effective depth", arguing that deeper models fail to fully utilize their layers for meaningful computation. Building on this, we systematically study how effective depth varies with model scale, training type, and task difficulty. First, we analyze the model behavior of Qwen-2.5 family (1.5B-32B) and find that while the number of effective layers grows with model size, the effective depth ratio remains stable. Besides, comparisons between base and corresponding long-CoT models show no increase in effective depth, suggesting that improved reasoning stems from longer context rather than deeper per-token computation. Furthermore, evaluations across tasks of varying difficulty indicate that models do not dynamically use more layers for harder problems. Our results suggest that current LLMs underuse available depth across scales, training paradigms and tasks of varying difficulties, pointing out research opportunities on increasing the layer utilization rate of LLMs, model pruning, and early exiting. Our code is released at https://github.com/AheadOFpotato/what_affects_effective_depth.