mlsanityData to 7 Oct 2026

Do Transformers Use their Depth Adaptively? Evidence from a Relational Reasoning Task

arXiv 2604.12426 ↗First posted 14 Apr 2026cs.LG cs.CLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We investigate whether transformers use their depth adaptively across tasks of increasing difficulty. Using a controlled multi-hop relational reasoning task based on family stories, where difficulty is determined by the number of relationship hops that must be composed, we monitor (i) how predictions evolve across layers via early readouts (the logit lens) and (ii) how task-relevant information is integrated across tokens via causal patching. For pretrained models, we find some limited evidence for adaptive depth use: some larger models need fewer layers to arrive at plausible answers for easier tasks, and models generally use more layers to integrate information across tokens as chain length increases. For models finetuned on the task, we find clearer and more consistent evidence of adaptive depth use, with the effect being stronger for less constrained finetuning regimes that do not preserve general language modeling abilities.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 26 of its 32 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

26 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Racing Thoughts: Explaining Contextualization Errors in Large Language Models2 Oct
  2. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrations18 Jul

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  5. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  6. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  7. 2024LoRA Learns Less and Forgets Lesscited by 313
  8. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 223
  9. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  10. 2022Predictability and Surprise in Large Generative Modelscited by 209
  11. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 155
  12. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
12 more
  1. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 137
  2. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 97
  3. 2024Looped Transformers for Length Generalizationcited by 91
  4. 2024Understanding Transformer Reasoning Capabilities via Graph Algorithmscited by 64
  5. 2024Transformer Layers as Painterscited by 50
  6. 2025A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformerscited by 49
  7. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  8. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 47
  9. 2024Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Modelscited by 30
  10. 2025How Do Transformers Learn Variable Binding in Symbolic Programs?cited by 19
  11. 2025How Do LLMs Use Their Depth?cited by 15
  12. 2025What Affects the Effective Depth of Large Language Models?cited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025How do Transformers Learn Implicit Reasoning?
  2. 2026Unveiling the Reasoning Process of Large Language Models
  3. 2026Do Agents Think Deeper? A Mechanistic Investigation of Layer-Wise Dynamics in Sequential Planning
  4. 2026Not All Thinking is Created Equal: Latent Reasoning Discovers a Recurrent Search Algorithm for Depth Generalization
  5. 2025The Curious Case of Analogies: Investigating Analogical Reasoning in Large Language Models
  6. 2026Quantifying the Necessity of Chain of Thought through Opaque Serial Depth
  7. 2025Disentangling Recall and Reasoning in Transformer Models through Layer-wise Attention and Activation Analysis
  8. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization