# The Impact of Depth on Compositional Generalization in Transformer Language Models

arXiv: https://arxiv.org/abs/2310.19956 · HTML: https://mlsanity.com/paper/2310.19956/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 42 references that are in the map.
- First posted on arXiv 30 Oct 2023.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [COGS: A Compositional Generalization Challenge Based on Semantic Interpretation](https://mlsanity.com/paper/2010.05465/) (2020)
- [Prompting GPT-3 To Be Reliable](https://mlsanity.com/paper/2210.09150/) (2022)
- [Saturated Transformers are Constant-Depth Threshold Circuits](https://mlsanity.com/paper/2106.16213/) (2021)
- [The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers](https://mlsanity.com/paper/2108.12284/) (2021)
- [Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers](https://mlsanity.com/paper/2109.10686/) (2021)
- [Do Attention Heads in BERT Track Syntactic Dependencies?](https://mlsanity.com/paper/1911.12246/) (2019)
- [Making Transformers Solve Compositional Tasks](https://mlsanity.com/paper/2108.04378/) (2021)
- [Compression of Deep Learning Models for Text: A Survey](https://mlsanity.com/paper/2008.05221/) (2020)
- [Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing](https://mlsanity.com/paper/2205.12253/) (2022)
- [Characterizing Intrinsic Compositionality in Transformers with Tree Projections](https://mlsanity.com/paper/2211.01288/) (2022)
- [Injecting structural hints: Using language models to study inductive biases in language learning](https://mlsanity.com/paper/2304.13060/) (2023)
- [Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models](https://mlsanity.com/paper/2203.09397/) (2022)
- [Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Models](https://mlsanity.com/paper/2212.10769/) (2022)
- [Wide Attention Is The Way Forward For Transformers?](https://mlsanity.com/paper/2210.00640/) (2022)

## Built on by (most cited first, 30 of 38)

- [Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries](https://mlsanity.com/paper/2406.12775/) (2024)
- [DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging](https://mlsanity.com/paper/2402.02622/) (2024)
- [A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers](https://mlsanity.com/paper/2503.03961/) (2025)
- [Theoretical limitations of multi-layer Transformer](https://mlsanity.com/paper/2412.02975/) (2024)
- [How Does Code Pretraining Affect Language Model Task Performance?](https://mlsanity.com/paper/2409.04556/) (2024)
- [Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?](https://mlsanity.com/paper/2411.16679/) (2024)
- [Gemstones: A Model Suite for Multi-Faceted Scaling Laws](https://mlsanity.com/paper/2502.06857/) (2025)
- [The Asymptotic Behavior of Attention in Transformers](https://mlsanity.com/paper/2412.02682/) (2024)
- [Position: A Theory of Deep Learning Must Include Compositional Sparsity](https://mlsanity.com/paper/2507.02550/) (2025)
- [Exploring Diffusion Transformer Designs via Grafting](https://mlsanity.com/paper/2506.05340/) (2025)
- [Lost in Backpropagation: The LM Head is a Gradient Bottleneck](https://mlsanity.com/paper/2603.10145/) (2026)
- [Breaking Chains: Unraveling the Links in Multi-Hop Knowledge Unlearning](https://mlsanity.com/paper/2410.13274/) (2024)
- [Can Interpretation Predict Behavior on Unseen Data?](https://mlsanity.com/paper/2507.06445/) (2025)
- [Super Tiny Language Models](https://mlsanity.com/paper/2405.14159/) (2024)
- [ALTA: Compiler-Based Analysis of Transformers](https://mlsanity.com/paper/2410.18077/) (2024)
- [Random Scaling of Emergent Capabilities](https://mlsanity.com/paper/2502.17356/) (2025)
- [Scaling can lead to compositional generalization](https://mlsanity.com/paper/2507.07207/) (2025)
- [TradeFM: A Generative Foundation Model for Trade-flow and Market Microstructure](https://mlsanity.com/paper/2602.23784/) (2026)
- [Interpreting token compositionality in LLMs: A robustness analysis](https://mlsanity.com/paper/2410.12924/) (2024)
- [Counterfactual reasoning: an analysis of in-context emergence](https://mlsanity.com/paper/2506.05188/) (2025)
- [Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs](https://mlsanity.com/paper/2510.18245/) (2025)
- [Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data?](https://mlsanity.com/paper/2501.15857/) (2025)
- [Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training](https://mlsanity.com/paper/2509.06518/) (2025)
- [CompoST: A Benchmark for Analyzing the Ability of LLMs To Compositionally Interpret Questions in a QALD Setting](https://mlsanity.com/paper/2507.21257/) (2025)
- [Revisiting the Shape Convention of Transformer Language Models](https://mlsanity.com/paper/2602.06471/) (2026)
- [Single-Pass, Depth-Selective Reading for Multi-Aspect Sentiment Analysis](https://mlsanity.com/paper/2605.20998/) (2026)
- [The Deterministic Horizon: When Extended Reasoning Fails and Tool Delegation Becomes Necessary](https://mlsanity.com/paper/2606.00376/) (2026)
- [Variable-Width Transformers](https://mlsanity.com/paper/2606.18246/) (2026)
- [DepthBench: Measuring How Residual Connections Enable More Computational Depth](https://mlsanity.com/paper/2609.32534/) (2026)
- [Exploring Compositional Generalization (in COGS/ReCOGS_pos) by Transformers using Restricted Access Sequence Processing (RASP)](https://mlsanity.com/paper/2504.15349/) (2025)

## Abstract (arXiv)

To process novel sentences, language models (LMs) must generalize compositionally -- combine familiar elements in new ways. What aspects of a model's structure promote compositional generalization? Focusing on transformers, we test the hypothesis, motivated by theoretical and empirical work, that deeper transformers generalize more compositionally. Simply adding layers increases the total number of parameters; to address this confound between depth and size, we construct three classes of models which trade off depth for width such that the total number of parameters is kept constant (41M, 134M and 374M parameters). We pretrain all models as LMs and fine-tune them on tasks that test for compositional generalization. We report three main conclusions: (1) after fine-tuning, deeper models generalize more compositionally than shallower models do, but the benefit of additional layers diminishes rapidly; (2) within each family, deeper models show better language modeling performance, but returns are similarly diminishing; (3) the benefits of depth for compositional generalization cannot be attributed solely to better performance on language modeling. Because model latency is approximately linear in the number of layers, these results lead us to the recommendation that, with a given total parameter budget, transformers can be made shallower than is typical without sacrificing performance.