# LSTMs Compose (and Learn) Bottom-Up

arXiv: https://arxiv.org/abs/2010.04650 · HTML: https://mlsanity.com/paper/2010.04650/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 11 of 11)

- [Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs](https://mlsanity.com/paper/2309.07311/) (2023)
- [The paradox of the compositionality of natural language: a neural machine translation case study](https://mlsanity.com/paper/2108.05885/) (2021)
- [Towards a theory of how the structure of language is acquired by deep neural networks](https://mlsanity.com/paper/2406.00048/) (2024)
- [Mechanistic?](https://mlsanity.com/paper/2410.09087/) (2024)
- [The Life Cycle of Knowledge in Big Language Models: A Survey](https://mlsanity.com/paper/2303.07616/) (2023)
- [Feature Interactions Reveal Linguistic Structure in Language Models](https://mlsanity.com/paper/2306.12181/) (2023)
- [Can Interpretation Predict Behavior on Unseen Data?](https://mlsanity.com/paper/2507.06445/) (2025)
- [Training Dynamics of Contextual N-Grams in Language Models](https://mlsanity.com/paper/2311.00863/) (2023)
- [Attention vs non-attention for a Shapley-based explanation method](https://mlsanity.com/paper/2104.12424/) (2021)
- [Representing Syntax and Composition with Geometric Transformations](https://mlsanity.com/paper/2106.01904/) (2021)
- [Using Shapley interactions to understand how models use structure](https://mlsanity.com/paper/2403.13106/) (2024)

## Abstract (arXiv)

Recent work in NLP shows that LSTM language models capture hierarchical structure in language data. In contrast to existing work, we consider the textitlearning process that leads to their compositional behavior. For a closer look at how an LSTM's sequential representations are composed hierarchically, we present a related measure of Decompositional Interdependence (DI) between word meanings in an LSTM, based on their gate interactions. We connect this measure to syntax with experiments on English language data, where DI is higher on pairs of words with lower syntactic distance. To explore the inductive biases that cause these compositional representations to arise during training, we conduct simple experiments on synthetic data. These synthetic experiments support a specific hypothesis about how hierarchical structures are discovered over the course of training: that LSTM constituent representations are learned bottom-up, relying on effective representations of their shorter children, rather than learning the longer-range relations independently from children.