# Tree-structured composition in neural networks without tree-structured architectures

arXiv: https://arxiv.org/abs/1506.04834 · HTML: https://mlsanity.com/paper/1506.04834/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2015.

## Built on by (most cited first, 23 of 23)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [A Benchmark for Compositional Visual Reasoning](https://mlsanity.com/paper/2206.05379/) (2022)
- [Examining the Inductive Bias of Neural Language Models with Artificial Languages](https://mlsanity.com/paper/2106.01044/) (2021)
- [Sparse Universal Transformer](https://mlsanity.com/paper/2310.07096/) (2023)
- [Can Models Learn Skill Composition from Examples?](https://mlsanity.com/paper/2409.19808/) (2024)
- [Learning Hierarchical Structures with Differentiable Nondeterministic Stacks](https://mlsanity.com/paper/2109.01982/) (2021)
- [Modeling Hierarchical Structures with Continuous Recursive Neural Networks](https://mlsanity.com/paper/2106.06038/) (2021)
- [A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practices](https://mlsanity.com/paper/2406.08787/) (2024)
- [A Theory for Length Generalization in Learning to Reason](https://mlsanity.com/paper/2404.00560/) (2024)
- [Whither symbols in the era of advanced neural networks?](https://mlsanity.com/paper/2508.05776/) (2025)
- [Beam Tree Recursive Cells](https://mlsanity.com/paper/2305.19999/) (2023)
- [Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability](https://mlsanity.com/paper/2311.04449/) (2023)
- [Investigating Recurrent Transformers with Dynamic Halt](https://mlsanity.com/paper/2402.00976/) (2024)
- [Efficient Beam Tree Recursion](https://mlsanity.com/paper/2307.10779/) (2023)
- [Tracing and Manipulating Intermediate Values in Neural Math Problem Solvers](https://mlsanity.com/paper/2301.06758/) (2023)
- [ORCHARD: A Benchmark For Measuring Systematic Generalization of Multi-Hierarchical Reasoning](https://mlsanity.com/paper/2111.14034/) (2021)
- [Language Cognition and Language Computation -- Human and Machine Language Understanding](https://mlsanity.com/paper/2301.04788/) (2023)
- [On the Design Space Between Transformers and Recursive Neural Nets](https://mlsanity.com/paper/2409.01531/) (2024)
- [FastTrees: Parallel Latent Tree-Induction for Faster Sequence Encoding](https://mlsanity.com/paper/2111.14031/) (2021)
- [Is the Computation of Abstract Sameness Relations Human-Like in Neural Language Models?](https://mlsanity.com/paper/2205.06149/) (2022)
- [Syntactic Inductive Biases for Deep Learning Methods](https://mlsanity.com/paper/2206.04806/) (2022)
- [A Short Survey of Systematic Generalization](https://mlsanity.com/paper/2211.11956/) (2022)
- [Ordered Memory Baselines](https://mlsanity.com/paper/2302.06451/) (2023)

## Abstract (arXiv)

Tree-structured neural networks encode a particular tree geometry for a sentence in the network design. However, these models have at best only slightly outperformed simpler sequence-based models. We hypothesize that neural sequence models like LSTMs are in fact able to discover and implicitly use recursive compositional structure, at least for tasks with clear cues to that structure in the data. We demonstrate this possibility using an artificial data task for which recursive compositional structure is crucial, and find an LSTM-based sequence model can indeed learn to exploit the underlying tree structure. However, its performance consistently lags behind that of tree models, even on large training sets, suggesting that tree-structured models are more effective at exploiting recursive structure.