# Learning to Compose Words into Sentences with Reinforcement Learning

arXiv: https://arxiv.org/abs/1611.09100 · HTML: https://mlsanity.com/paper/1611.09100/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2016.

## Built on by (most cited first, 30 of 30)

- [R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling](https://mlsanity.com/paper/2107.00967/) (2021)
- [Structured Reordering for Modeling Latent Alignments in Sequence Transduction](https://mlsanity.com/paper/2106.03257/) (2021)
- [Modeling Hierarchical Structures with Continuous Recursive Neural Networks](https://mlsanity.com/paper/2106.06038/) (2021)
- [Fast-R2D2: A Pretrained Recursive Neural Network based on Pruned CKY for Grammar Induction and Text Representation](https://mlsanity.com/paper/2203.00281/) (2022)
- [Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale](https://mlsanity.com/paper/2403.08293/) (2024)
- [Leveraging Recursive Gumbel-Max Trick for Approximate Inference in Combinatorial Spaces](https://mlsanity.com/paper/2110.15072/) (2021)
- [Beam Tree Recursive Cells](https://mlsanity.com/paper/2305.19999/) (2023)
- [Augmenting Transformers with Recursively Composed Multi-grained Representations](https://mlsanity.com/paper/2309.16319/) (2023)
- [Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability](https://mlsanity.com/paper/2311.04449/) (2023)
- [Learning Syntax from Naturally-Occurring Bracketings](https://mlsanity.com/paper/2104.13933/) (2021)
- [Rule Augmented Unsupervised Constituency Parsing](https://mlsanity.com/paper/2105.10193/) (2021)
- [Efficient Beam Tree Recursion](https://mlsanity.com/paper/2307.10779/) (2023)
- [Weakly Supervised Reasoning by Neuro-Symbolic Approaches](https://mlsanity.com/paper/2309.13072/) (2023)
- [Tree Transformers are an Ineffective Model of Syntactic Constituency](https://mlsanity.com/paper/2411.16993/) (2024)
- [Learning with Latent Structures in Natural Language Processing: A Survey](https://mlsanity.com/paper/2201.00490/) (2022)
- [Learning Discrete Structured Variational Auto-Encoder using Natural Evolution Strategies](https://mlsanity.com/paper/2205.01324/) (2022)
- [StrAE: Autoencoding for Pre-Trained Embeddings using Explicit Structure](https://mlsanity.com/paper/2305.05588/) (2023)
- [The Emergence of Chunking Structures with Hierarchical RNN](https://mlsanity.com/paper/2309.04919/) (2023)
- [Unsupervised Learning of Explainable Parse Trees for Improved Generalisation](https://mlsanity.com/paper/2104.04998/) (2021)
- [ORCHARD: A Benchmark For Measuring Systematic Generalization of Multi-Hierarchical Reasoning](https://mlsanity.com/paper/2111.14034/) (2021)
- [A Neural Model for Regular Grammar Induction](https://mlsanity.com/paper/2209.11628/) (2022)
- [Uncertainty-Aware Reward-based Deep Reinforcement Learning for Intent Analysis of Social Media Information](https://mlsanity.com/paper/2302.10195/) (2023)
- [Unsupervised Full Constituency Parsing with Neighboring Distribution Divergence](https://mlsanity.com/paper/2110.15931/) (2021)
- [FastTrees: Parallel Latent Tree-Induction for Faster Sequence Encoding](https://mlsanity.com/paper/2111.14031/) (2021)
- [Recognizing semantic relation in sentence pairs using Tree-RNNs and Typed dependencies](https://mlsanity.com/paper/2201.04810/) (2022)
- [Syntactic Inductive Biases for Deep Learning Methods](https://mlsanity.com/paper/2206.04806/) (2022)
- [Unsupervised Inference of Data-Driven Discourse Structures using a Tree Auto-Encoder](https://mlsanity.com/paper/2210.09559/) (2022)
- [Interpreting Sentiment Composition with Latent Semantic Tree](https://mlsanity.com/paper/2308.16588/) (2023)
- [Grammar Induction from Visual, Speech and Text](https://mlsanity.com/paper/2410.03739/) (2024)
- [Understanding Syntactic Generalization in Structure-inducing Language Models](https://mlsanity.com/paper/2508.07969/) (2025)

## Abstract (arXiv)

We use reinforcement learning to learn tree-structured neural networks for computing representations of natural language sentences. In contrast with prior work on tree-structured models in which the trees are either provided as input or predicted using supervision from explicit treebank annotations, the tree structures in this work are optimized to improve performance on a downstream task. Experiments demonstrate the benefit of learning task-specific composition orders, outperforming both sequential encoders and recursive encoders based on treebank annotations. We analyze the induced trees and show that while they discover some linguistically intuitive structures (e.g., noun phrases, simple verb phrases), they are different than conventional English syntactic structures.