# Learning compositionally through attentive guidance

arXiv: https://arxiv.org/abs/1805.09657 · HTML: https://mlsanity.com/paper/1805.09657/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 May 2018.

## Built on by (most cited first, 11 of 11)

- [The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization](https://mlsanity.com/paper/2110.07732/) (2021)
- [Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks](https://mlsanity.com/paper/2311.12997/) (2023)
- [Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformers](https://mlsanity.com/paper/2501.08537/) (2025)
- [Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing](https://mlsanity.com/paper/2405.05409/) (2024)
- [Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN](https://mlsanity.com/paper/2107.01366/) (2021)
- [An Analysis for Reasoning Bias of Language Models with Small Initialization](https://mlsanity.com/paper/2502.04375/) (2025)
- [CTL++: Evaluating Generalization on Never-Seen Compositional Patterns of Known Functions, and Compatibility of Neural Representations](https://mlsanity.com/paper/2210.06350/) (2022)
- [On the Interplay Between Fine-tuning and Composition in Transformers](https://mlsanity.com/paper/2105.14668/) (2021)
- [One Semantic Parser to Parse Them All: Sequence to Sequence Multi-Task Learning on Semantic Parsing Datasets](https://mlsanity.com/paper/2106.04476/) (2021)
- [Layer-wise Representation Fusion for Compositional Generalization](https://mlsanity.com/paper/2307.10799/) (2023)
- [Self-Organising Neural Discrete Representation Learning `a la Kohonen](https://mlsanity.com/paper/2302.07950/) (2023)

## Abstract (arXiv)

While neural network models have been successfully applied to domains that require substantial generalisation skills, recent studies have implied that they struggle when solving the task they are trained on requires inferring its underlying compositional structure. In this paper, we introduce Attentive Guidance, a mechanism to direct a sequence to sequence model equipped with attention to find more compositional solutions. We test it on two tasks, devised precisely to assess the compositional capabilities of neural models, and we show that vanilla sequence to sequence models with attention overfit the training distribution, while the guided versions come up with compositional solutions that fit the training and testing distributions almost equally well. Moreover, the learned solutions generalise even in cases where the training and testing distributions strongly diverge. In this way, we demonstrate that sequence to sequence models are capable of finding compositional solutions without requiring extra components. These results helps to disentangle the causes for the lack of systematic compositionality in neural networks, which can in turn fuel future work.