# Modular Networks for Compositional Instruction Following

arXiv: https://arxiv.org/abs/2010.12764 · HTML: https://mlsanity.com/paper/2010.12764/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2020.

## Built on by (most cited first, 5 of 5)

- [Continual Learning via Local Module Composition](https://mlsanity.com/paper/2111.07736/) (2021)
- [Look Wide and Interpret Twice: Improving Performance on Interactive Instruction-following Tasks](https://mlsanity.com/paper/2106.00596/) (2021)
- [Modular Framework for Visuomotor Language Grounding](https://mlsanity.com/paper/2109.02161/) (2021)
- [Are you doing what I say? On modalities alignment in ALFRED](https://mlsanity.com/paper/2110.05665/) (2021)
- [Machine Intelligence that Understands Visual and Linguistic Information and Interacts with Humans and Environments](https://mlsanity.com/paper/2605.24020/) (2026)

## Abstract (arXiv)

Standard architectures used in instruction following often struggle on novel compositions of subgoals (e.g. navigating to landmarks or picking up objects) observed during training. We propose a modular architecture for following natural language instructions that describe sequences of diverse subgoals. In our approach, subgoal modules each carry out natural language instructions for a specific subgoal type. A sequence of modules to execute is chosen by learning to segment the instructions and predicting a subgoal type for each segment. When compared to standard, non-modular sequence-to-sequence approaches on ALFRED, a challenging instruction following benchmark, we find that modularization improves generalization to novel subgoal compositions, as well as to environments unseen in training.