# Generalization in Multimodal Language Learning from Simulation

arXiv: https://arxiv.org/abs/2108.02319 · HTML: https://mlsanity.com/paper/2108.02319/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 3 Aug 2021.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks](https://mlsanity.com/paper/1812.07252/) (2018)
- [Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?](https://mlsanity.com/paper/2010.12725/) (2020)
- [A Benchmark for Systematic Generalization in Grounded Language Understanding](https://mlsanity.com/paper/2003.05161/) (2020)
- [Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks](https://mlsanity.com/paper/1807.07545/) (2018)
- [Environmental drivers of systematicity and generalization in a situated agent](https://mlsanity.com/paper/1910.00571/) (2019)
- [Modeling Semantic Compositionality with Sememe Knowledge](https://mlsanity.com/paper/1907.04744/) (2019)
- [Crossmodal Language Grounding in an Embodied Neurocognitive Model](https://mlsanity.com/paper/2006.13546/) (2020)
- [Sensorimotor Input as a Language Generalisation Tool: A Neurorobotics Model for Generation and Generalisation of Noun-Verb Combinations with Sensorimotor Inputs](https://mlsanity.com/paper/1605.03261/) (2016)

## Built on by (most cited first, 6 of 6)

- [Language Model-Based Paired Variational Autoencoders for Robotic Language Learning](https://mlsanity.com/paper/2201.06317/) (2022)
- [Impact Makes a Sound and Sound Makes an Impact: Sound Guides Representations and Explorations](https://mlsanity.com/paper/2208.02680/) (2022)
- [Learning Flexible Translation between Robot Actions and Language Descriptions](https://mlsanity.com/paper/2207.07437/) (2022)
- [Learning Bidirectional Action-Language Translation with Limited Supervision and Incongruent Input](https://mlsanity.com/paper/2301.03353/) (2023)
- [Enhancing Length Extrapolation in Sequential Models with Pointer-Augmented Neural Memory](https://mlsanity.com/paper/2404.11870/) (2024)
- [Skill Generalization with Verbs](https://mlsanity.com/paper/2410.14118/) (2024)

## Abstract (arXiv)

Neural networks can be powerful function approximators, which are able to model high-dimensional feature distributions from a subset of examples drawn from the target distribution. Naturally, they perform well at generalizing within the limits of their target function, but they often fail to generalize outside of the explicitly learned feature space. It is therefore an open research topic whether and how neural network-based architectures can be deployed for systematic reasoning. Many studies have shown evidence for poor generalization, but they often work with abstract data or are limited to single-channel input. Humans, however, learn and interact through a combination of multiple sensory modalities, and rarely rely on just one. To investigate compositional generalization in a multimodal setting, we generate an extensible dataset with multimodal input sequences from simulation. We investigate the influence of the underlying training data distribution on compostional generalization in a minimal LSTM-based network trained in a supervised, time continuous setting. We find compositional generalization to fail in simple setups while improving with the number of objects, actions, and particularly with a lot of color overlaps between objects. Furthermore, multimodality strongly improves compositional generalization in settings where a pure vision model struggles to generalize.