# Analogical Reasoning for Visually Grounded Language Acquisition

arXiv: https://arxiv.org/abs/2007.11668 · HTML: https://mlsanity.com/paper/2007.11668/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jul 2020.

## Built on by (most cited first, 4 of 4)

- [Multimodal Analogical Reasoning over Knowledge Graphs](https://mlsanity.com/paper/2210.00312/) (2022)
- [Navigating Hallucinations for Reasoning of Unintentional Activities](https://mlsanity.com/paper/2402.19405/) (2024)
- [Improving the Robustness of Neural Multiplication Units with Reversible Stochasticity](https://mlsanity.com/paper/2211.05624/) (2022)
- [Enhancing Length Extrapolation in Sequential Models with Pointer-Augmented Neural Memory](https://mlsanity.com/paper/2404.11870/) (2024)

## Abstract (arXiv)

Children acquire language subconsciously by observing the surrounding world and listening to descriptions. They can discover the meaning of words even without explicit language knowledge, and generalize to novel compositions effortlessly. In this paper, we bring this ability to AI, by studying the task of Visually grounded Language Acquisition (VLA). We propose a multimodal transformer model augmented with a novel mechanism for analogical reasoning, which approximates novel compositions by learning semantic mapping and reasoning operations from previously seen compositions. Our proposed method, Analogical Reasoning Transformer Networks (ARTNet), is trained on raw multimedia data (video frames and transcripts), and after observing a set of compositions such as "washing apple" or "cutting carrot", it can generalize and recognize new compositions in new video frames, such as "washing carrot" or "cutting apple". To this end, ARTNet refers to relevant instances in the training data and uses their visual features and captions to establish analogies with the query image. Then it chooses the suitable verb and noun to create a new composition that describes the new image best. Extensive experiments on an instructional video dataset demonstrate that the proposed method achieves significantly better generalization capability and recognition accuracy compared to state-of-the-art transformer models.