# Multimodal Grounding for Language Processing

arXiv: https://arxiv.org/abs/1806.06371 · HTML: https://mlsanity.com/paper/1806.06371/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2018.

## Built on by (most cited first, 14 of 14)

- [Relative Importance in Sentence Processing](https://mlsanity.com/paper/2106.03471/) (2021)
- [Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Case](https://mlsanity.com/paper/2102.11115/) (2021)
- [MultiSubs: A Large-scale Multimodal and Multilingual Dataset](https://mlsanity.com/paper/2103.01910/) (2021)
- [TRACE: Real-Time Multimodal Common Ground Tracking in Situated Collaborative Dialogues](https://mlsanity.com/paper/2503.09511/) (2025)
- [Embodying Pre-Trained Word Embeddings Through Robot Actions](https://mlsanity.com/paper/2104.08521/) (2021)
- [Multimodal Representation Learning and Fusion](https://mlsanity.com/paper/2506.20494/) (2025)
- [Learning to Play Chess from Textbooks (LEAP): a Corpus for Evaluating Chess Moves based on Sentiment Analysis](https://mlsanity.com/paper/2310.20260/) (2023)
- [Aspect-based Sentiment Evaluation of Chess Moves (ASSESS): an NLP-based Method for Evaluating Chess Strategies from Textbooks](https://mlsanity.com/paper/2405.06499/) (2024)
- [Brotherhood at WMT 2024: Leveraging LLM-Generated Contextual Conversations for Cross-Lingual Image Captioning](https://mlsanity.com/paper/2409.15052/) (2024)
- [Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questions](https://mlsanity.com/paper/2502.13034/) (2025)
- [Describe me an Aucklet: Generating Grounded Perceptual Category Descriptions](https://mlsanity.com/paper/2303.04053/) (2023)
- [Learning Multimodal Cues of Children's Uncertainty](https://mlsanity.com/paper/2410.14050/) (2024)
- [A Picture is Worth a Thousand (Correct) Captions: A Vision-Guided Judge-Corrector System for Multimodal Machine Translation](https://mlsanity.com/paper/2511.07010/) (2025)
- [Don't Learn, Ground: A Case for Natural Language Inference with Visual Grounding](https://mlsanity.com/paper/2511.17358/) (2025)

## Abstract (arXiv)

This survey discusses how recent developments in multimodal processing facilitate conceptual grounding of language. We categorize the information flow in multimodal processing with respect to cognitive models of human information processing and analyze different methods for combining multimodal representations. Based on this methodological inventory, we discuss the benefit of multimodal grounding for a variety of language processing tasks and the challenges that arise. We particularly focus on multimodal grounding of verbs which play a crucial role for the compositional power of language.