# Grounded and Well-rounded: A Methodological Approach to the Study of Cross-modal and Cross-lingual Grounding

arXiv: https://arxiv.org/abs/2310.11938 · HTML: https://mlsanity.com/paper/2310.11938/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 37 references that are in the map.
- First posted on arXiv 18 Oct 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs](https://mlsanity.com/paper/2010.05953/) (2020)
- [Grounding 'Grounding' in NLP](https://mlsanity.com/paper/2106.02192/) (2021)
- [Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models](https://mlsanity.com/paper/2103.08849/) (2021)
- [Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigation](https://mlsanity.com/paper/2109.14200/) (2021)
- [Emerging Language Spaces Learned From Massively Multilingual Corpora](https://mlsanity.com/paper/1802.00273/) (2018)
- [Grounding and Distinguishing Conceptual Vocabulary Through Similarity Learning in Embodied Simulations](https://mlsanity.com/paper/2305.13668/) (2023)
- [Leverage Points in Modality Shifts: Comparing Language-only and Multimodal Word Representations](https://mlsanity.com/paper/2306.02348/) (2023)

## Built on by (most cited first, 1 of 1)

- [Grounded Concreteness: Human-Like Concreteness Sensitivity in Vision-Language Models](https://mlsanity.com/paper/2601.18065/) (2026)

## Abstract (arXiv)

Grounding has been argued to be a crucial component towards the development of more complete and truly semantically competent artificial intelligence systems. Literature has divided into two camps: While some argue that grounding allows for qualitatively different generalizations, others believe it can be compensated by mono-modal data quantity. Limited empirical evidence has emerged for or against either position, which we argue is due to the methodological challenges that come with studying grounding and its effects on NLP systems. In this paper, we establish a methodological framework for studying what the effects are - if any - of providing models with richer input sources than text-only. The crux of it lies in the construction of comparable samples of populations of models trained on different input modalities, so that we can tease apart the qualitative effects of different input sources from quantifiable model performances. Experiments using this framework reveal qualitative differences in model behavior between cross-modally grounded, cross-lingually grounded, and ungrounded models, which we measure both at a global dataset level as well as for specific word representations, depending on how concrete their semantics is.