# Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Case

arXiv: https://arxiv.org/abs/2102.11115 · HTML: https://mlsanity.com/paper/2102.11115/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 22 Feb 2021.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Compositionality decomposed: how do neural networks generalise?](https://mlsanity.com/paper/1908.08351/) (2019)
- [Zero-Shot Learning Through Cross-Modal Transfer](https://mlsanity.com/paper/1301.3666/) (2013)
- [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://mlsanity.com/paper/1608.04207/) (2016)
- [Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?](https://mlsanity.com/paper/2005.01831/) (2020)
- [FOIL it! Find One mismatch between Image and Language caption](https://mlsanity.com/paper/1705.01359/) (2017)
- [Understanding the Origins of Bias in Word Embeddings](https://mlsanity.com/paper/1810.03611/) (2018)
- [Multi-modal Cycle-consistent Generalized Zero-Shot Learning](https://mlsanity.com/paper/1808.00136/) (2018)
- [Probing for Semantic Classes: Diagnosing the Meaning Content of Word Embeddings](https://mlsanity.com/paper/1906.03608/) (2019)
- [Multimodal Grounding for Language Processing](https://mlsanity.com/paper/1806.06371/) (2018)
- [Learning Visually-Grounded Semantics from Contrastive Adversarial Samples](https://mlsanity.com/paper/1806.10348/) (2018)
- [HAL: Improved Text-Image Matching by Mitigating Visual Semantic Hubs](https://mlsanity.com/paper/1911.10097/) (2019)

## Built on by (most cited first, 20 of 20)

- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Cross-modal Information Flow in Multimodal Large Language Models](https://mlsanity.com/paper/2411.18620/) (2024)
- [VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers](https://mlsanity.com/paper/2203.17247/) (2022)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions](https://mlsanity.com/paper/2107.13782/) (2021)
- [A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models](https://mlsanity.com/paper/2502.17516/) (2025)
- [Can Vision-Language Models be a Good Guesser? Exploring VLMs for Times and Location Reasoning](https://mlsanity.com/paper/2307.06166/) (2023)
- [Probing a Vision-Language-Action Model for Symbolic States and Integration into a Cognitive Architecture](https://mlsanity.com/paper/2502.04558/) (2025)
- [Probing Multimodal Large Language Models for Global and Local Semantic Representations](https://mlsanity.com/paper/2402.17304/) (2024)
- [Lost in Space: Probing Fine-grained Spatial Understanding in Vision and Language Resamplers](https://mlsanity.com/paper/2404.13594/) (2024)
- [What Vision-Language Models `See' when they See Scenes](https://mlsanity.com/paper/2109.07301/) (2021)
- [An Empirical Study on How Video-LLMs Answer Video Questions](https://mlsanity.com/paper/2508.15360/) (2025)
- [Probing Cross-modal Semantics Alignment Capability from the Textual Perspective](https://mlsanity.com/paper/2210.09550/) (2022)
- [Quantifying and Enabling the Interpretability of CLIP-like Models](https://mlsanity.com/paper/2409.06579/) (2024)
- [Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models](https://mlsanity.com/paper/2212.00281/) (2022)
- [Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models](https://mlsanity.com/paper/2410.12011/) (2024)
- [A Survey on Interpretability in Visual Recognition](https://mlsanity.com/paper/2507.11099/) (2025)
- [Pruning the Paradox: How CLIP's Most Informative Heads Enhance Performance While Amplifying Bias](https://mlsanity.com/paper/2503.11103/) (2025)
- [Towards Human Cognition: Visual Context Guides Syntactic Priming in Fusion-Encoded Models](https://mlsanity.com/paper/2502.17669/) (2025)
- [Vision-language models know more about agriculture than they show and rubric-grounded verifications close the gap](https://mlsanity.com/paper/2609.09417/) (2026)

## Abstract (arXiv)

Semantic embeddings have advanced the state of the art for countless natural language processing tasks, and various extensions to multimodal domains, such as visual-semantic embeddings, have been proposed. While the power of visual-semantic embeddings comes from the distillation and enrichment of information through machine learning, their inner workings are poorly understood and there is a shortage of analysis tools. To address this problem, we generalize the notion of probing tasks to the visual-semantic case. To this end, we (i) discuss the formalization of probing tasks for embeddings of image-caption pairs, (ii) define three concrete probing tasks within our general framework, (iii) train classifiers to probe for those properties, and (iv) compare various state-of-the-art embeddings under the lens of the proposed probing tasks. Our experiments reveal an up to 12% increase in accuracy on visual-semantic embeddings compared to the corresponding unimodal embeddings, which suggest that the text and image dimensions represented in the former do complement each other.