# Vision-Language Models Struggle to Align Entities across Modalities

arXiv: https://arxiv.org/abs/2503.03854 · HTML: https://mlsanity.com/paper/2503.03854/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 3 of its 31 references that are in the map.
- First posted on arXiv 5 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 3 of 3)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Modeling human visual search: A combined Bayesian searcher and saliency map approach for eye movement guidance in natural scenes](https://mlsanity.com/paper/2009.08373/) (2020)

## Built on by (most cited first, 14 of 14)

- [A Survey of Multimodal Hallucination Evaluation and Detection](https://mlsanity.com/paper/2507.19024/) (2025)
- [MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphs](https://mlsanity.com/paper/2507.20804/) (2025)
- [Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs](https://mlsanity.com/paper/2512.08923/) (2025)
- [mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.24073/) (2025)
- [UNICBench: UNIfied Counting Benchmark for MLLM](https://mlsanity.com/paper/2603.00595/) (2026)
- [TextTIGER: Text-based Intelligent Generation with Entity Prompt Refinement for Text-to-Image Generation](https://mlsanity.com/paper/2504.18269/) (2025)
- [Cross-Modal Coreference Alignment: Enabling Reliable Information Transfer in Omni-LLMs](https://mlsanity.com/paper/2604.05522/) (2026)
- [Towards Multimodal Graph Large Language Model](https://mlsanity.com/paper/2506.09738/) (2025)
- [Where is the multimodal goal post? On the Ability of Foundation Models to Recognize Highlights in Videos](https://mlsanity.com/paper/2601.16333/) (2026)
- [CAPT: Confusion-Aware Prompt Tuning for Reducing Vision-Language Misalignment](https://mlsanity.com/paper/2603.02557/) (2026)
- [DISC: Decoupling Instruction from State-Conditioned Control via Policy Generation](https://mlsanity.com/paper/2605.20856/) (2026)
- [TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs](https://mlsanity.com/paper/2607.28640/) (2026)
- [BindCLIP: One Balanced Coupling For Compositional Vision Language Scoring](https://mlsanity.com/paper/2609.23717/) (2026)
- [Human-Like Attention? A Psychophysical Comparison of Visual Search in Humans and MLLMs](https://mlsanity.com/paper/2610.05463/) (2026)

## Abstract (arXiv)

Cross-modal entity linking refers to the ability to align entities and their attributes across different modalities. While cross-modal entity linking is a fundamental skill needed for real-world applications such as multimodal code generation, fake news detection, or scene understanding, it has not been thoroughly studied in the literature. In this paper, we introduce a new task and benchmark to address this gap. Our benchmark, MATE, consists of 5.5k evaluation instances featuring visual scenes aligned with their textual representations. To evaluate cross-modal entity linking performance, we design a question-answering task that involves retrieving one attribute of an object in one modality based on a unique attribute of that object in another modality. We evaluate state-of-the-art Vision-Language Models (VLMs) and humans on this task, and find that VLMs struggle significantly compared to humans, particularly as the number of objects in the scene increases. Our analysis also shows that, while chain-of-thought prompting can improve VLM performance, models remain far from achieving human-level proficiency. These findings highlight the need for further research in cross-modal entity linking and show that MATE is a strong benchmark to support that progress.