mlsanityData to 7 Oct 2026

MIRROR: Aligning Semantic Relations from Language to Image via Gromov--Wasserstein

arXiv 2606.29462 ↗First posted 28 Jun 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) inherit rich relational priors from their language backbones, yet often fail when asked to apply these relationships in visual contexts. We trace this failure to a structural blind spot: projection-based alignment trains each visual token to carry the right semantics, but never asks whether the relationships between concepts survive the crossing from language to vision. To address this, we propose MIRROR (Mapping Inter-concept Relations from language to visual Representation via Optimal-transport-based Regularization), a geometric regularization framework that transfers relational priors from language to vision by exploiting the rich relational structure encoded in language representations. Specifically, we derive a surrogate loss from the proposed Semi-Inverse Gromov-Wasserstein (SI-GW) problem, an inverse geometric problem that aligns visual representations with language-derived relational priors. We show that this formulation admits a unique closed-form solution that prescribes the ideal visual relational structure implied by language geometry and cross-modal coupling. The structure of the formulation also enables efficient computation, making it applicable to long token sequences. Applying SI-GW inside decoder-only Transformers requires careful design. We introduce targeted strategies at the layer, head, and token levels to ensure stable extraction without additional parameters or inference cost. MIRROR improves relational consistency while preserving performance on general vision-language tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 48 references that are in the map. First posted on arXiv 28 Jun 2026.

Builds on

31 of 48 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025VISTA: Enhancing Vision-Text Alignment in MLLMs via Cross-Modal Mutual Information Maximization16 May
  2. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  3. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  4. 2024Probing Multimodal Large Language Models for Global and Local Semantic Representations27 Feb
  5. 2023VILA: On Pre-training for Visual Language Models12 Dec
  6. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  7. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2025Qwen2.5-VL Technical Reportcited by 5,418
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2023Mistral 7Bcited by 3,065
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  12. 2021Grounded Language-Image Pre-trainingcited by 1,147
11 more
  1. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  2. 2019Are Sixteen Heads Really Better than One?cited by 803
  3. 2017Near-linear time approximation algorithms for optimal transport via Sinkhorn iterationcited by 219
  4. 2019On Efficient Optimal Transport: An Analysis of Greedy and Accelerated Mirror Descent Algorithmscited by 60
  5. 2020The Unbalanced Gromov Wasserstein Distance: Conic Formulation and Relaxationcited by 55
  6. 2013Regularized Discrete Optimal Transportcited by 54
  7. 2024Identifying Semantic Induction Heads to Understand In-Context Learningcited by 46
  8. 2018Massively scalable Sinkhorn distances via the Nystr"om methodcited by 40
  9. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-trainingcited by 23
  10. 2025The Intrinsic Dimension of Prompts in Internal Representations of Large Language Modelscited by 17
  11. 2023Semidefinite Relaxations of the Gromov-Wasserstein Distancecited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment
  2. 2024RelationVLM: Making Large Vision-Language Models Understand Visual Relations
  3. 2025BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models
  4. 2025Visual Representation Alignment for Multimodal Large Language Models
  5. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models
  6. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability
  7. 2026Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models
  8. 2026Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning