# Anisotropic Modality Align

arXiv: https://arxiv.org/abs/2605.07825 · HTML: https://mlsanity.com/paper/2605.07825/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 20 references that are in the map.
- First posted on arXiv 8 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts](https://mlsanity.com/paper/2407.04973/) (2024)
- [Efficient Multimodal Learning from Data-centric Perspective](https://mlsanity.com/paper/2402.11530/) (2024)
- [The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World](https://mlsanity.com/paper/2308.01907/) (2023)
- [VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models](https://mlsanity.com/paper/2504.15279/) (2025)
- [Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data](https://mlsanity.com/paper/2401.08567/) (2024)
- [LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representation](https://mlsanity.com/paper/2411.04997/) (2024)
- [Text-Only Data Synthesis for Vision Language Model Training](https://mlsanity.com/paper/2503.22655/) (2025)

## Built on by (most cited first, 2 of 2)

- [Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities](https://mlsanity.com/paper/2605.16889/) (2026)
- [Multi-scale Object-Aware Gaze Estimation via Geometric Reasoning](https://mlsanity.com/paper/2606.29334/) (2026)

## Abstract (arXiv)

Training multimodal large language models has long been limited by the scarcity of high-quality paired multimodal data. Recent studies show that the shared representation space of pretrained multimodal contrastive models can serve as a bridge, enabling models to perform multimodal training with unimodal data. However, the key premise of this paradigm remains insufficiently understood: can representations from different modalities be reliably interchanged? The core obstacle lies in the persistent Modality Gap in the shared space. In this work, we revisit the geometric nature of the modality gap. We find that modality representations already share compatible dominant semantic geometry. What truly hinders modality interchangeability is not a simple global shift, but an anisotropic residual structure concentrated along a small number of dominant directions. Based on this finding, we further propose the principle of anisotropic modality gap alignment: effective modality alignment should align with the target-modality distribution while preserving the semantic structure of the source modality. Guided by this principle, we propose an anisotropic geometric correction framework, AnisoAlign, for unpaired modality alignment. This framework leverages the internal geometric prior of the target modality and performs bounded correction on source-modality representations, thereby constructing substitute representations in the target modality. Experiments confirm its benefits in both geometric diagnostics and text-only MLLM training. Overall, this work recasts the modality gap from an empirical observation into a correctable, structured geometric phenomenon and provides a new representation alignment perspective for training multimodal models with unimodal data.