Mind the Gap No More: Achieving Zero-Gap Multimodal Integration via One Tokenizer
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
A central challenge in developing Multimodal Large Language Models (MLLMs) is effectively integrating heterogeneous inputs into a cohesive reasoning engine. Current paradigms predominantly rely on modular architectures that introduce modality-specific encoders and cross-modal fusion mechanisms. However, these designs are fundamentally bottlenecked by a geometric modality gap, forcing the LLM to expend significant computational capacity on geometric reconciliation rather than deep cross-modal reasoning. In this work, we formally characterize this modality gap and theoretically demonstrate that native architectures, specifically those employing a unified vocabulary, intrinsically maintain a zero-gap state across all hidden layers. Guided by these theoretical findings, we propose textitOne Tokenizer, a native architecture that maps all modalities directly into a shared token space. We empirically validate this framework on a DNA--text multimodal testbed. Our extensive evaluations reveal that by achieving seamless integration within the LLM's native latent space, One Tokenizer consistently outperforms encoder-based modular counterparts, providing a fundamentally superior framework for deep biological reasoning.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 36 references that are in the map. First posted on arXiv 21 Jan 2026.
Builds on
28 of 36 references.
Closest ancestors · 7
Same direction, newest first
- 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey3 Dec
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
- 2024A Concept-Based Explainability Framework for Large Multimodal Models12 Jun
- 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark7 Feb
- 2023Improved Baselines with Visual Instruction Tuning5 Oct
- 2023A Survey on Multimodal Large Language Models23 Jun
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
Other references · 21
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2025Qwen3 Technical Reportcited by 8,195
- 2023Visual Instruction Tuningcited by 7,792
- 2016Layer Normalizationcited by 5,579
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
- 2024Emu3: Next-Token Prediction is All You Needcited by 761
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
- 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
- 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
- 2025Kimi-VL Technical Reportcited by 381
9 more
- 2018Invertible Residual Networkscited by 364
- 2020The Lipschitz Constant of Self-Attentioncited by 171
- 2022MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generationcited by 125
- 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 108
- 2024Unveiling Encoder-Free Vision-Language Modelscited by 90
- 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
- 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
- 2025BioReason: Incentivizing Multimodal Biological Reasoning within a DNA-LLM Modelcited by 35
- 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 22
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs
- 2025Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
- 2025Rethinking the Text-Vision Reasoning Imbalance in MLLMs through the Lens of Training Recipes
- 2026Scaling-Aware Adapter for Structure-Grounded LLM Reasoning
- 2025Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning
- 2026Toward Native Multimodal Modeling: A Roadmap
- 2025OmniBridge: Unified Multimodal Understanding, Generation, and Retrieval via Latent Space Alignment
- 2026Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models