mlsanityData to 7 Oct 2026

Mind the Gap No More: Achieving Zero-Gap Multimodal Integration via One Tokenizer

arXiv 2602.12286 ↗First posted 21 Jan 2026q-bio.GN cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

A central challenge in developing Multimodal Large Language Models (MLLMs) is effectively integrating heterogeneous inputs into a cohesive reasoning engine. Current paradigms predominantly rely on modular architectures that introduce modality-specific encoders and cross-modal fusion mechanisms. However, these designs are fundamentally bottlenecked by a geometric modality gap, forcing the LLM to expend significant computational capacity on geometric reconciliation rather than deep cross-modal reasoning. In this work, we formally characterize this modality gap and theoretically demonstrate that native architectures, specifically those employing a unified vocabulary, intrinsically maintain a zero-gap state across all hidden layers. Guided by these theoretical findings, we propose textitOne Tokenizer, a native architecture that maps all modalities directly into a shared token space. We empirically validate this framework on a DNA--text multimodal testbed. Our extensive evaluations reveal that by achieving seamless integration within the LLM's native latent space, One Tokenizer consistently outperforms encoder-based modular counterparts, providing a fundamentally superior framework for deep biological reasoning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 36 references that are in the map. First posted on arXiv 21 Jan 2026.

Builds on

28 of 36 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey3 Dec
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  3. 2024A Concept-Based Explainability Framework for Large Multimodal Models12 Jun
  4. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark7 Feb
  5. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  6. 2023A Survey on Multimodal Large Language Models23 Jun
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2016Layer Normalizationcited by 5,579
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  7. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  8. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  9. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  10. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  11. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  12. 2025Kimi-VL Technical Reportcited by 381
9 more
  1. 2018Invertible Residual Networkscited by 364
  2. 2020The Lipschitz Constant of Self-Attentioncited by 171
  3. 2022MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generationcited by 125
  4. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 108
  5. 2024Unveiling Encoder-Free Vision-Language Modelscited by 90
  6. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  7. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  8. 2025BioReason: Incentivizing Multimodal Biological Reasoning within a DNA-LLM Modelcited by 35
  9. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 22

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs
  2. 2025Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
  3. 2025Rethinking the Text-Vision Reasoning Imbalance in MLLMs through the Lens of Training Recipes
  4. 2026Scaling-Aware Adapter for Structure-Grounded LLM Reasoning
  5. 2025Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning
  6. 2026Toward Native Multimodal Modeling: A Roadmap
  7. 2025OmniBridge: Unified Multimodal Understanding, Generation, and Retrieval via Latent Space Alignment
  8. 2026Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models