mlsanityData to 7 Oct 2026

MultiModal Code-Switching: Interleaving Visual Objects into Language for Explicit Object-Level Alignment

arXiv 2608.11167 ↗First posted 11 Aug 2026cs.CV cs.CL cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Existing Multimodal Large Language Models (MLLMs) predominantly rely on image-text pairs for modality alignment pretraining, mapping global image representations to long textual descriptions. However, this image-level alignment suffers from referential ambiguity: models struggle to infer the correspondences between multiple visual objects and textual entities from the global representation, leading to data inefficiency and suboptimal semantic grounding. To address this, we propose MultiModal Code-Switching (MMCS), a novel pretraining paradigm that provides explicit object-level supervision. Inspired by the linguistic phenomenon of code-switching, MMCS interleaves vision and language by replacing textual entities with their corresponding visual objects, enforcing local vision-language grounding. We further develop a scalable data synthesis pipeline to generate a pretraining dataset of 773K samples with accurate object-entity correspondences. Experiments show that MMCS is highly data-efficient: with only 50K samples, it matches or surpasses models trained on 600K image-text pairs. Furthermore, MMCS consistently improves visual grounding and perception capabilities across varying model scales.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 65 references that are in the map. First posted on arXiv 11 Aug 2026.

Builds on

42 of 65 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training28 Sep
  2. 2025Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models22 May
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  4. 2024SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs21 Aug
  5. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception11 Jul
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  7. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  8. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  9. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  12. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
1 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 29

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  11. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
17 more
  1. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  3. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  4. 2016A Diagram Is Worth A Dozen Imagescited by 793
  5. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  6. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  7. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  8. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  9. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  10. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  11. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  12. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
  13. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
  14. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 56
  15. 2025Scalable Vision Language Model Training via High Quality Data Curationcited by 54
  16. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learningcited by 36
  17. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Visual Representation Alignment for Multimodal Large Language Models
  2. 2025BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models
  3. 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding
  4. 2025Attention Guided Alignment in Efficient Vision-Language Models
  5. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
  6. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs
  7. 2026Visual Instruction Tuning Aligns Modalities through Abstraction
  8. 2025Prompt the Unseen: Evaluating Visual-Language Alignment Beyond Supervision