mlsanityData to 14 Oct 2026

Decoupled Vision-Language System for Multimodal Understanding and Generation

arXiv 2608.20382 ↗First posted 29 Jun 2026cs.CL cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We introduce a new architecture design for multimodal large language models (MLLMs), Libra, capable of both multimodal understanding and generation. Libra architecture contains one vision system and one language system, connected by cross-modal bridges. This design decouples self-modal modeling and cross-modal interaction, enabling each modality to learn its unique representations while maintaining effective cross-modal comprehension. The decoupling is mainly achieved in a switch attention module and a switch FFN module, which dynamically routes the computation flow for self-modal modeling and cross-modal interaction scenarios. We evaluate the effectiveness in two important settings: Libra-1 for the understanding-only image-to-text setting, and Libra-2 for unified image-to-text understanding and text-to-image generation. In addition to the architecture design, we discuss various improvements on tokenization, positional encoding, and supervision. Experiments demonstrate that the dedicated Libra design enables mutual improvements on multimodal understanding and generation, achieving strong performance on both understanding and generation benchmarks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 76 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

65 of 76 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  2. 2023Generative Multimodal Models are In-Context Learners20 Dec
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  4. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  5. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  7. 2023JourneyDB: A Benchmark for Generative Image Understanding3 Jul
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  9. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2024The Llama 3 Herd of Modelscited by 14,715
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  10. 2023Segment Anythingcited by 8,234
  11. 2023Visual Instruction Tuningcited by 7,869
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
38 more
  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2022Classifier-Free Diffusion Guidancecited by 5,847
  4. 2017Neural Discrete Representation Learningcited by 4,712
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2013Auto-Encoding Variational Bayescited by 4,053
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  13. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  14. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  15. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  16. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  17. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  18. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  19. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  20. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  21. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  22. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  23. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  24. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  25. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  26. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  27. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  28. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  29. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  30. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  31. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  32. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  33. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  34. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  35. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  36. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  37. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  38. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 227

50 most cited of 56 All 56 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  2. 2026LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model
  3. 2025VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMs
  4. 2025ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability
  5. 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding
  6. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations
  7. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
  8. 2026Toward Native Multimodal Modeling: A Roadmap