mlsanityData to 7 Oct 2026

Optimizing Vision-Language Interactions Through Decoder-Only Models

arXiv 2412.10758 ↗First posted 14 Dec 2024cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Vision-Language Models (VLMs) have emerged as key enablers for multimodal tasks, but their reliance on separate visual encoders introduces challenges in efficiency, scalability, and modality alignment. To address these limitations, we propose MUDAIF (Multimodal Unified Decoder with Adaptive Input Fusion), a decoder-only vision-language model that seamlessly integrates visual and textual inputs through a novel Vision-Token Adapter (VTA) and adaptive co-attention mechanism. By eliminating the need for a visual encoder, MUDAIF achieves enhanced efficiency, flexibility, and cross-modal understanding. Trained on a large-scale dataset of 45M image-text pairs, MUDAIF consistently outperforms state-of-the-art methods across multiple benchmarks, including VQA, image captioning, and multimodal reasoning tasks. Extensive analyses and human evaluations demonstrate MUDAIF's robustness, generalization capabilities, and practical usability, establishing it as a new standard in encoder-free vision-language models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 18 references that are in the map. First posted on arXiv 14 Dec 2024.

Builds on

16 of 18 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks12 Jun
  2. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec

Other references · 13

  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  2. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  3. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  4. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  5. 2024Visual In-Context Learning for Large Vision-Language Modelscited by 124
  6. 2024Unveiling Encoder-Free Vision-Language Modelscited by 90
  7. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  8. 2022ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classificationcited by 45
  9. 2024Rethinking Visual Dependency in Long-Context Reasoning for Large Vision-Language Modelscited by 41
  10. 2021EventBERT: A Pre-Trained Model for Event Correlation Reasoningcited by 40
  11. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  12. 2023Multimodal Event Transformer for Image-guided Story Ending Generationcited by 21
1 more
  1. 2023Style-Aware Contrastive Learning for Multi-Style Image Captioningcited by 21

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  2. 2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance
  3. 2026Firebolt-VL: Efficient Vision-Language Understanding with Cross-Modality Modulation
  4. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
  5. 2024Improving Multi-modal Large Language Model through Boosting Vision Capabilities
  6. 2025Rethinking Visual Information Processing in Multimodal LLMs
  7. 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding
  8. 2025MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models