mlsanityData to 14 Oct 2026

Connector-S: A Survey of Connectors in Multi-modal Large Language Models

arXiv 2502.11453 ↗First posted 17 Feb 2025cs.LG cs.AIcited by 7

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

With the rapid advancements in multi-modal large language models (MLLMs), connectors play a pivotal role in bridging diverse modalities and enhancing model performance. However, the design and evolution of connectors have not been comprehensively analyzed, leaving gaps in understanding how these components function and hindering the development of more powerful connectors. In this survey, we systematically review the current progress of connectors in MLLMs and present a structured taxonomy that categorizes connectors into atomic operations (mapping, compression, mixture of experts) and holistic designs (multi-layer, multi-encoder, multi-modal scenarios), highlighting their technical contributions and advancements. Furthermore, we discuss several promising research frontiers and challenges, including high-resolution input, dynamic compression, guide information selection, combination strategy, and interpretability. This survey is intended to serve as a foundational reference and a clear roadmap for researchers, providing valuable insights into the design and optimization of next-generation connectors to enhance the performance and adaptability of MLLMs.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 55 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

48 of 55 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understanding15 Oct
  2. 2024To Preserve or To Compress: An In-Depth Study of Connector Selection in Multimodal Large Language Models9 Oct
  3. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders28 Aug
  4. 2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models17 Jul
  5. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  6. 2024MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Model17 Jun
  7. 2024Dense Connector for MLLMs22 May
  8. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts9 May
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  10. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  11. 2024MM-LLMs: Recent Advances in MultiModal Large Language Models24 Jan
  12. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
7 more
  1. 2023Honeybee: Locality-enhanced Projector for Multimodal LLM11 Dec
  2. 2023OneLLM: One Framework to Align All Modalities with Language6 Dec
  3. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge20 Nov
  4. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models13 Oct
  5. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  6. 2023A Survey on Multimodal Large Language Models23 Jun
  7. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration15 Jun

Other references · 29

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2024Yi: Open Foundation Models by 01.AIcited by 719
  6. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  7. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  8. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  9. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  10. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  11. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 172
  12. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
17 more
  1. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  2. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Modelscited by 75
  3. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 69
  4. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  5. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  6. 2024ParGo: Bridging Vision-Language with Partial and Global Viewscited by 40
  7. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  8. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understandingcited by 32
  9. 2023Voila-A: Aligning Vision-Language Models with User's Gaze Attentioncited by 28
  10. 2024HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Modelscited by 28
  11. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Timecited by 26
  12. 2024FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compressioncited by 22
  13. 2024PPLLaVA: Varied Video Sequence Understanding With Prompt Guidancecited by 20
  14. 2024Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoEcited by 19
  15. 2024PlanLLM: Video Procedure Planning with Refinable Large Language Modelscited by 10
  16. 2024World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Drivingcited by 9
  17. 2024LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasoundcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 1.

7 papers.

  1. 2025Lost in Embeddings: Information Loss in Vision-Language Modelscited by 20
  2. 2025LangBridge: Interpreting Image as a Combination of Language Embeddingscited by 10
  3. 2025From Turn-Taking to Synchronous Dialogue: A Survey of Full-Duplex Spoken Language Modelscited by 10
  4. 2025Widget2Code: From Visual Widgets to UI Code via Multimodal LLMscited by 9
  5. 2025ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoningcited by 6
  6. 2026Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMscited by 2
  7. 2025(Almost) Free Modality Stitching of Foundation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performance
  2. 2026Break Through the Compression Bottleneck: From Theory to Practice
  3. 2023A Survey on Model Compression for Large Language Models
  4. 2024Model Composition for Multimodal Large Language Models
  5. 2025A Survey of Token Compression for Efficient Multimodal Large Language Models
  6. 2025Self-Improvement in Multimodal Large Language Models: A Survey
  7. 2023Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper
  8. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective