# Connector-S: A Survey of Connectors in Multi-modal Large Language Models

arXiv: https://arxiv.org/abs/2502.11453 · HTML: https://mlsanity.com/paper/2502.11453/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 48 of its 55 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 30 of 48)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs](https://mlsanity.com/paper/2312.14135/) (2023)
- [PandaGPT: One Model To Instruction-Follow Them All](https://mlsanity.com/paper/2305.16355/) (2023)
- [PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning](https://mlsanity.com/paper/2404.16994/) (2024)
- [mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models](https://mlsanity.com/paper/2408.04840/) (2024)
- [MM-LLMs: Recent Advances in MultiModal Large Language Models](https://mlsanity.com/paper/2401.13601/) (2024)
- [MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training](https://mlsanity.com/paper/2403.09611/) (2024)
- [Honeybee: Locality-enhanced Projector for Multimodal LLM](https://mlsanity.com/paper/2312.06742/) (2023)
- [OneLLM: One Framework to Align All Modalities with Language](https://mlsanity.com/paper/2312.03700/) (2023)
- [Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration](https://mlsanity.com/paper/2306.09093/) (2023)
- [TokenPacker: Efficient Visual Projector for Multimodal LLM](https://mlsanity.com/paper/2407.02392/) (2024)
- [Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders](https://mlsanity.com/paper/2408.15998/) (2024)
- [GroundingGPT:Language Enhanced Multi-modal Grounding Model](https://mlsanity.com/paper/2401.06071/) (2024)
- [From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models](https://mlsanity.com/paper/2310.08825/) (2023)
- [AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model](https://mlsanity.com/paper/2309.16058/) (2023)
- [LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge](https://mlsanity.com/paper/2311.11860/) (2023)
- [DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models](https://mlsanity.com/paper/2405.20985/) (2024)
- [BRAVE: Broadening the visual encoding of vision-language models](https://mlsanity.com/paper/2404.07204/) (2024)
- [CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts](https://mlsanity.com/paper/2405.05949/) (2024)
- [CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2403.04640/) (2024)

## Built on by (most cited first, 7 of 7)

- [Lost in Embeddings: Information Loss in Vision-Language Models](https://mlsanity.com/paper/2509.11986/) (2025)
- [LangBridge: Interpreting Image as a Combination of Language Embeddings](https://mlsanity.com/paper/2503.19404/) (2025)
- [From Turn-Taking to Synchronous Dialogue: A Survey of Full-Duplex Spoken Language Models](https://mlsanity.com/paper/2509.14515/) (2025)
- [Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs](https://mlsanity.com/paper/2512.19918/) (2025)
- [ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoning](https://mlsanity.com/paper/2512.00305/) (2025)
- [Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs](https://mlsanity.com/paper/2605.13737/) (2026)
- [(Almost) Free Modality Stitching of Foundation Models](https://mlsanity.com/paper/2507.10015/) (2025)

## Abstract (arXiv)

With the rapid advancements in multi-modal large language models (MLLMs), connectors play a pivotal role in bridging diverse modalities and enhancing model performance. However, the design and evolution of connectors have not been comprehensively analyzed, leaving gaps in understanding how these components function and hindering the development of more powerful connectors. In this survey, we systematically review the current progress of connectors in MLLMs and present a structured taxonomy that categorizes connectors into atomic operations (mapping, compression, mixture of experts) and holistic designs (multi-layer, multi-encoder, multi-modal scenarios), highlighting their technical contributions and advancements. Furthermore, we discuss several promising research frontiers and challenges, including high-resolution input, dynamic compression, guide information selection, combination strategy, and interpretability. This survey is intended to serve as a foundational reference and a clear roadmap for researchers, providing valuable insights into the design and optimization of next-generation connectors to enhance the performance and adaptability of MLLMs.