mlsanityData to 7 Oct 2026

Honeybee: Locality-enhanced Projector for Multimodal LLM

arXiv 2312.06742 ↗First posted 11 Dec 2023cs.CV cs.AI cs.CLcited by 195

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In Multimodal Large Language Models (MLLMs), a visual projector plays a crucial role in bridging pre-trained vision encoders with LLMs, enabling profound visual understanding while harnessing the LLMs' robust capabilities. Despite the importance of the visual projector, it has been relatively less explored. In this study, we first identify two essential projector properties: (i) flexibility in managing the number of visual tokens, crucial for MLLMs' overall efficiency, and (ii) preservation of local context from visual features, vital for spatial understanding. Based on these findings, we propose a novel projector design that is both flexible and locality-enhanced, effectively satisfying the two desirable properties. Additionally, we present comprehensive strategies to effectively utilize multiple and multifaceted instruction datasets. Through extensive experiments, we examine the impact of individual design choices. Finally, our proposed MLLM, Honeybee, remarkably outperforms previous state-of-the-art methods across various benchmarks, including MME, MMBench, SEED-Bench, and LLaVA-Bench, achieving significantly higher efficiency. Code and models are available at https://github.com/kakaobrain/honeybee.

Key facts. Cited by 195 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 68 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

57 of 68 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs1 Oct
  3. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models18 Sep
  4. 2023TouchStone: Evaluating Vision-Language Models by Language Models31 Aug
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2017Squeeze-and-Excitation Networkscited by 4,763
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2022A ConvNet for the 2020scited by 3,545
35 more
  1. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  13. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  14. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  15. 2016Modeling Context in Referring Expressionscited by 1,054
  16. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  17. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  18. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  19. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  20. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  21. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  22. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  23. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  24. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  25. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  26. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  27. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  28. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  29. 2022Visual Spatial Reasoningcited by 288
  30. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  31. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  32. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  33. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  34. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  35. 2023Large Language Models are Temporal and Causal Reasoners for Video Question Answeringcited by 51

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 72; 2025: 81; 2026: 42.

195 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  3. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  4. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  5. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  6. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  7. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  8. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  9. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  10. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  11. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  12. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
38 more
  1. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  2. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  3. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  4. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  5. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Modelscited by 89
  6. 2024Matryoshka Multimodal Modelscited by 82
  7. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 76
  8. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Modelscited by 74
  9. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  10. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsificationcited by 59
  11. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  12. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  13. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 48
  14. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 48
  15. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  16. 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 44
  17. 2024ParGo: Bridging Vision-Language with Partial and Global Viewscited by 40
  18. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 40
  19. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 40
  20. 2024Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Accelerationcited by 38
  21. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Modelscited by 37
  22. 2025SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Modelscited by 37
  23. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  24. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  25. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
  26. 2024LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillationcited by 32
  27. 2025Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Modelscited by 32
  28. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  29. 2025Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMscited by 30
  30. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 29
  31. 2024Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reductioncited by 28
  32. 2024Safety Alignment for Vision Language Modelscited by 27
  33. 2024Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuningcited by 27
  34. 2024ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Modelscited by 26
  35. 2024MCTBench: Multimodal Cognition towards Text-Rich Visual Scenes Benchmarkcited by 26
  36. 2024Concept-skill Transferability-based Data Selection for Large Vision-Language Modelscited by 24
  37. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  38. 2024An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Modelscited by 22

50 most cited of 195 All 195 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
  2. 2026Projector Is All You Train
  3. 2025Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
  4. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
  5. 2024Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation
  6. 2025Introducing Visual Perception Token into Multimodal Large Language Model
  7. 2025Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models
  8. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Models