mlsanityData to 14 Oct 2026

VCoder: Versatile Vision Encoders for Multimodal Large Language Models

arXiv 2312.14233 ↗First posted 21 Dec 2023cs.CVcited by 42

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Humans possess the remarkable skill of Visual Perception, the ability to see and understand the seen, helping them make sense of the visual world and, in turn, reason. Multimodal Large Language Models (MLLM) have recently achieved impressive performance on vision-language tasks ranging from visual question-answering and image captioning to visual reasoning and image generation. However, when prompted to identify or count (perceive) the entities in a given image, existing MLLM systems fail. Working towards developing an accurate MLLM system for perception and reasoning, we propose using Versatile vision enCoders (VCoder) as perception eyes for Multimodal LLMs. We feed the VCoder with perception modalities such as segmentation or depth maps, improving the MLLM's perception abilities. Secondly, we leverage the images from COCO and outputs from off-the-shelf vision perception models to create our COCO Segmentation Text (COST) dataset for training and evaluating MLLMs on the object perception task. Thirdly, we introduce metrics to assess the object perception abilities in MLLMs on our COST dataset. Lastly, we provide extensive experimental evidence proving the VCoder's improved object-level perception skills over existing Multimodal LLMs, including GPT-4V. We open-source our dataset, code, and models to promote research. We open-source our code at https://github.com/SHI-Labs/VCoder

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 74 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

56 of 74 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  3. 2023Generating Images with Multimodal Language Models26 May
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  10. 2023Visual Instruction Tuningcited by 7,869
  11. 2020End-to-End Object Detection with Transformerscited by 7,414
  12. 2017Mask R-CNNcited by 7,277
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  3. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  11. 2021Vision Transformers for Dense Predictioncited by 1,803
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  13. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  14. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  15. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,274
  16. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  17. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  18. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  19. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  20. 2018Object Hallucination in Image Captioningcited by 669
  21. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  22. 2019Panoptic Feature Pyramid Networkscited by 645
  23. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  24. 2014Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFscited by 554
  25. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  26. 2016Deeper Depth Prediction with Fully Convolutional Residual Networkscited by 403
  27. 2019Panoptic-DeepLabcited by 363
  28. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 328
  29. 2013DeepPose: Human Pose Estimation via Deep Neural Networkscited by 305
  30. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  31. 2022Neighborhood Attention Transformercited by 239
  32. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 215
  33. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  34. 2018Neural Baby Talkcited by 107
  35. 2023Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenizationcited by 87
  36. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  37. 2022Dilated Neighborhood Attention Transformercited by 64
  38. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 19; 2026: 5.

42 papers.

Sort:

By area:

  1. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  2. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  3. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 152
  4. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  5. 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agentcited by 65
  6. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  7. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMscited by 49
  8. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Modelscited by 47
  9. 2024EVLM: An Efficient Vision-Language Model for Visual Understandingcited by 25
  10. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  11. 2024MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation modelscited by 23
  12. 2024Visually Descriptive Language Model for Vector Graphics Reasoningcited by 14
30 more
  1. 2025Localizing Before Answering: A Hallucination Evaluation Benchmark for Grounded Medical Multimodal LLMscited by 14
  2. 2024Locality Alignment Improves Vision-Language Modelscited by 13
  3. 2024Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Modelscited by 12
  4. 2025Unifying Specialized Visual Encoders for Video Language Modelscited by 10
  5. 2025Mitigating Hallucination in Multimodal LLMs with Layer Contrastive Decodingcited by 10
  6. 2024Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillationcited by 9
  7. 2024RAC3: Retrieval-Augmented Corner Case Comprehension for Autonomous Driving with Vision-Language Modelscited by 8
  8. 2025Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineeringcited by 8
  9. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Modelscited by 8
  10. 2025Fundamentals of Building Autonomous LLM Agentscited by 8
  11. 2024Prompting Medical Large Vision-Language Models to Diagnose Pathologies by Visual Question Answeringcited by 6
  12. 2025SAGE: Training Smart Any-Horizon Agents for Long Video Reasoning with Reinforcement Learningcited by 6
  13. 2025ReFineVLA: Reasoning-Aware Teacher-Guided Transfer Fine-Tuningcited by 5
  14. 2025Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Modelscited by 4
  15. 2025Measuring Epistemic Humility in Multimodal Large Language Modelscited by 4
  16. 2026Grounding the Score: Explicit Visual Premise Verification for Reliable Vision-Language Process Reward Modelscited by 4
  17. 2024CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Modelscited by 3
  18. 2025FaceInsight: A Multimodal Large Language Model for Face Perceptioncited by 3
  19. 2025Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractorcited by 3
  20. 2025Vision-Centric Activation and Coordination for Multimodal Large Language Modelscited by 3
  21. 2025NanoVLMs: How small can we go and still make coherent Vision Language Models?cited by 2
  22. 2025Draft and Refine with Visual Expertscited by 2
  23. 2026ReFineVLA: Multimodal Reasoning-Aware Generalist Robotic Policies via Teacher-Guided Fine-Tuningcited by 1
  24. 2025TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMscited by 0
  25. 2025Towards Open World Detection: A Surveycited by 0
  26. 2025Harnessing Object Grounding for Time-Sensitive Video Understandingcited by 0
  27. 2025Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomycited by 0
  28. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  29. 2026MLLMs Hallucinate when Information Distribution Drifts in Synergy Headscited by 0
  30. 2026DSPO: Diversity-aware Subjective Policy Optimization for Robust Emotional Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  2. 2024Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion
  3. 2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Models
  4. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception
  5. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  6. 2025Introducing Visual Perception Token into Multimodal Large Language Model
  7. 2024Enhancing Advanced Visual Reasoning Ability of Large Language Models
  8. 2026A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models