mlsanityData to 7 Oct 2026

CoLLaVO: Crayon Large Language and Vision mOdel

arXiv 2402.11248 ↗First posted 17 Feb 2024cs.CVcited by 26

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The remarkable success of Large Language Models (LLMs) and instruction tuning drives the evolution of Vision Language Models (VLMs) towards a versatile general-purpose model. Yet, it remains unexplored whether current VLMs genuinely possess quality object-level image understanding capabilities determined from 'what objects are in the image?' or 'which object corresponds to a specified bounding box?'. Our findings reveal that the image understanding capabilities of current VLMs are strongly correlated with their zero-shot performance on vision language (VL) tasks. This suggests that prioritizing basic image understanding is crucial for VLMs to excel at VL tasks. To enhance object-level image understanding, we propose Crayon Large Language and Vision mOdel (CoLLaVO), which incorporates instruction tuning with Crayon Prompt as a new visual prompt tuning scheme based on panoptic color maps. Furthermore, we present a learning strategy of Dual QLoRA to preserve object-level image understanding without forgetting it during visual instruction tuning, thereby achieving a significant leap in numerous VL benchmarks in a zero-shot setting.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 84 references that are in the map. First posted on arXiv 17 Feb 2024.

Builds on

54 of 84 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts1 Dec
  2. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  3. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  4. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  5. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  9. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  11. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  12. 2017Neural Discrete Representation Learningcited by 4,650
31 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  13. 2016A Diagram Is Worth A Dozen Imagescited by 793
  14. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  15. 2023Segment Everything Everywhere All at Oncecited by 546
  16. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  17. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  18. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  19. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  20. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  21. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  22. 2022Scene Text Recognition with Permuted Autoregressive Sequence Modelscited by 148
  23. 2023Explicit Visual Prompting for Low-Level Structure Segmentationscited by 113
  24. 2022Understanding and Improving Visual Prompting: A Label-Mapping Perspectivecited by 86
  25. 2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
  26. 2022Distilling Robust and Non-Robust Features in Adversarial Examples by Information Bottleneckcited by 30
  27. 2022Masking Adversarial Damage: Finding Adversarial Saliency for Robust and Sparse Networkcited by 21
  28. 2023Demystifying Causal Features on Adversarial Examples and Causal Inoculation for Robust Network by Adversarial Instrumental Variable Regressioncited by 18
  29. 2023Causal Unsupervised Semantic Segmentationcited by 18
  30. 2023Mitigating Adversarial Vulnerability through Causal Parameter Estimation by Adversarial Double Machine Learningcited by 11
  31. 2023Integrating Language-Derived Appearance Elements with Visual Cues in Pedestrian Detectioncited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 10; 2026: 3.

26 papers.

Sort:

By area:

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  3. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 52
  4. 2024MoAI: Mixture of All Intelligence for Large Language and Vision Modelscited by 36
  5. 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
  6. 2024Unified Parameter-Efficient Unlearning for LLMscited by 24
  7. 2024EVLM: An Efficient Vision-Language Model for Visual Understandingcited by 23
  8. 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
  9. 2024Phantom of Latent for Large Language and Vision Modelscited by 15
  10. 2024TroL: Traversal of Layers for Large Language and Vision Modelscited by 13
  11. 2025RefineBench: Evaluating Refinement Capability of Language Models via Checklistscited by 13
  12. 2025VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMscited by 12
14 more
  1. 2024VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Modelscited by 10
  2. 2025DIP-R1: Deep Inspection and Perception with RL Looking Through and Understanding Complex Scenescited by 10
  3. 2025MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Modelscited by 10
  4. 2025GenRecal: Generation after Recalibration from Large to Small Vision-Language Modelscited by 9
  5. 2024SPARK: Multi-Vision Sensor Perception and Reasoning Benchmark for Large-scale Vision-Language Modelscited by 8
  6. 2025Unified Reinforcement and Imitation Learning for Vision-Language Modelscited by 8
  7. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  8. 2025A Structured Review of Underwater Object Detection Challenges and Solutions: From Traditional to Large Vision Language Modelscited by 4
  9. 2026Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Taskscited by 4
  10. 2024AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cuescited by 2
  11. 2026Recursive Think-Answer Process for LLMs and VLMscited by 2
  12. 2024Intriguing Properties of Large Language and Vision Modelscited by 1
  13. 20254D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillationcited by 1
  14. 2026Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
  2. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs
  3. 2026Visual Instruction Tuning Aligns Modalities through Abstraction
  4. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning
  5. 2026Boosting Visual Instruction Tuning with Self-Supervised Guidance
  6. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  7. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models
  8. 2025Weaving Context Across Images: Improving Vision-Language Models through Focus-Centric Visual Chains