mlsanityData to 7 Oct 2026

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

arXiv 2406.16860 ↗First posted 24 Jun 2024cs.CVcited by 822

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach. While stronger language models can enhance multimodal capabilities, the design choices for vision components are often insufficiently explored and disconnected from visual representation learning research. This gap hinders accurate sensory grounding in real-world scenarios. Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations, offering new insights into different models and architectures -- self-supervised, strongly supervised, or combinations thereof -- based on experiments with over 20 vision encoders. We critically examine existing MLLM benchmarks, address the difficulties involved in consolidating and interpreting results from various tasks, and introduce a new vision-centric benchmark, CV-Bench. To further improve visual grounding, we propose the Spatial Vision Aggregator (SVA), a dynamic and spatially-aware connector that integrates high-resolution vision features with LLMs while reducing the number of tokens. Additionally, we discuss the curation of high-quality visual instruction-tuning data from publicly available sources, emphasizing the importance of data source balancing and distribution ratio. Collectively, Cambrian-1 not only achieves state-of-the-art performance but also serves as a comprehensive, open cookbook for instruction-tuned MLLMs. We provide model weights, code, supporting tools, datasets, and detailed instruction-tuning and evaluation recipes. We hope our release will inspire and accelerate advancements in multimodal systems and visual representation learning.

Key facts. Cited by 822 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 124 of its 142 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

124 of 142 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  2. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  3. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  4. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models8 Feb
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  6. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  7. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  8. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Models19 Sep
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 114

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  11. 2023Segment Anythingcited by 8,158
  12. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
38 more
  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2022A ConvNet for the 2020scited by 3,545
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  13. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  14. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  15. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  16. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  17. 2019Towards VQA Models That Can Readcited by 1,696
  18. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  19. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  20. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  21. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  22. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  23. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  24. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  25. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  26. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  27. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  28. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  29. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  30. 2016Modeling Context in Referring Expressionscited by 1,054
  31. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  32. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  33. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  34. 2016A Diagram Is Worth A Dozen Imagescited by 793
  35. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  36. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  37. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  38. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718

50 most cited of 114 All 114 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 159; 2025: 406; 2026: 257.

822 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  5. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  6. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  7. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  8. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  9. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  10. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  11. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  12. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
38 more
  1. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  2. 2025Kimi-VL Technical Reportcited by 381
  3. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  4. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  5. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  6. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  7. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  8. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  9. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  10. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  11. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  12. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  13. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  14. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  15. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  16. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  17. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 204
  18. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  19. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  20. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172
  21. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  22. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 169
  23. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  24. 2025R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Modelcited by 163
  25. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  26. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  27. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  28. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  29. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  30. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
  31. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  32. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  33. 2025Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Bettercited by 142
  34. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  35. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  36. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  37. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 135
  38. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134

50 most cited of 822 All 822 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion
  2. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs
  3. 2024Instruction Tuning-free Visual Token Complement for Multimodal LLMs
  4. 2025Rethinking the Mixture of Vision Encoders Paradigm for Enhanced Visual Understanding in Multimodal LLMs
  5. 2025How Visual Representations Map to Language Feature Space in Multimodal LLMs
  6. 2023Vision Enhancing LLMs: Empowering Multimodal Knowledge Storage and Sharing in LLMs
  7. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge
  8. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs