mlsanityData to 7 Oct 2026

Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models

arXiv 2409.17146 ↗First posted 25 Sep 2024cs.CV cs.CL cs.LGcited by 321

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Today's most advanced vision-language models (VLMs) remain proprietary. The strongest open-weight models rely heavily on synthetic data from proprietary VLMs to achieve good performance, effectively distilling these closed VLMs into open ones. As a result, the community has been missing foundational knowledge about how to build performant VLMs from scratch. We present Molmo, a new family of VLMs that are state-of-the-art in their class of openness. Our key contribution is a collection of new datasets called PixMo, including a dataset of highly detailed image captions for pre-training, a free-form image Q&A dataset for fine-tuning, and an innovative 2D pointing dataset, all collected without the use of external VLMs. The success of our approach relies on careful modeling choices, a well-tuned training pipeline, and, most critically, the quality of our newly collected datasets. Our best-in-class 72B model not only outperforms others in the class of open weight and data models, but also outperforms larger proprietary models including Claude 3.5 Sonnet, and Gemini 1.5 Pro and Flash, second only to GPT-4o based on both academic benchmarks and on a large human evaluation. Our model weights, new datasets, and source code are available at https://molmo.allenai.org/blog.

Key facts. Cited by 321 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 113 of its 136 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

113 of 136 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024CogVLM2: Visual Language Models for Image and Video Understanding29 Aug
  2. 2024Building and better understanding vision-language models: insights and future directions22 Aug
  3. 2024VILA2: VILA Augmented VILA24 Jul
  4. 2024PaliGemma: A versatile 3B VLM for transfer10 Jul
  5. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models18 Feb
  6. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model19 Dec
  7. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  8. 2023PaLI-3 Vision Language Models: Smaller, Faster, Stronger13 Oct
  9. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  10. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
1 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 100

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2024The Llama 3 Herd of Modelscited by 14,192
  7. 2023Segment Anythingcited by 8,158
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  11. 2016Layer Normalizationcited by 5,579
  12. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
38 more
  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  13. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  14. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  15. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  16. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  17. 2024Qwen2 Technical Reportcited by 2,076
  18. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  19. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  20. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  21. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  22. 2019Towards VQA Models That Can Readcited by 1,696
  23. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  24. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  25. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  26. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  27. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  28. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  29. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  30. 2016Modeling Context in Referring Expressionscited by 1,054
  31. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  32. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  33. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  34. 2016A Diagram Is Worth A Dozen Imagescited by 793
  35. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  36. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  37. 2024Yi: Open Foundation Models by 01.AIcited by 706
  38. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704

50 most cited of 100 All 100 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 130; 2026: 169.

321 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  3. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  4. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  5. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  6. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
  7. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 204
  8. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  9. 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMscited by 145
  10. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 111
  11. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  12. 2025Visual Spatial Tuningcited by 85
38 more
  1. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  2. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 80
  3. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  4. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  5. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 63
  6. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  7. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 61
  8. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 56
  9. 2026Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Modelscited by 47
  10. 2025Visual Agentic AI for Spatial Reasoning with a Dynamic APIcited by 42
  11. 2025Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulationcited by 41
  12. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Modelscited by 40
  13. 2026Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agentscited by 37
  14. 2025CAPTURe: Evaluating Spatial Reasoning in Vision Language Models via Occluded Object Countingcited by 36
  15. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 36
  16. 2025OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigationcited by 34
  17. 2025Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuningcited by 31
  18. 2025MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMscited by 31
  19. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 30
  20. 2025Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluationcited by 28
  21. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  22. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuningcited by 27
  23. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  24. 2025Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoningcited by 23
  25. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  26. 2024Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answeringcited by 20
  27. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 20
  28. 2024HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Taskscited by 19
  29. 2024LinVT: Empower Your Image-level Large Language Model to Understand Videoscited by 19
  30. 2025SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Datacited by 19
  31. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 19
  32. 2026Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Controlcited by 19
  33. 2025To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Modelscited by 18
  34. 2026HY-Embodied-0.5: Embodied Foundation Models for Real-World Agentscited by 18
  35. 2025Forgotten Polygons: Multimodal Large Language Models are Shape-Blindcited by 16
  36. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  37. 2025NVIDIA Nemotron Nano V2 VLcited by 15
  38. 2025AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Modelscited by 15

50 most cited of 321 All 321 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding
  2. 2024An Introduction to Vision-Language Modeling
  3. 2026DataComp-VLM: Improved Open Datasets for Vision-Language Models
  4. 2023LOVM: Language-Only Vision Model Selection
  5. 2026JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation
  6. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  7. 2026PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models
  8. 2024What matters when building vision-language models?