mlsanityData to 14 Oct 2026

Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding

arXiv 2601.10611 ↗First posted 15 Jan 2026cs.CV cs.AIcited by 139

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Today's strongest video-language models (VLMs) remain proprietary. The strongest open-weight models either rely on synthetic data from proprietary VLMs, effectively distilling from them, or do not disclose their training data or recipe. As a result, the open-source community lacks the foundations needed to improve on the state-of-the-art video (and image) language models. Crucially, many downstream applications require more than just high-level video understanding; they require grounding -- either by pointing or by tracking in pixels. Even proprietary models lack this capability. We present Molmo2, a new family of VLMs that are state-of-the-art among open-source models and demonstrate exceptional new capabilities in point-driven grounding in single image, multi-image, and video tasks. Our key contribution is a collection of 7 new video datasets and 2 multi-image datasets, including a dataset of highly detailed video captions for pre-training, a free-form video Q&A dataset for fine-tuning, a new object tracking dataset with complex queries, and an innovative new video pointing dataset, all collected without the use of closed VLMs. We also present a training recipe for this data utilizing an efficient packing and message-tree encoding scheme, and show bi-directional attention on vision tokens and a novel token-weight strategy improves performance. Our best-in-class 8B model outperforms others in the class of open weight and data models on short videos, counting, and captioning, and is competitive on long-videos. On video-grounding Molmo2 significantly outperforms existing open-weight models like Qwen3-VL (35.5 vs 29.6 accuracy on video counting) and surpasses proprietary models like Gemini 3 Pro on some tasks (38.4 vs 20.0 F1 on video pointing and 56.2 vs 41.1 J&F on video tracking).

Key facts. Cited by 139 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 190 references that are in the map. First posted on arXiv 15 Jan 2026.

Builds on

47 of 190 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Kwai Keye-VL 1.5 Technical Report1 Sep
  2. 2025VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation20 May
  3. 2025PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding17 Apr
  4. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling31 Dec
  5. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding22 Oct

Other references · 42

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2025Qwen3-VL Technical Reportcited by 2,793
  11. 2024Qwen2 Technical Reportcited by 2,120
  12. 2017The Kinetics Human Action Video Datasetcited by 1,909
30 more
  1. 2025Gemma 3 Technical Reportcited by 1,824
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  3. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  5. 2025SAM 3: Segment Anything with Conceptscited by 906
  6. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  7. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  8. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  9. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
  10. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 466
  11. 2020MOT20: A benchmark for multi object tracking in crowded scenescited by 357
  12. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 319
  13. 2025MolmoAct: Action Reasoning Models that can Reason in Spacecited by 197
  14. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 186
  15. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  16. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 125
  17. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 114
  18. 2025Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfercited by 87
  19. 2025Detect Anything via Next Point Predictioncited by 73
  20. 2025Towards Understanding Camera Motions in Any Videocited by 64
  21. 2025MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenescited by 62
  22. 2025VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learningcited by 51
  23. 2025PointArena: Probing Multimodal Grounding Through Language-Guided Pointingcited by 31
  24. 2020RefVOS: A Closer Look at Referring Expressions for Video Object Segmentationcited by 23
  25. 2025VideoMolmo: Spatio-Temporal Grounding Meets Pointingcited by 14
  26. 2025ReasoningTrack: Chain-of-Thought Reasoning for Long-term Vision-Language Trackingcited by 9
  27. 2022SportsTrack: An Innovative Method for Tracking Athletes in Sports Scenescited by 5
  28. 2024ViBe: A Text-to-Video Benchmark for Evaluating Hallucination in Large Multimodal Modelscited by 3
  29. 2025SMIR: Efficient Synthetic Data Pipeline To Improve Multi-Image Reasoningcited by 3
  30. 2025Poivre: Self-Refining Visual Pointing with Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 139.

139 papers.

Sort:

By area:

  1. 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulationcited by 88
  2. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 60
  3. 2026MolmoAct2: Action Reasoning Models for Real-world Deploymentcited by 50
  4. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 39
  5. 2026MolmoPoint: Better Pointing for VLMs with Grounding Tokenscited by 15
  6. 2026EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understandingcited by 13
  7. 2026The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agentscited by 12
  8. 2026TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMscited by 11
  9. 2026VideoVeritas: AI-Generated Video Detection via Perception Pretext Reinforcement Learningcited by 11
  10. 2026Vero: An Open RL Recipe for General Visual Reasoningcited by 11
  11. 2026VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understandingcited by 10
  12. 2026Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Modelcited by 10
38 more
  1. 2026CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioningcited by 8
  2. 2026SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMscited by 7
  3. 2026Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoderscited by 7
  4. 2026VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verificationcited by 7
  5. 2026AdaCodec: A Predictive Visual Code for Video MLLMscited by 7
  6. 2026SVCBench: A Streaming Video Counting Benchmark for Spatial-Temporal State Maintenancecited by 6
  7. 2026Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Modelscited by 6
  8. 2026Small Vision-Language Models are Smart Compressors for Long Video Understandingcited by 5
  9. 2026InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoningcited by 5
  10. 2026MolmoMotion: Forecasting Point Trajectories in 3D with Language Instructioncited by 5
  11. 2026VFIG: Vectorizing Complex Figures in SVG with Vision-Language Modelscited by 4
  12. 2026PerceptionComp: A Video Benchmark for Complex Perception-Centric Reasoningcited by 4
  13. 2026Scaling Video Understanding via Compact Latent Multi-Agent Collaborationcited by 4
  14. 2026Artifact-Bench: Evaluating MLLMs on Detecting and Assessing the Artifacts of AI-Generated Videoscited by 4
  15. 2026On Test-Time Scaling for Vision-Language Modelscited by 4
  16. 2026Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMscited by 4
  17. 2026Can Vision-Language Models Solve the Shell Game?cited by 3
  18. 2026Video-Based Reward Modeling for Computer-Use Agentscited by 3
  19. 2026VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observationscited by 3
  20. 2026To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMscited by 3
  21. 2026MultihopSpatial: Multi-hop Compositional Spatial Reasoning Benchmark for Vision-Language Modelcited by 3
  22. 2026VideoTIR: Accurate Understanding for Long Videos with Efficient Tool-Integrated Reasoningcited by 3
  23. 2026VideoSeeker: Incentivizing Instance-level Video Understanding via Native Agentic Tool Invocationcited by 3
  24. 2026RoboStressBench: Benchmarking VLM Robustness to Physical Visual Stress in Embodied Scenescited by 3
  25. 2026Claim-Level Rubric Rewards for Video Caption Reinforcement Learningcited by 3
  26. 2026Towards Predictive, Aligned, and Scalable Robot Learningcited by 3
  27. 2026TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMscited by 3
  28. 2026Robostral Navigatecited by 3
  29. 2026Are Video Reasoning Models Ready to Go Outside?cited by 2
  30. 2026CL-VISTA: Benchmarking Continual Learning in Video Large Language Modelscited by 2
  31. 2026ViPO: Visual Preference Optimization at Scalecited by 2
  32. 2026ChartREG++: Towards Benchmarking and Improving Chart Referring Expression Grounding under Diverse referring clues and Multi-Target Referringcited by 2
  33. 2026ZAYA1-VL-8B Technical Reportcited by 2
  34. 2026SATURN: Symbolic Spatial Reasoning for Multi-Perspective Groundingcited by 2
  35. 2026DataComp-VLM: Improved Open Datasets for Vision-Language Modelscited by 2
  36. 2026PixelEyes: Decoupling Perception and Reasoning for Pinpoint Visual Evidence Seekingcited by 2
  37. 2026DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulationcited by 2
  38. 2026ICDAR 2026 Competition on Information Extraction from Atomic Layer Deposition/Etching (ALD/E) Scientific Figurescited by 2

50 most cited of 139 All 139 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
  2. 2024CogVLM2: Visual Language Models for Image and Video Understanding
  3. 2024An Introduction to Vision-Language Modeling
  4. 2024Q-GroundCAM: Quantifying Grounding in Vision Language Models via GradCAM
  5. 2025Leveraging Vision-Language Models for Open-Vocabulary Instance Segmentation and Tracking
  6. 2024Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models
  7. 2023ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models
  8. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs