mlsanityData to 14 Oct 2026

GRIT: Teaching MLLMs to Think with Images

arXiv 2505.15879 ↗First posted 21 May 2025cs.CV cs.AI cs.CLcited by 125

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Recent studies have demonstrated the efficacy of using Reinforcement Learning (RL) in building reasoning models that articulate chains of thoughts prior to producing final answers. However, despite ongoing advances that aim at enabling reasoning for vision-language tasks, existing open-source visual reasoning models typically generate reasoning content with pure natural language, lacking explicit integration of visual information. This limits their ability to produce clearly articulated and visually grounded reasoning chains. To this end, we propose Grounded Reasoning with Images and Texts (GRIT), a novel method for training MLLMs to think with images. GRIT introduces a grounded reasoning paradigm, in which models generate reasoning chains that interleave natural language and explicit bounding box coordinates. These coordinates point to regions of the input image that the model consults during its reasoning process. Additionally, GRIT is equipped with a reinforcement learning approach, GRPO-GR, built upon the GRPO algorithm. GRPO-GR employs robust rewards focused on the final answer accuracy and format of the grounded reasoning output, which eliminates the need for data with reasoning chain annotations or explicit bounding box labels. As a result, GRIT achieves exceptional data efficiency, requiring as few as 20 image-question-answer triplets from existing datasets. Comprehensive evaluations demonstrate that GRIT effectively trains MLLMs to produce coherent and visually grounded reasoning chains, showing a successful unification of reasoning and grounding abilities.

Key facts. Cited by 125 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 35 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

25 of 35 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimization25 Apr
  2. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models10 Apr
  3. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model10 Apr
  4. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization13 Mar
  5. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models9 Mar
  6. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning25 Mar
  7. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Models27 Nov
  8. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  9. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb

Other references · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  9. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  10. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  11. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  12. 2025Reasoning Models Don't Always Say What They Thinkcited by 415
4 more
  1. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  2. 2022Visual Spatial Reasoningcited by 293
  3. 2018TallyQA: Answering Complex Counting Questionscited by 153
  4. 2023How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detectioncited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 49; 2026: 76.

125 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  4. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 125
  5. 2025Perception-Aware Policy Optimization for Multimodal Reasoningcited by 99
  6. 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Modelscited by 90
  7. 2025Monet: Reasoning in Latent Visual Space Beyond Images and Languagecited by 82
  8. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 69
  9. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 65
  10. 2025LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Callingcited by 61
  11. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 55
  12. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 49
38 more
  1. 2025ACTIVE-o3: Empowering MLLMs with Active Perception via Pure Reinforcement Learningcited by 43
  2. 2025Spotlight on Token Perception for Multimodal Reinforcement Learningcited by 39
  3. 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Modelscited by 37
  4. 2025MedEyes: Learning Dynamic Visual Focus for Medical Progressive Diagnosiscited by 32
  5. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 31
  6. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoningcited by 29
  7. 2025Reasoning Within the Mind: Dynamic Multimodal Interleaving in Latent Spacecited by 28
  8. 2025LOVE-R1: Advancing Long Video Understanding with an Adaptive Zoom-in Mechanism via Multi-Step Reasoningcited by 27
  9. 2025SIFThinker: Spatially-Aware Image Focus for Visual Reasoningcited by 25
  10. 2025Simple o3: Towards Interleaved Vision-Language Reasoningcited by 24
  11. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 24
  12. 2026Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddingscited by 24
  13. 2025Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMscited by 17
  14. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 16
  15. 2025Thinking with Programming Vision: Towards a Unified View for Thinking with Imagescited by 15
  16. 2026Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Modelscited by 14
  17. 2025AVATAR: Reinforcement Learning to See, Hear, and Reason Over Videocited by 12
  18. 2026OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attentioncited by 11
  19. 2026V-Zero: Answer-Label-Free On-Policy Distillation with Contrastive Evidence Gating for Fine-Grained Visual Reasoningcited by 11
  20. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 10
  21. 2025DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoningcited by 10
  22. 2025SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Modelscited by 10
  23. 2025Reinforcement Learning for Large Model: A Surveycited by 9
  24. 2025Perception-Consistency Multimodal Large Language Models Reasoning via Caption-Regularized Policy Optimizationcited by 9
  25. 2026Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thoughtcited by 9
  26. 2026Seeing with You: Perception-Reasoning Coevolution for Multimodal Reasoningcited by 9
  27. 2025AD-FM: Multimodal LLMs for Anomaly Detection via Multi-Stage Reasoning and Fine-Grained Reward Optimizationcited by 8
  28. 2025Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculationcited by 8
  29. 2025Revisiting the Necessity of Lengthy Chain-of-Thought in Vision-centric Reasoning Generalizationcited by 8
  30. 2025Evidence-Grounded Trustworthy Multimodal Reasoning and Evaluation Benchmark in Complex Urban Scenescited by 7
  31. 2025Mixture-of-Visual-Thoughts: Exploring Context-Adaptive Reasoning Mode Selection for General Visual Reasoningcited by 7
  32. 2026MentisOculi: Revealing the Limits of Reasoning with Mental Imagerycited by 7
  33. 2026VideoBrain: Learning Adaptive Frame Sampling for Long Video Understandingcited by 7
  34. 2026What, Whether and How? Unveiling Process Reward Models for Thinking with Images Reasoningcited by 7
  35. 2026TerraScope: Pixel-Grounded Visual Reasoning for Earth Observationcited by 7
  36. 2026Scientific Graphics Program Synthesis via Dual Self-Consistency Reinforcement Learningcited by 6
  37. 2025DeepSport: A Multimodal Large Language Model for Comprehensive Sports Video Reasoning via Agentic Reinforcement Learningcited by 5
  38. 2025Know-Show: Benchmarking Video-Language Models on Spatio-Temporal Grounded Reasoningcited by 5

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learning
  2. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use
  3. 2024Plug-and-Play Grounding of Reasoning in Multimodal Large Language Models
  4. 2026Think with Structured Grounding: Perceptual Reinforcement Learning for Chart and Visual-Tabular Understanding
  5. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning
  6. 2025MiCo: Multi-image Contrast for Reinforcement Visual Reasoning
  7. 2025Grounded Reinforcement Learning for Visual Reasoning
  8. 2026See Further, Think Deeper: Advancing VLM's Reasoning Ability with Low-level Visual Cues and Reflection