mlsanityData to 7 Oct 2026

TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning

arXiv 2511.01833 ↗First posted 3 Nov 2025cs.CVcited by 34

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

The frontier of visual reasoning is shifting toward models like OpenAI o3, which can intelligently create and operate tools to transform images for problem-solving, also known as thinking-textitwith-images in chain-of-thought. Yet existing benchmarks fail to fully capture this advanced capability. Even Visual Search, the most common benchmark for current thinking-textitwith-images methods, tests only basic operations such as localization and cropping, offering little insight into more complex, dynamic, and tool-dependent reasoning. We introduce TIR-Bench, a comprehensive benchmark for evaluating agentic thinking-with-images across 13 diverse tasks, each requiring novel tool use for image processing and manipulation in chain-of-thought. We evaluate 22 multimodal large language models (MLLMs), from leading open-sourced and proprietary models to those with explicit tool-use augmentation. Results show that TIR-Bench is universally challenging, and strong performance requires genuine thinking-with-images capabilities. Finally, we present a pilot study comparing direct versus agentic fine-tuning.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 33 references that are in the map. First posted on arXiv 3 Nov 2025.

Builds on

31 of 33 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Reinforced Visual Perception with Tools1 Sep
  2. 2025Thyme: Think Beyond Images15 Aug
  3. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers30 Jun
  4. 2025Play to Generalize: Learning to Reason Through Game Play9 Jun
  5. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning21 May
  6. 2025Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMs21 May
  7. 2025ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustness10 Apr
  8. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?1 Jul
  9. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models13 Jun
  10. 2024CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning6 Feb
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
1 more
  1. 2023ViperGPT: Visual Inference via Python Execution for Reasoning14 Mar

Other references · 18

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  9. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  10. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  11. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  12. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
6 more
  1. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  2. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  3. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 151
  4. 2025Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Searchcited by 116
  5. 2025PyVision: Agentic Vision with Dynamic Toolingcited by 71
  6. 2024EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmarkcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 31.

34 papers.

Sort:

By area:

  1. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
  2. 2026XSkill: Continual Learning from Experience and Skills in Multimodal Agentscited by 71
  3. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 27
  4. 2026AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarioscited by 22
  5. 2026PyVision-RL: Forging Open Agentic Vision Models via RLcited by 14
  6. 2026VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chainingcited by 12
  7. 2025MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluationcited by 11
  8. 2025Octopus: Agentic Multimodal Reasoning with Six-Capability Orchestrationcited by 8
  9. 2026Agent Skill Evaluation and Evolution: Frameworks and Benchmarkscited by 6
  10. 2026Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?cited by 4
  11. 2026Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gainscited by 4
  12. 2026AdaptMMBench: Benchmarking Adaptive Multimodal Reasoning for Mode Selection and Reasoning Processcited by 3
22 more
  1. 2026GeoBrowse: A Geolocation Benchmark for Agentic Tool Use with Expert-Annotated Reasoning Tracescited by 3
  2. 2026LEDGERMIND: Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledgercited by 3
  3. 2026SenseNova-U1.5: Towards Native Unified Visual Intelligencecited by 3
  4. 2026Ace-Skill: Bootstrapping Multimodal Agents with Prioritized and Clustered Evolutioncited by 2
  5. 2026MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agentscited by 2
  6. 2026SPyCE: Skill-Policy Co-evolution for Multimodal Agentscited by 2
  7. 2026Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoningcited by 2
  8. 2026Project Ariadne: A Structural Causal Framework for Auditing Faithfulness in LLM Agentscited by 1
  9. 2026Seek-and-Solve: Benchmarking MLLMs for Visual Clue-Driven Reasoning in Daily Scenarioscited by 1
  10. 2026VistaHop: Benchmarking Long-Horizon Visual DeepSearchcited by 1
  11. 2026Beacon: Knowing When and How to Perform Agentic Visual Reasoningcited by 1
  12. 2026Self-Evolving Code-with-Image Reasoningcited by 1
  13. 2026V-ABS: Action-Observer Driven Beam Search for Dynamic Visual Reasoningcited by 0
  14. 2026VisualNeedle: Benchmarking Active Visual Search in Information-Dense Scenescited by 0
  15. 2026InterSketch: An Interleaved Reasoning Model with Self-correcting Visual Sketch and Stepwise Rewardcited by 0
  16. 2026MUSE: A Unified Agentic Harness for MLLMscited by 0
  17. 2026VC-Tooler: Learning Compositional and Adaptive Visual Tool Usecited by 0
  18. 2026From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Modelscited by 0
  19. 2026VLM-in-Sandbox: Visual Workspaces for Agentic Visual Reasoningcited by 0
  20. 2026SkillRubric: Co-Evolving Actor Guidance and Evaluator Rubrics for Multimodal Agentscited by 0
  21. 2026V-Gym: Enhancing Agentic Visual Reasoning via Skill-Data Co-Evolutioncited by 0
  22. 2026Visual Parallel Search: Learning to Search High-Resolution Images with Parallel Tile Inspection and Adaptive Zoomcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Visual Agentic Reinforcement Fine-Tuning
  2. 2025When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought
  3. 2025Thinking with Programming Vision: Towards a Unified View for Thinking with Images
  4. 2026Beyond Accuracy: Evaluating Grounded Visual Evidence in Thinking with Images
  5. 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning
  6. 2025Reasoning Within the Mind: Dynamic Multimodal Interleaving in Latent Space
  7. 2026ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both
  8. 2025Diagnosing Visual Reasoning: Challenges, Insights, and a Path Forward