mlsanityData to 14 Oct 2026

PyVision: Agentic Vision with Dynamic Tooling

arXiv 2507.07998 ↗First posted 10 Jul 2025cs.CL cs.AI cs.CVcited by 79

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

LLMs are increasingly deployed as agents, systems capable of planning, reasoning, and dynamically calling external tools. However, in visual reasoning, prior approaches largely remain limited by predefined workflows and static toolsets. In this report, we present PyVision, an interactive, multi-turn framework that enables MLLMs to autonomously generate, execute, and refine Python-based tools tailored to the task at hand, unlocking flexible and interpretable problem-solving. We develop a taxonomy of the tools created by PyVision and analyze their usage across a diverse set of benchmarks. Quantitatively, PyVision achieves consistent performance gains, boosting GPT-4.1 by +7.8% on V* and Claude-4.0-Sonnet by +31.1% on VLMsAreBlind-mini. These results point to a broader shift: dynamic tooling allows models not just to use tools, but to invent them, advancing toward more agentic visual reasoning.

Key facts. Cited by 79 papers in the map (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 37 of its 50 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

37 of 50 references.

References · 37

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  8. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  9. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  10. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  11. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  12. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
25 more
  1. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  2. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
  3. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  4. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  5. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  6. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 392
  7. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensingcited by 328
  8. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
  9. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 278
  10. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
  11. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  12. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  13. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 184
  14. 2025OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learningcited by 163
  15. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147
  16. 2025OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automationcited by 131
  17. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  18. 2025Alita: Generalist Agent Enabling Scalable Agentic Reasoning with Minimal Predefinition and Maximal Self-Evolutioncited by 116
  19. 2025Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMscited by 96
  20. 2025Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agentscited by 80
  21. 2025OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoningcited by 69
  22. 2024CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoningcited by 67
  23. 2024Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Modelscited by 65
  24. 2025VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledgecited by 55
  25. 2025Thinking with Visual Abstract: Enhancing Multimodal Reasoning via Visual Abstractioncited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 24; 2026: 55.

79 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 303
  2. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  3. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 125
  4. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 119
  5. 2025MemEvolve: Meta-Evolution of Agent Memory Systemscited by 95
  6. 2025Monet: Reasoning in Latent Visual Space Beyond Images and Languagecited by 82
  7. 2026XSkill: Continual Learning from Experience and Skills in Multimodal Agentscited by 75
  8. 2025Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agentscited by 62
  9. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  10. 2025Reinforced Visual Perception with Toolscited by 38
  11. 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Modelscited by 37
  12. 2025TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoningcited by 36
38 more
  1. 2025Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearchcited by 27
  2. 2025Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Toolscited by 26
  3. 2025V-Thinker: Interactive Thinking with Imagescited by 26
  4. 2025CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimizationcited by 26
  5. 2025Sketch-in-Latents: Eliciting Unified Reasoning in MLLMscited by 25
  6. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 24
  7. 2026MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baselinecited by 20
  8. 2026Self-Improvements in Modern Agentic Systems: A Surveycited by 19
  9. 2025MGA: Memory-Driven GUI Agent for Observation-Centric Interactioncited by 18
  10. 2026SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoningcited by 17
  11. 2026PyVision-RL: Forging Open Agentic Vision Models via RLcited by 16
  12. 2026Act Wisely: Cultivating Meta-Cognitive Tool Use in Agentic Multimodal Modelscited by 14
  13. 2026Imagination Helps Visual Reasoning, But Not Yet in Latent Spacecited by 12
  14. 2026VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chainingcited by 12
  15. 2025Geoint-R1: Formalizing Multimodal Geometric Reasoning with Dynamic Auxiliary Constructionscited by 11
  16. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluationcited by 9
  17. 2026What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generationcited by 8
  18. 2026ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoningcited by 8
  19. 2025PhysicsMinions: Winning Gold Medals in the Latest Physics Olympiads with a Coevolutionary Multimodal Multi-Agent Systemcited by 7
  20. 2025Alignment Tipping Process: How Self-Evolution Pushes LLM Agents Off the Railscited by 7
  21. 2026Beyond NL2Code: A Structured Survey of Multimodal Code Intelligencecited by 6
  22. 2025ShowTable: Unlocking Creative Table Visualization with Collaborative Reflection and Refinementcited by 5
  23. 2026ContextRL: Enhancing MLLM's Knowledge Discovery Efficiency with Context-Augmented RLcited by 5
  24. 2026AgenticOCR: Parsing Only What You Need for Efficient Retrieval-Augmented Generationcited by 5
  25. 2025Visual Programmability: A Guide for Code-as-Thought in Chart Understandingcited by 4
  26. 2025Code-in-the-Loop Forensics: Agentic Tool Use for Image Forgery Detectioncited by 4
  27. 2026EVE: Verifiable Self-Evolution of MLLMs via Executable Visual Transformationscited by 4
  28. 2026Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Modelscited by 4
  29. 2026Agent Explorative Policy Optimization for Multimodal Agentic Reasoningcited by 4
  30. 2026Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Docited by 4
  31. 2026AdaptMMBench: Benchmarking Adaptive Multimodal Reasoning for Mode Selection and Reasoning Processcited by 3
  32. 2026GeoBrowse: A Geolocation Benchmark for Agentic Tool Use with Expert-Annotated Reasoning Tracescited by 3
  33. 2026Self-Consistent Latent Reasoning: Long Latent Sequence Reasoning for Vision-Language Modelcited by 3
  34. 2026VideoSeeker: Incentivizing Instance-level Video Understanding via Native Agentic Tool Invocationcited by 3
  35. 2026Sandboxed Coding Agents are Competitive Omni-modal Task Solverscited by 3
  36. 2026Perceive-to-Reason: Decoupling Perception and Reasoning for Fine-Grained Visual Reasoningcited by 3
  37. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  38. 2026Seek-and-Solve: Benchmarking MLLMs for Visual Clue-Driven Reasoning in Daily Scenarioscited by 2

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Visual Agentic AI for Spatial Reasoning with a Dynamic API
  2. 2026Beacon: Knowing When and How to Perform Agentic Visual Reasoning
  3. 2025ToolScope: An Agentic Framework for Vision-Guided and Long-Horizon Tool Use
  4. 2025Thinking with Programming Vision: Towards a Unified View for Thinking with Images
  5. 2026ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both
  6. 2026Socratic-Geo: Synthetic Data Generation and Geometric Reasoning via Multi-Agent Interaction
  7. 2025CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoning
  8. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoning