mlsanityData to 7 Oct 2026

Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models

arXiv 2406.09403 ↗First posted 13 Jun 2024cs.CV cs.CLcited by 260

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Humans draw to facilitate reasoning: we draw auxiliary lines when solving geometry problems; we mark and circle when reasoning on maps; we use sketches to amplify our ideas and relieve our limited-capacity working memory. However, such actions are missing in current multimodal language models (LMs). Current chain-of-thought and tool-use paradigms only use text as intermediate reasoning steps. In this work, we introduce Sketchpad, a framework that gives multimodal LMs a visual sketchpad and tools to draw on the sketchpad. The LM conducts planning and reasoning according to the visual artifacts it has drawn. Different from prior work, which uses text-to-image models to enable LMs to draw, Sketchpad enables LMs to draw with lines, boxes, marks, etc., which is closer to human sketching and better facilitates reasoning. Sketchpad can also use specialist vision models during the sketching process (e.g., draw bounding boxes with object detection models, draw masks with segmentation models), to further enhance visual perception and reasoning. We experiment with a wide range of math tasks (including geometry, functions, graphs, and chess) and complex visual reasoning tasks. Sketchpad substantially improves performance on all tasks over strong base models with no sketching, yielding an average gain of 12.7% on math tasks, and 8.6% on vision tasks. GPT-4o with Sketchpad sets a new state of the art on all tasks, including V*Bench (80.3%), BLINK spatial reasoning (83.9%), and visual correspondence (80.8%). All codes and data are in https://visualsketchpad.github.io/.

Key facts. Cited by 260 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 55 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

39 of 55 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models5 Dec
  2. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action20 Mar
  3. 2023ViperGPT: Visual Inference via Python Execution for Reasoning14 Mar
  4. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb
  5. 2022Visual Programming: Compositional visual reasoning without training18 Nov
  6. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language1 Apr

Other references · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Segment Anythingcited by 8,158
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  11. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  12. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
21 more
  1. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  2. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  3. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  4. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  5. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  6. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  7. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  8. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  9. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  10. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  11. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  12. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  13. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  14. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  15. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  16. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  17. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 66
  18. 2023EcoAssistant: Using LLM Assistant More Affordably and Accuratelycited by 47
  19. 2024m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Taskscited by 38
  20. 20233DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4Vcited by 15
  21. 2023Analyzing Modular Approaches for Visual Question Decompositioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 123; 2026: 118.

260 papers.

Sort:

By area:

  1. 2025s1: Simple test-time scalingcited by 1,311
  2. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  3. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
  4. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thoughtcited by 204
  5. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  6. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  7. 2025R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Modelcited by 163
  8. 2025OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learningcited by 163
  9. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  10. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
  11. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 130
  12. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
38 more
  1. 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokenscited by 120
  2. 2024Improving Multi-Agent Debate with Sparse Communication Topologycited by 103
  3. 2024ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Explorationcited by 99
  4. 2025Latent Visual Reasoningcited by 99
  5. 2025Gold-medalist Performance in Solving Olympiad Geometry with AlphaGeometry2cited by 95
  6. 2025Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMscited by 94
  7. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  8. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 88
  9. 2025Visual Planning: Let's Think Only with Imagescited by 85
  10. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 84
  11. 2025ReFocus: Visual Editing as a Chain of Thought for Structured Image Understandingcited by 81
  12. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 80
  13. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 78
  14. 2025Thinking with Generated Imagescited by 71
  15. 2025PyVision: Agentic Vision with Dynamic Toolingcited by 71
  16. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 71
  17. 2025OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoningcited by 69
  18. 2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Modelscited by 68
  19. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 64
  20. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 55
  21. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  22. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 54
  23. 2025SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agentcited by 53
  24. 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Visioncited by 52
  25. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 47
  26. 2025Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learningcited by 46
  27. 2025MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoningcited by 45
  28. 2025MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoningcited by 45
  29. 2025Advancing Multimodal Reasoning via Reinforcement Learning with Cold Startcited by 41
  30. 2024CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solverscited by 40
  31. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  32. 2025Reinforced Visual Perception with Toolscited by 38
  33. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
  34. 2025MathCanvas: Intrinsic Visual Chain-of-Thought for Multimodal Mathematical Reasoningcited by 35
  35. 2025Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thoughtcited by 34
  36. 2025VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selectioncited by 34
  37. 2025TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoningcited by 34
  38. 2024Theorem-Validated Reverse Chain-of-Thought Problem Generation for Geometric Reasoningcited by 32

50 most cited of 260 All 260 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMs
  2. 2026SketchJudge: A Diagnostic Benchmark for Grading Hand-drawn Diagrams with Multimodal Large Language Models
  3. 2025Sketch-in-Latents: Eliciting Unified Reasoning in MLLMs
  4. 2025Forgotten Polygons: Multimodal Large Language Models are Shape-Blind
  5. 2026SketchThinker-R1: Towards Efficient Sketch-Style Reasoning in Large Multimodal Models
  6. 2024Whiteboard-of-Thought: Thinking Step-by-Step Across Modalities
  7. 2026PainterBench: A Figural Divergent-Thinking Benchmark for Tool-Using Language Models
  8. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning