mlsanityData to 7 Oct 2026

Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning

arXiv 2507.16746 ↗First posted 22 Jul 2025cs.CV cs.CL cs.LGcited by 78

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Humans often use visual aids, for example diagrams or sketches, when solving complex problems. Training multimodal models to do the same, known as Visual Chain of Thought (Visual CoT), is challenging due to: (1) poor off-the-shelf visual CoT performance, which hinders reinforcement learning, and (2) the lack of high-quality visual CoT training data. We introduce Zebra-CoT, a diverse large-scale dataset with 182,384 samples, containing logically coherent interleaved text-image reasoning traces. We focus on four categories of tasks where sketching or visual reasoning is especially natural, spanning scientific questions such as geometry, physics, and algorithms; 2D visual reasoning tasks like visual search and jigsaw puzzles; 3D reasoning tasks including 3D multi-hop inference, embodied and robot planning; visual logic problems and strategic games like chess. Fine-tuning the Anole-7B model on the Zebra-CoT training corpus results in an improvement of +12% in our test-set accuracy and yields up to +13% performance gain on standard VLM benchmark evaluations. Fine-tuning Bagel-7B yields a model that generates high-quality interleaved visual reasoning chains, underscoring Zebra-CoT's effectiveness for developing multimodal reasoning abilities. We open-source our dataset and models to support development and evaluation of visual CoT.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 60 of its 77 references that are in the map. First posted on arXiv 22 Jul 2025.

Builds on

60 of 77 references.

Closest ancestors · 22

Same direction, newest first

  1. 2025Thinking with Generated Images28 May
  2. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning20 May
  3. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models21 Apr
  4. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization13 Mar
  5. 2025MM-Verify: Enhancing Multimodal Reasoning with Chain-of-Thought Verification19 Feb
  6. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thought13 Jan
  7. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark9 Jan
  8. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale6 Dec
  9. 2024ARC Prize 2024: Technical Report5 Dec
  10. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Step15 Nov
  11. 2024VideoCoT: A Video Chain-of-Thought Dataset with Active Annotation Tool7 Jul
  12. 2024Whiteboard-of-Thought: Thinking Step-by-Step Across Modalities20 Jun
10 more
  1. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models13 Jun
  2. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning25 Mar
  3. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  4. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Models27 Nov
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  6. 2023Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings3 May
  7. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action20 Mar
  8. 2023ViperGPT: Visual Inference via Python Execution for Reasoning14 Mar
  9. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 38

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2024GPT-4o System Cardcited by 4,467
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  9. 2025Gemma 3 Technical Reportcited by 1,686
  10. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  11. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  12. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
26 more
  1. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  2. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  3. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  4. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  5. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  6. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  7. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  8. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  9. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  10. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  11. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  12. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 225
  13. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  14. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  15. 2024Token-level Direct Preference Optimizationcited by 135
  16. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 112
  17. 2024Video as the New Language for Real-World Decision Makingcited by 86
  18. 2025Visual Planning: Let's Think Only with Imagescited by 85
  19. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 58
  20. 2020BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Stepscited by 49
  21. 2024MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokenscited by 47
  22. 2024IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representationscited by 37
  23. 2023A Configurable Library for Generating and Manipulating Maze Datasetscited by 31
  24. 2024MM-PhyQA: Multimodal Physics Question-Answering With Multi-Image CoT Promptingcited by 15
  25. 2024TLDR: Token-Level Detective Reward Model for Large Vision Language Modelscited by 15
  26. 2023VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitionscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 63.

78 papers.

Sort:

By area:

  1. 2025Monet: Reasoning in Latent Visual Space Beyond Images and Languagecited by 75
  2. 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoningcited by 55
  3. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 47
  4. 2025OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipecited by 37
  5. 2025MathCanvas: Intrinsic Visual Chain-of-Thought for Multimodal Mathematical Reasoningcited by 35
  6. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligencecited by 28
  7. 2026MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methodscited by 27
  8. 2025Cognitive Foundations for Reasoning and Their Manifestation in LLMscited by 26
  9. 2026BagelVLA: Enhancing Long-Horizon Manipulation via Interleaved Vision-Language-Action Generationcited by 25
  10. 2025Sketch-in-Latents: Eliciting Unified Reasoning in MLLMscited by 24
  11. 2025CodePlot-CoT: Mathematical Visual Reasoning by Thinking with Code-Driven Imagescited by 22
  12. 2026Visual Generation Unlocks Human-Like Reasoning through Multimodal World Modelscited by 18
38 more
  1. 2026Vision-aligned Latent Reasoning for Multi-modal Large Language Modelcited by 16
  2. 2025Mull-Tokens: Modality-Agnostic Latent Thinkingcited by 15
  3. 2025DeepSketcher: Internalizing Visual Manipulation for Multimodal Reasoningcited by 13
  4. 2025ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generationcited by 13
  5. 2026LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Modelcited by 12
  6. 2026Thinking in Frames: How Visual Context and Test-Time Scaling Empower Video Reasoningcited by 10
  7. 2026Imagination Helps Visual Reasoning, But Not Yet in Latent Spacecited by 10
  8. 2025COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligencecited by 9
  9. 2026GeoEyes: On-Demand Visual Focusing for Evidence-Grounded Understanding of Ultra-High-Resolution Remote Sensing Imagerycited by 9
  10. 2026Omni-R1: Towards the Unified Generative Paradigm for Multimodal Reasoningcited by 7
  11. 2026Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policycited by 6
  12. 2026MentisOculi: Revealing the Limits of Reasoning with Mental Imagerycited by 6
  13. 2026UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Modelscited by 6
  14. 2026HyLaR: Hybrid Latent Reasoning with Decoupled Policy Optimizationcited by 6
  15. 2026UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMscited by 6
  16. 2025VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoningcited by 5
  17. 2026Learning Modal-Mixed Chain-of-Thought Reasoning with Latent Embeddingscited by 5
  18. 2026DeepVision-103K: A Visually Diverse, Broad-Coverage, and Verifiable Mathematical Dataset for Multimodal Reasoningcited by 5
  19. 2026SwimBird: Eliciting Switchable Reasoning Mode in Hybrid Autoregressive MLLMscited by 4
  20. 2026Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Taskscited by 4
  21. 2026Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Processcited by 4
  22. 2026VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoningcited by 4
  23. 2025GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generationcited by 3
  24. 2026TIR-Flow: Active Video Search and Reasoning with Frozen VLMscited by 3
  25. 2026TwiFF (Think With Future Frames): A Large-Scale Dataset for Dynamic Visual Reasoningcited by 3
  26. 2026Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generationcited by 3
  27. 2026DuoGen: Towards General Purpose Interleaved Multimodal Generationcited by 2
  28. 2026Generative Visual Chain-of-Thought for Image Editingcited by 2
  29. 2026Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programscited by 2
  30. 2026STARFlow2: Bridging Language Models and Normalizing Flows for Unified Multimodal Generationcited by 2
  31. 2026ETCHR: Editing To Clarify and Harness Reasoningcited by 2
  32. 2026Scaling Participation in Modular AI Systemscited by 2
  33. 2026ProLaViT: Learning Progressive Latent Visual Thoughts in Structured Latent Spacecited by 2
  34. 2026Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoningcited by 2
  35. 2026Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoningcited by 2
  36. 2025Figure It Out: Improve the Frontier of Reasoning with Executable Visual Statescited by 1
  37. 2026Thinking with Constructions: A Benchmark and Policy Optimization for Visual-Text Interleaved Geometric Reasoningcited by 1
  38. 2026DeepLatent: Think with Images via Parallel Latent Visual Reasoningcited by 1

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Vision
  2. 2025Revisiting the Necessity of Lengthy Chain-of-Thought in Vision-centric Reasoning Generalization
  3. 2026InterSketch: An Interleaved Reasoning Model with Self-correcting Visual Sketch and Stepwise Reward
  4. 2025MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoning
  5. 2025Diagnosing Visual Reasoning: Challenges, Insights, and a Path Forward
  6. 2026TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding
  7. 2025MM-CoT:A Benchmark for Probing Visual Chain-of-Thought Reasoning in Multimodal Models
  8. 2026Render-of-Thought: Rendering Textual Chain-of-Thought as Images for Visual Latent Reasoning