mlsanityData to 14 Oct 2026

LATTE: Learning to Think with Vision Specialists

arXiv 2412.05479 ↗First posted 7 Dec 2024cs.CVcited by 14

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

While open-source vision-language models perform well on simple question-answering, they still struggle with complex questions that require both perceptual and reasoning capabilities. We propose LATTE, a family of vision-language models that have LeArned to Think wiTh vision spEcialists. By offloading perception to state-of-the-art vision models, our approach enables vision-language models to focus solely on reasoning over high-quality perceptual information. To train LATTE, we synthesize and filter a large dataset of 293K multi-modal reasoning traces over perceptual outputs of vision specialists. LATTE trained on this data achieves significant 4-5% gains over baselines across 6 benchmarks covering both perception and reasoning abilities. Ablation studies reveal that the effectiveness of multi-modal reasoning traces depends on the data sources, formats, and quality of thoughts.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 48 references that are in the map. First posted on arXiv 7 Dec 2024.

Builds on

25 of 48 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Step15 Nov
  2. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models13 Jun
  3. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought26 May
  4. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning25 Mar
  5. 2024CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning6 Feb

Other references · 20

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  5. 2024Depth Anything V2cited by 1,422
  6. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  7. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  9. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  10. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  11. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  12. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
8 more
  1. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  2. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  3. 2024What matters when building vision-language models?cited by 326
  4. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  5. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  6. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 193
  7. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 97
  8. 2024Task Me Anythingcited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 5.

14 papers.

Sort:

By area:

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  2. 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoningcited by 68
  3. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 55
  4. 2025BioReason: Incentivizing Multimodal Biological Reasoning within a DNA-LLM Modelcited by 37
  5. 2025Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuningcited by 32
  6. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoningcited by 29
  7. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  8. 2025COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligencecited by 9
  9. 2026AdaReasoner: Dynamic Tool Orchestration for Iterative Visual Reasoningcited by 7
  10. 2025UR-Bench: A Benchmark for Multi-Hop Reasoning over Ultra-High-Resolution Imagescited by 2
  11. 2026APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughtscited by 0
  12. 2026Think, Look, and Revise: Inconsistency-Aware Visual Self-Correction in MLLMscited by 0
2 more
  1. 2026On-Policy Visual Evidence Distillationcited by 0
  2. 2026VepAgent: Bridging Causal-Transition via Tool-Augmented Reinforcement Learning for Video Event Predictioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025See, Think, Learn: A Self-Taught Multimodal Reasoner
  2. 2026From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models
  3. 2024Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
  4. 2025Elevating Visual Question Answering through Implicitly Learned Reasoning Pathways in LVLMs
  5. 2025Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Models
  6. 2026From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models
  7. 2024ProReason: Multi-Modal Proactive Reasoning with Decoupled Eyesight and Wisdom
  8. 2026Learning Adaptive Reasoning Paths for Efficient Visual Reasoning