mlsanityData to 7 Oct 2026

ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration

arXiv 2411.16044 ↗First posted 25 Nov 2024cs.CVcited by 99

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in vision-language understanding. Recently, with the integration of test-time scaling techniques, these models have also shown strong potential in visual reasoning. However, most existing reasoning approaches remain text-level in nature: MLLMs are prompted to explore various combinations of textual tokens via their underlying language model, while the visual input remains fixed throughout the reasoning process. This paradigm limits the model's ability to fully exploit rich visual information, particularly when dealing with images containing numerous fine-grained elements. In such cases, vision-level reasoning becomes crucial - where models dynamically zoom into specific regions of the image to gather detailed visual cues necessary for accurate decision-making. In this paper, we propose Zoom Eye, a training-free, model-agnostic tree search algorithm tailored for vision-level reasoning. Zoom Eye treats an image as a hierarchical tree structure, where each child node represents a zoomed-in sub-region of its parent, and the root corresponds to the full image. The algorithm enables MLLMs to simulate human-like zooming behavior by navigating from root to leaf nodes in search of task-relevant visual evidence. We experiment on a series of high-resolution benchmarks and the results demonstrate that Zoom Eye consistently improves the performance of multiple MLLMs by a large margin (e.g., InternVL2.5-8B increases by 15.71% and 17.69% on HR-Bench) and also enables small 3-8B MLLMs to outperform strong large models such as GPT-4o. Code: https://github.com/om-ai-lab/ZoomEye

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 65 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

43 of 65 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models28 Aug
  2. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?23 Aug
  3. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  5. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models5 Mar
  6. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  7. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
22 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  3. 2024Qwen2 Technical Reportcited by 2,076
  4. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  5. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  6. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  7. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  8. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  9. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  10. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  11. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  12. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  13. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  14. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  15. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  16. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 112
  17. 2022Solving Math Word Problems via Cooperative Reasoning induced Language Modelscited by 76
  18. 2024VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Modelscited by 42
  19. 2024How Much Can RAG Help the Reasoning of LLM?cited by 31
  20. 2024FunnelRAG: A Coarse-to-Fine Progressive Retrieval Paradigm for RAGcited by 18
  21. 2024OmChat: A Recipe to Train Multimodal Language Models with Strong Long Context and Video Understandingcited by 10
  22. 2024SEER: Self-Aligned Evidence Extraction for Retrieval-Augmented Generationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 42; 2026: 57.

99 papers.

Sort:

By area:

  1. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  2. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  3. 2025Thyme: Think Beyond Imagescited by 160
  4. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 71
  5. 2025VGR: Visual Grounded Reasoningcited by 70
  6. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 54
  7. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 47
  8. 2025ACTIVE-o3: Empowering MLLMs with Active Perception via Pure Reinforcement Learningcited by 41
  9. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  10. 2025Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learningcited by 32
  11. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoningcited by 28
  12. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 28
38 more
  1. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  2. 2025Retrieval-Augmented Perception: High-Resolution Image Perception Meets Visual RAGcited by 22
  3. 2025Multimodal Language Models See Better When They Look Shallowercited by 22
  4. 2025FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answeringcited by 20
  5. 2025R-VLM: Region-Aware Vision Language Model for Precise GUI Groundingcited by 20
  6. 2025Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinementcited by 18
  7. 2025When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thoughtcited by 18
  8. 2025On the Faithfulness of Visual Thinking: Measurement and Enhancementcited by 17
  9. 2025HiDe: Rethinking The Zoom-IN method in High Resolution MLLMs via Hierarchical Decouplingcited by 16
  10. 2025Thinking in 360deg: Humanoid Visual Search in the Wildcited by 14
  11. 2026Act Wisely: Cultivating Meta-Cognitive Tool Use in Agentic Multimodal Modelscited by 13
  12. 2025VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMscited by 12
  13. 2026VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chainingcited by 12
  14. 2025VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoningcited by 11
  15. 2025Look Where It Matters: Training-Free Ultra-HR Remote Sensing VQA via Adaptive Zoom Searchcited by 11
  16. 2025ZoomV: Temporal Zoom-in for Efficient Long Video Understandingcited by 10
  17. 2025GUI-ARP: Enhancing Grounding with Adaptive Region Perception for GUI Agentscited by 10
  18. 2025Look Less, Reason More: Rollout-Guided Adaptive Pixel-Space Reasoningcited by 9
  19. 2025Video-CoM: Interactive Video Reasoning via Chain of Manipulationscited by 9
  20. 2026GeoEyes: On-Demand Visual Focusing for Evidence-Grounded Understanding of Ultra-High-Resolution Remote Sensing Imagerycited by 9
  21. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Modelscited by 7
  22. 2025Mixture-of-Visual-Thoughts: Exploring Context-Adaptive Reasoning Mode Selection for General Visual Reasoningcited by 7
  23. 2026What, Whether and How? Unveiling Process Reward Models for Thinking with Images Reasoningcited by 7
  24. 2025Reinforcing VLMs to Use Tools for Detailed Visual Reasoning Under Resource Constraintscited by 6
  25. 2025Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculationcited by 6
  26. 2026MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learningcited by 6
  27. 2026DeepScan: A Training-Free Framework for Visually Grounded Reasoning in Large Vision-Language Modelscited by 6
  28. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 6
  29. 2026HyLaR: Hybrid Latent Reasoning with Decoupled Policy Optimizationcited by 6
  30. 2025Unifying Language Agent Algorithms with Graph-based Orchestration Engine for Reproducible Agent Researchcited by 5
  31. 2026Improving Visual Reasoning with Iterative Evidence Refinementcited by 5
  32. 2026Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Modelscited by 5
  33. 2026VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillationcited by 5
  34. 2025Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMscited by 4
  35. 2025FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learningcited by 4
  36. 2025Thinking with Images via Self-Calling Agentcited by 4
  37. 2025Training Multi-Image Vision Agents via End2End Reinforcement Learningcited by 4
  38. 2026SwimBird: Eliciting Switchable Reasoning Mode in Hybrid Autoregressive MLLMscited by 4

50 most cited of 99 All 99 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Zoomer: Adaptive Image Focus Optimization for Black-box MLLM
  2. 2026LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models
  3. 2025Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMs
  4. 2023Enhance Reasoning Ability of Visual-Language Models via Large Language Models
  5. 2025How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding
  6. 2025Visual Funnel: Resolving Contextual Blindness in Multimodal Large Language Models
  7. 2025Diagnosing Visual Reasoning: Challenges, Insights, and a Path Forward
  8. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies