# DyFo: A Training-Free Dynamic Focus Visual Search for Enhancing LMMs in Fine-Grained Visual Understanding

arXiv: https://arxiv.org/abs/2504.14920 · HTML: https://mlsanity.com/paper/2504.14920/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 64 references that are in the map.
- First posted on arXiv 21 Apr 2025.

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [Visual Programming: Compositional visual reasoning without training](https://mlsanity.com/paper/2211.11559/) (2022)
- [Ferret: Refer and Ground Anything Anywhere at Any Granularity](https://mlsanity.com/paper/2310.07704/) (2023)
- [V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs](https://mlsanity.com/paper/2312.14135/) (2023)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)

## Built on by (most cited first, 30 of 49)

- [DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning](https://mlsanity.com/paper/2505.14362/) (2025)
- [Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search](https://mlsanity.com/paper/2509.07969/) (2025)
- [Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMs](https://mlsanity.com/paper/2505.15436/) (2025)
- [Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning](https://mlsanity.com/paper/2508.04416/) (2025)
- [Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinement](https://mlsanity.com/paper/2506.01663/) (2025)
- [HiDe: Rethinking The Zoom-IN method in High Resolution MLLMs via Hierarchical Decoupling](https://mlsanity.com/paper/2510.00054/) (2025)
- [Thinking in 360deg: Humanoid Visual Search in the Wild](https://mlsanity.com/paper/2511.20351/) (2025)
- [Look Where It Matters: Training-Free Ultra-HR Remote Sensing VQA via Adaptive Zoom Search](https://mlsanity.com/paper/2511.20460/) (2025)
- [From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoning](https://mlsanity.com/paper/2511.23031/) (2025)
- [GeoEyes: On-Demand Visual Focusing for Evidence-Grounded Understanding of Ultra-High-Resolution Remote Sensing Imagery](https://mlsanity.com/paper/2602.14201/) (2026)
- [What, Whether and How? Unveiling Process Reward Models for Thinking with Images Reasoning](https://mlsanity.com/paper/2602.08346/) (2026)
- [Reinforcing VLMs to Use Tools for Detailed Visual Reasoning Under Resource Constraints](https://mlsanity.com/paper/2506.14821/) (2025)
- [CoFFT: Chain of Foresight-Focus Thought for Visual Language Models](https://mlsanity.com/paper/2509.22010/) (2025)
- [DeepScan: A Training-Free Framework for Visually Grounded Reasoning in Large Vision-Language Models](https://mlsanity.com/paper/2603.03857/) (2026)
- [Improving Visual Reasoning with Iterative Evidence Refinement](https://mlsanity.com/paper/2603.14117/) (2026)
- [DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding](https://mlsanity.com/paper/2605.15542/) (2026)
- [CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding](https://mlsanity.com/paper/2509.23379/) (2025)
- [Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMs](https://mlsanity.com/paper/2510.00705/) (2025)
- [GeoViS: Geospatially Rewarded Visual Search for Remote Sensing Visual Grounding](https://mlsanity.com/paper/2512.02715/) (2025)
- [Thinking with Images via Self-Calling Agent](https://mlsanity.com/paper/2512.08511/) (2025)
- [Training Multi-Image Vision Agents via End2End Reinforcement Learning](https://mlsanity.com/paper/2512.08980/) (2025)
- [Glance-or-Gaze: Incentivizing LMMs to Adaptively Focus Search via Reinforcement Learning](https://mlsanity.com/paper/2601.13942/) (2026)
- [E3VS-Bench: A Benchmark for Viewpoint-Dependent Active Perception in 3D Gaussian Splatting Scenes](https://mlsanity.com/paper/2604.17969/) (2026)
- [AutoFocus: Uncertainty-Aware Active Visual Search for GUI Grounding](https://mlsanity.com/paper/2605.02630/) (2026)
- [Blink: Dynamic Visual Token Resolution for Enhanced Multimodal Understanding](https://mlsanity.com/paper/2512.10548/) (2025)
- [Seeing as Experts Do: A Knowledge-Augmented Agent for Open-Set Fine-Grained Visual Understanding](https://mlsanity.com/paper/2603.03762/) (2026)
- [Reinforce to Learn, Elect to Reason: A Dual Paradigm for Video Reasoning](https://mlsanity.com/paper/2604.04379/) (2026)
- [Are Tools Always Beneficial? Learning to Invoke Tools Adaptively for Dual-Mode Multimodal LLM Reasoning](https://mlsanity.com/paper/2605.19852/) (2026)
- [Evaluating and Enhancing Negation Comprehension in Remote Sensing MLLMs](https://mlsanity.com/paper/2606.20177/) (2026)
- [Iterative Zoom-In: Temporal Interval Exploration for Long Video Understanding](https://mlsanity.com/paper/2507.02946/) (2025)

## Abstract (arXiv)

Humans can effortlessly locate desired objects in cluttered environments, relying on a cognitive mechanism known as visual search to efficiently filter out irrelevant information and focus on task-related regions. Inspired by this process, we propose Dyfo (Dynamic Focus), a training-free dynamic focusing visual search method that enhances fine-grained visual understanding in large multimodal models (LMMs). Unlike existing approaches which require additional modules or data collection, Dyfo leverages a bidirectional interaction between LMMs and visual experts, using a Monte Carlo Tree Search (MCTS) algorithm to simulate human-like focus adjustments. This enables LMMs to focus on key visual regions while filtering out irrelevant content, without introducing additional training caused by vocabulary expansion or the integration of specialized localization modules. Experimental results demonstrate that Dyfo significantly improves fine-grained visual understanding and reduces hallucination issues in LMMs, achieving superior performance across both fixed and dynamic resolution models. The code is available at https://github.com/PKU-ICST-MIPL/DyFo_CVPR2025