mlsanityData to 7 Oct 2026

DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Models

arXiv 2402.14767 ↗First posted 22 Feb 2024cs.CVcited by 14

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We present DualFocus, a novel framework for integrating macro and micro perspectives within multi-modal large language models (MLLMs) to enhance vision-language task performance. Current MLLMs typically singularly focus on inputs at a predefined resolution, resulting in deficiencies in detailed questions involving local regions. We introduced a DualFocus mechanism where the model concentrates on the image from a macro perspective, responses to the question, and identifies suitable sub-regions to zoom in for subsequent micro perspective analysis. Via the integration of answers from both macro and micro perspectives, the model is adept at addressing tasks that encompass global, detailed, and combined considerations. To endows the DualFocus mechanism in MLLMs, we curated a tailored dataset derived from the Visual Genome (VG) and adapted it to align with the training regimen of DualFocus. Through comparative studies across different model sizes and benchmarks, we demonstrate DualFocus's superiority in balancing detailed examination with holistic insight, significantly reducing hallucination instances in MLLMs and improving their performance in various vision-language tasks.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 41 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

32 of 41 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model29 Jan
  2. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  3. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023MLLM-DataEngine: An Iterative Refinement Approach for MLLM25 Aug
  6. 2023WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large Models21 Aug
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  8. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2019Towards VQA Models That Can Readcited by 1,696
12 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  4. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  5. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  6. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  7. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  8. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  9. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  10. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  11. 2023V3Det: Vast Vocabulary Visual Detection Datasetcited by 81
  12. 2023OtterHD: A High-Resolution Multi-modality Modelcited by 72

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 6; 2026: 2.

14 papers.

Sort:

By area:

  1. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  2. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  3. 2024Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Modelscited by 65
  4. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  5. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 37
  6. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  7. 2025Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinementcited by 18
  8. 2025VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Modelscited by 13
  9. 2025Look Where It Matters: Training-Free Ultra-HR Remote Sensing VQA via Adaptive Zoom Searchcited by 11
  10. 2024Right this way: Can VLMs Guide Us to See More to Answer Questions?cited by 10
  11. 2025TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connectioncited by 7
  12. 2025VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoningcited by 5
2 more
  1. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  2. 2026Event-Aware Instructed Assistant for Referring Video Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing Right but Saying Wrong: Inter- and Intra-Layer Refinement in MLLMs without Training
  2. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
  3. 2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance
  4. 2024EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model
  5. 2025Investigating The Functional Roles of Attention Heads in Vision Language Models: Evidence for Reasoning Modules
  6. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach
  7. 2025Language-Specific Layer Matters: Efficient Multilingual Enhancement for Large Vision-Language Models
  8. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge