GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Multimodal Large Language Models (MLLMs) have demonstrated impressive progress in single-image grounding and general multi-image understanding. Recently, some methods begin to address multi-image grounding. However, they are constrained by single-target localization and limited types of practical tasks, due to the lack of unified modeling for generalized grounding tasks. Therefore, we propose GeM-VG, an MLLM capable of Generalized Multi-image Visual Grounding. To support this, we systematically categorize and organize existing multi-image grounding tasks according to their reliance of cross-image cues and reasoning, and introduce the MG-Data-240K dataset, addressing the limitations of existing datasets regarding target quantity and image relation. To tackle the challenges of robustly handling diverse multi-image grounding tasks, we further propose a hybrid reinforcement finetuning strategy that integrates chain-of-thought (CoT) reasoning and direct answering, considering their complementary strengths. This strategy adopts an R1-like algorithm guided by a carefully designed rule-based reward, effectively enhancing the model's overall perception and reasoning capabilities. Extensive experiments demonstrate the superior generalized grounding capabilities of our model. For multi-image grounding, it outperforms the previous leading MLLMs by 2.0% and 9.7% on MIG-Bench and MC-Bench, respectively. In single-image grounding, it achieves a 9.1% improvement over the base model on ODINW. Furthermore, our model retains strong capabilities in general multi-image understanding.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 49 references that are in the map. First posted on arXiv 8 Jan 2026.
Builds on
32 of 49 references.
Closest ancestors · 5
Same direction, newest first
- 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs16 Oct
- 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos29 Sep
- 2024Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referring14 Mar
- 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
- 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
Other references · 27
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
- 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
- 2016Modeling Context in Referring Expressionscited by 1,054
- 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
15 more
- 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
- 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
- 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
- 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
- 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
- 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
- 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
- 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
- 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
- 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
- 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 83
- 2024LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoningcited by 52
- 2023Described Object Detection: Liberating Object Detection with Flexible Expressionscited by 40
- 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Imagescited by 36
- 2025KARL: Knowledge-Aware Reasoning and Reinforcement Learning for Knowledge-Intensive Visual Groundingcited by 21
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Improving the Reasoning of Multi-Image Grounding in MLLMs via Reinforcement Learning
- 2025Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models
- 2025Towards Understanding Visual Grounding in Visual Language Models
- 2025Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning
- 2025MIRG-RL: Multi-Image Reasoning and Grounding with Reinforcement Learning
- 2023Language-Guided Diffusion Model for Visual Grounding
- 2025PostAlign: Multimodal Grounding as a Corrective Lens for MLLMs
- 2026Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding