# Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks

arXiv: https://arxiv.org/abs/2602.23898 · HTML: https://mlsanity.com/paper/2602.23898/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 14 of its 39 references that are in the map.
- First posted on arXiv 27 Feb 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 14 of 14)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://mlsanity.com/paper/2412.10302/) (2024)
- [GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning](https://mlsanity.com/paper/2507.01006/) (2025)
- [Kimi-VL Technical Report](https://mlsanity.com/paper/2504.07491/) (2025)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [CogVLM2: Visual Language Models for Image and Video Understanding](https://mlsanity.com/paper/2408.16500/) (2024)
- [Semantic-SAM: Segment and Recognize Anything at Any Granularity](https://mlsanity.com/paper/2307.04767/) (2023)
- [Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Models](https://mlsanity.com/paper/2406.16866/) (2024)
- [Think or Not Think: A Study of Explicit Thinking in Rule-Based Visual Reinforcement Fine-Tuning](https://mlsanity.com/paper/2503.16188/) (2025)
- [Visual Referring Expression Recognition: What Do Systems Actually Learn?](https://mlsanity.com/paper/1805.11818/) (2018)
- [CoT Referring: Improving Referring Expression Tasks with Grounded Reasoning](https://mlsanity.com/paper/2510.06243/) (2025)

## Built on by (most cited first, 9 of 9)

- [Iterative Visual Thinking and the Self-Correction Mirage in VLM Grounding](https://mlsanity.com/paper/2606.13156/) (2026)
- [MUSE: A Unified Agentic Harness for MLLMs](https://mlsanity.com/paper/2606.03005/) (2026)
- [ROSE: Benchmarking the Perception-to-Action Gap in Multimodal Models](https://mlsanity.com/paper/2606.19965/) (2026)
- [Switch-Reasoner: Learn When to Think in Multitask Mixtures via Reinforcement Learning](https://mlsanity.com/paper/2607.08572/) (2026)
- [When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs](https://mlsanity.com/paper/2608.23978/) (2026)
- [Pay More Attention To Text In High-Resolution MLLMs](https://mlsanity.com/paper/2609.23495/) (2026)
- [FORUM: Frozen Outputs Reconciled Using Model Agreement for Visual Grounding](https://mlsanity.com/paper/2609.37488/) (2026)
- [EviRover: Reinforcing Agentic Perception Beyond a Glance](https://mlsanity.com/paper/2609.40230/) (2026)
- [ReferFlow: Beyond Final-Target Localization for Complex Referring Expressions](https://mlsanity.com/paper/2610.20352/) (2026)

## Abstract (arXiv)

Referring Expression Comprehension (REC) links language to region level visual perception. Standard benchmarks (RefCOCO, RefCOCO+, RefCOCOg) have progressed rapidly with multimodal LLMs but remain weak tests of visual reasoning and grounding: (i) many expressions are very short, leaving little reasoning demand; (ii) images often contain few distractors, making the target easy to find; and (iii) redundant descriptors enable shortcut solutions that bypass genuine text understanding and visual reasoning. We introduce Ref-Adv, a modern REC benchmark that suppresses shortcuts by pairing linguistically nontrivial expressions with only the information necessary to uniquely identify the target. The dataset contains referring expressions on real images, curated with hard distractors and annotated with reasoning facets including negation. We conduct comprehensive ablations (word order perturbations and descriptor deletion sufficiency) to show that solving Ref-Adv requires reasoning beyond simple cues, and we evaluate a broad suite of contemporary multimodal LLMs on Ref-Adv. Despite strong results on RefCOCO, RefCOCO+, and RefCOCOg, models drop markedly on Ref-Adv, revealing reliance on shortcuts and gaps in visual reasoning and grounding. We provide an in depth failure analysis and aim for Ref-Adv to guide future work on visual reasoning and grounding in MLLMs.