# Beyond Unimodal Shortcuts: MLLMs as Cross-Modal Reasoners for Grounded Named Entity Recognition

arXiv: https://arxiv.org/abs/2602.04486 · HTML: https://mlsanity.com/paper/2602.04486/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 28 of its 57 references that are in the map.
- First posted on arXiv 4 Feb 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 28 of 28)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [Visual-RFT: Visual Reinforcement Fine-Tuning](https://mlsanity.com/paper/2503.01785/) (2025)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025)
- [Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph](https://mlsanity.com/paper/2307.07697/) (2023)
- [Imagine while Reasoning in Space: Multimodal Visualization-of-Thought](https://mlsanity.com/paper/2501.07542/) (2025)
- [MiMo-VL Technical Report](https://mlsanity.com/paper/2506.03569/) (2025)
- [mDPO: Conditional Preference Optimization for Multimodal Large Language Models](https://mlsanity.com/paper/2406.11839/) (2024)
- [The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio](https://mlsanity.com/paper/2410.12787/) (2024)
- [R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPO](https://mlsanity.com/paper/2505.16673/) (2025)
- [Multimodal Named Entity Recognition for Short Social Media Posts](https://mlsanity.com/paper/1802.07862/) (2018)
- [GREC: Generalized Referring Expression Comprehension](https://mlsanity.com/paper/2308.16182/) (2023)
- [Evaluating and Steering Modality Preferences in Multimodal Large Language Model](https://mlsanity.com/paper/2505.20977/) (2025)
- [ReasonGen-R1: CoT for Autoregressive Image generation models through SFT and RL](https://mlsanity.com/paper/2505.24875/) (2025)
- [Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives](https://mlsanity.com/paper/2510.04996/) (2025)
- [Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoning](https://mlsanity.com/paper/2506.01748/) (2025)
- [ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learning](https://mlsanity.com/paper/2507.02834/) (2025)
- [Memory-augmented Query Reconstruction for LLM-based Knowledge Graph Reasoning](https://mlsanity.com/paper/2503.05193/) (2025)
- [ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search](https://mlsanity.com/paper/2508.19576/) (2025)
- [LLM with Relation Classifier for Document-Level Relation Extraction](https://mlsanity.com/paper/2408.13889/) (2024)
- [LLMs as Bridges: Reformulating Grounded Multimodal Named Entity Recognition](https://mlsanity.com/paper/2402.09989/) (2024)
- [LoCoT2V-Bench: Benchmarking Long-Form and Complex Text-to-Video Generation](https://mlsanity.com/paper/2510.26412/) (2025)
- [SAMChat: Introducing Chain of Thought Reasoning and GRPO to a Multimodal Small Language Model for Small Scale Remote Sensing](https://mlsanity.com/paper/2505.07984/) (2025)
- [SCANNER: Knowledge-Enhanced Approach for Robust Multi-modal Named Entity Recognition of Unseen Entities](https://mlsanity.com/paper/2404.01914/) (2024)
- [ReFineG: Synergizing Small Supervised Models and LLMs for Low-Resource Grounded Multimodal NER](https://mlsanity.com/paper/2509.10975/) (2025)

## Built on by (most cited first, 6 of 6)

- [Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation](https://mlsanity.com/paper/2605.18740/) (2026)
- [Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception](https://mlsanity.com/paper/2602.11858/) (2026)
- [Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition](https://mlsanity.com/paper/2606.00959/) (2026)
- [MLLMRec-R1: Incentivizing Reasoning Capability in Large Language Models for Multimodal Sequential Recommendation](https://mlsanity.com/paper/2603.06243/) (2026)
- [Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs](https://mlsanity.com/paper/2608.04759/) (2026)
- [Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Models](https://mlsanity.com/paper/2609.37568/) (2026)

## Abstract (arXiv)

Grounded Multimodal Named Entity Recognition (GMNER) aims to extract text-based entities, assign them semantic categories, and ground them to corresponding visual regions. In this work, we explore the potential of Multimodal Large Language Models (MLLMs) to perform GMNER in an end-to-end manner, moving beyond their typical role as auxiliary tools within cascaded pipelines. Crucially, our investigation reveals a fundamental challenge: MLLMs exhibit modality bias, including visual bias and textual bias, which stems from their tendency to take unimodal shortcuts rather than rigorous cross-modal verification. To address this, we propose Modality-aware Consistency Reasoning (MCR), which enforces structured cross-modal reasoning through Multi-style Reasoning Schema Injection (MRSI) and Constraint-guided Verifiable Optimization (CVO). MRSI transforms abstract constraints into executable reasoning chains, while CVO empowers the model to dynamically align its reasoning trajectories with Group Relative Policy Optimization (GRPO). Experiments on GMNER and visual grounding tasks demonstrate that MCR effectively mitigates modality bias and achieves superior performance compared to existing baselines.