Beyond Unimodal Shortcuts: MLLMs as Cross-Modal Reasoners for Grounded Named Entity Recognition
On the map: LLM reasoning → Multimodal reasoning
Abstract (arXiv)
Grounded Multimodal Named Entity Recognition (GMNER) aims to extract text-based entities, assign them semantic categories, and ground them to corresponding visual regions. In this work, we explore the potential of Multimodal Large Language Models (MLLMs) to perform GMNER in an end-to-end manner, moving beyond their typical role as auxiliary tools within cascaded pipelines. Crucially, our investigation reveals a fundamental challenge: MLLMs exhibit modality bias, including visual bias and textual bias, which stems from their tendency to take unimodal shortcuts rather than rigorous cross-modal verification. To address this, we propose Modality-aware Consistency Reasoning (MCR), which enforces structured cross-modal reasoning through Multi-style Reasoning Schema Injection (MRSI) and Constraint-guided Verifiable Optimization (CVO). MRSI transforms abstract constraints into executable reasoning chains, while CVO empowers the model to dynamically align its reasoning trajectories with Group Relative Policy Optimization (GRPO). Experiments on GMNER and visual grounding tasks demonstrate that MCR effectively mitigates modality bias and achieves superior performance compared to existing baselines.
Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 57 references that are in the map. First posted on arXiv 4 Feb 2026.
Builds on
28 of 57 references.
Closest ancestors · 6
Same direction, newest first
- 2025MiMo-VL Technical Report4 Jun
- 2025ReasonGen-R1: CoT for Autoregressive Image generation models through SFT and RL30 May
- 2025R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPO22 May
- 2025Visual-RFT: Visual Reinforcement Fine-Tuning3 Mar
- 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thought13 Jan
- 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb
Other references · 22
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
- 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
- 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 235
- 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
- 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 48
- 2018Multimodal Named Entity Recognition for Short Social Media Postscited by 43
- 2023GREC: Generalized Referring Expression Comprehensioncited by 39
- 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Modelcited by 33
- 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectivescited by 23
10 more
- 2025Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoningcited by 14
- 2025ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learningcited by 13
- 2025Memory-augmented Query Reconstruction for LLM-based Knowledge Graph Reasoningcited by 10
- 2025ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Searchcited by 9
- 2024LLM with Relation Classifier for Document-Level Relation Extractioncited by 8
- 2024LLMs as Bridges: Reformulating Grounded Multimodal Named Entity Recognitioncited by 7
- 2025SAMChat: Introducing Chain of Thought Reasoning and GRPO to a Multimodal Small Language Model for Small Scale Remote Sensingcited by 6
- 2025LoCoT2V-Bench: Benchmarking Long-Form and Complex Text-to-Video Generationcited by 6
- 2024SCANNER: Knowledge-Enhanced Approach for Robust Multi-modal Named Entity Recognition of Unseen Entitiescited by 5
- 2025ReFineG: Synergizing Small Supervised Models and LLMs for Low-Resource Grounded Multimodal NERcited by 1
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 71
- 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 47
- 2026Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decompositioncited by 4
- 2026MLLMRec-R1: Incentivizing Reasoning Capability in Large Language Models for Multimodal Sequential Recommendationcited by 0
- 2026Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMscited by 0
- 2026Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Modelscited by 0
- 2026Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Models29 Sep 2026
- 2026Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs5 Aug 2026
- 2026Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition31 May 2026
- 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation18 May 2026
- 2026MLLMRec-R1: Incentivizing Reasoning Capability in Large Language Models for Multimodal Sequential Recommendation6 Mar 2026
- 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception12 Feb 2026
Similar, not cited
Close in wording, no citation either way
- 2026A Proposal-Free Query-Guided Network for Grounded Multimodal Named Entity Recognition
- 2026E2E-GMNER: End-to-End Generative Grounded Multimodal Named Entity Recognition
- 2023Chain-of-Thought Prompt Distillation for Multimodal Named Entity Recognition and Multimodal Relation Extraction
- 2026SAKE: Self-aware Knowledge Exploitation-Exploration for Grounded Multimodal Named Entity Recognition
- 2024MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge Editing
- 2026Mind the Gap No More: Achieving Zero-Gap Multimodal Integration via One Tokenizer
- 2025Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
- 2025Exploring and Evaluating Multimodal Knowledge Reasoning Consistency of Multimodal Large Language Models