mlsanityData to 7 Oct 2026

Through the Lens of Character: Resolving Modality-Role Interference in Multimodal Role-Playing Agent

arXiv 2605.09443 ↗First posted 10 May 2026cs.CV cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

The advancement of Multimodal Large Language Models (MLLMs) has expanded Role-Playing Agents (RPAs) into visually grounded environments. However, human vision is inherently subjective and identity-driven, whereas existing MLLMs extract objective, character-agnostic features for general tasks. In RPAs, this generic visual noise overpowers fragile character traits, causing Modality-Role Interference (MRI), where agents struggle to integrate visual grounding and character consistency. To address this, we introduce the training-free Character-Aware Visual Intervention (CAVI) framework, enabling agents to perceive the world through the lens of character. CAVI systematically targets MRI: macroscopically, Character-Guided Token Pruning (CTP) restricts the visual receptive field to role-relevant entities; microscopically, Orthogonal Feature Modulation (OFM) projects tokens onto a character-context subspace to extract aligned facts; and during decoding, Modality-Adaptive Role Steering (MARS) dynamically optimizes steering intensity based on visual reliance. Extensive experiments show CAVI effectively alleviates MRI, significantly enhancing character-consistent multimodal interactions.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 46 references that are in the map. First posted on arXiv 10 May 2026.

Builds on

40 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023A Survey on Multimodal Large Language Models23 Jun
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 38

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  3. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  4. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  5. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  6. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 335
  7. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  8. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  9. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  10. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  11. 2023InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviewscited by 177
  12. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
26 more
  1. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  2. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  3. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedbackcited by 80
  4. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  5. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  6. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimizationcited by 67
  7. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 61
  8. 2024Multi-Object Hallucination in Vision-Language Modelscited by 60
  9. 2023Multi-Party Chat: Conversational Agents in Group Settings with Humans and Modelscited by 53
  10. 2024Neeko: Leveraging Dynamic LoRA for Efficient Multi-Character Role-Playing Agentcited by 52
  11. 2024Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Modelscited by 47
  12. 2024The Oscars of AI Theater: A Survey on Role-Playing with Language Modelscited by 47
  13. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucinationcited by 47
  14. 2025ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Modelscited by 40
  15. 2024Character is Destiny: Can Role-Playing Language Agents Make Persona-Driven Decisions?cited by 34
  16. 2023Enhancing Personalized Dialogue Generation with Contrastive Latent Variables: Combining Sparse and Dense Personacited by 24
  17. 2023Editing Personality for Large Language Modelscited by 22
  18. 2024MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agentscited by 21
  19. 2024Identity-Driven Hierarchical Role-Playing Agentscited by 18
  20. 2024Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Modelscited by 15
  21. 2025Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoningcited by 14
  22. 2024CharacterGPT: A Persona Reconstruction Framework for Role-Playing Agentscited by 10
  23. 2025RoleRAG: Enhancing LLM Role-Playing via Graph Guided Retrievalcited by 7
  24. 2026Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigationcited by 7
  25. 2026Character-R1: Enhancing Role-Aware Reasoning in Role-Playing Agents via RLVRcited by 5
  26. 2025Video2Roleplay: A Multimodal Dataset and Framework for Video-Guided Role-playing Agentscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Customized Multimodal Role-Play
  2. 2026Identifying and Mitigating Bottlenecks in Role-Playing Agents: A Systematic Study of Disentangling Character Profile Axes
  3. 2026Character Beyond Speech: Leveraging Role-Playing Evaluation in Audio Large Language Models via Reinforcement Learning
  4. 2026Reward-Decomposed Reinforcement Learning for Immersive Video Role-Playing
  5. 2026AdaMARP: An Adaptive Multi-Agent Interaction Framework for General Immersive Role-Playing
  6. 2026MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents
  7. 2025Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models
  8. 2025V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Models