Towards Customized Multimodal Role-Play
Not on the map. Why?
Abstract (arXiv)
Unified multimodal understanding and generation models enable richer human-AI interaction. Yet jointly customizing a character's persona, dialogue style, and visual identity while maintaining output consistency across modalities remains largely unexplored. To mitigate this gap, we introduce a new task, Customized Multimodal Role-Play (CMRP). We construct the RoleScape-20 dataset comprising 20 characters, including training and evaluation data that cover persona, stylistic descriptions, visual/expressive cues, and text-image interactions. Building on a unified model, we devise UniCharacter, a two-stage training framework containing Unified Supervised Finetuning (Unified-SFT) and character-specific group relative policy optimization (Character-GRPO). Given only 10 images plus corresponding interaction examples, the model acquires the target character and exhibits coherent persona, style, and visual identity in both generated text and images. This process takes about 100 GPU hours. Experiments on the RoleScape-20 dataset show that the proposed method substantially outperforms prior approaches. Ablation studies further validate the effectiveness of our cross-modal consistency design and few-shot customization strategy. We argue that CMRP, coupled with unified modeling, provides a basis for next-generation characterful and immersive interactive agents.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 43 references that are in the map. First posted on arXiv 1 May 2026.
Builds on
35 of 43 references.
References · 35
- 2025Qwen3 Technical Reportcited by 8,195
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024GPT-4o System Cardcited by 4,467
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
- 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
- 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
- 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
- 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
23 more
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
- 2023Character-LLM: A Trainable Agent for Role-Playingcited by 335
- 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 303
- 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
- 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
- 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 204
- 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
- 2024Yo'LLaVA: Your Personalized Language and Vision Assistantcited by 61
- 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
- 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 55
- 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 53
- 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 52
- 2024Towards Language-Driven Video Inpainting via Multimodal Large Language Modelscited by 33
- 2023Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentationcited by 28
- 2025UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learningcited by 23
- 2025YoChameleon: Personalized Vision and Language Generationcited by 20
- 2024DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generationcited by 16
- 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
- 2025Decouple and Track: Benchmarking and Improving Video Diffusion Transformers for Motion Transfercited by 14
- 2025RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Modelscited by 13
- 2024DreamRelation: Bridging Customization and Relation Generationcited by 9
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Crafting Customisable Characters with LLMs: A Persona-Driven Role-Playing Agent Framework
- 2026Through the Lens of Character: Resolving Modality-Role Interference in Multimodal Role-Playing Agent
- 2025Video2Roleplay: A Multimodal Dataset and Framework for Video-Guided Role-playing Agents
- 2024MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents
- 2025A Thousand Words or An Image: Studying the Influence of Persona Modality in Multimodal LLMs
- 2024Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment
- 2025VoxRole: A Comprehensive Benchmark for Evaluating Speech-Based Role-Playing Agents
- 2026Character Beyond Speech: Leveraging Role-Playing Evaluation in Audio Large Language Models via Reinforcement Learning