# Video2Roleplay: A Multimodal Dataset and Framework for Video-Guided Role-playing Agents

arXiv: https://arxiv.org/abs/2509.15233 · HTML: https://mlsanity.com/paper/2509.15233/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 17 Sep 2025.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding](https://mlsanity.com/paper/2410.17434/) (2024)
- [Character-LLM: A Trainable Agent for Role-Playing](https://mlsanity.com/paper/2310.10158/) (2023)
- [VideoAgent: Long-form Video Understanding with Large Language Model as Agent](https://mlsanity.com/paper/2403.10517/) (2024)
- [CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation](https://mlsanity.com/paper/2401.01275/) (2024)
- [Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment](https://mlsanity.com/paper/2401.12474/) (2024)
- [Video ReCap: Recursive Captioning of Hour-Long Videos](https://mlsanity.com/paper/2402.13250/) (2024)
- [ReVideo: Remake a Video with Motion and Content Control](https://mlsanity.com/paper/2405.13865/) (2024)
- [The Oscars of AI Theater: A Survey on Role-Playing with Language Models](https://mlsanity.com/paper/2407.11484/) (2024)
- [GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generation](https://mlsanity.com/paper/2311.16511/) (2023)
- [OpenCharacter: Training Customizable Role-Playing LLMs with Large-Scale Synthetic Personas](https://mlsanity.com/paper/2501.15427/) (2025)
- [RoleMRC: A Fine-Grained Composite Benchmark for Role-Playing and Instruction-Following](https://mlsanity.com/paper/2502.11387/) (2025)
- [EduPlanner: LLM-Based Multi-Agent Systems for Customized and Intelligent Instructional Design](https://mlsanity.com/paper/2504.05370/) (2025)
- [Capturing Minds, Not Just Words: Enhancing Role-Playing Language Models with Personality-Indicative Data](https://mlsanity.com/paper/2406.18921/) (2024)

## Built on by (most cited first, 3 of 3)

- [Reward-Decomposed Reinforcement Learning for Immersive Video Role-Playing](https://mlsanity.com/paper/2605.04733/) (2026)
- [Through the Lens of Character: Resolving Modality-Role Interference in Multimodal Role-Playing Agent](https://mlsanity.com/paper/2605.09443/) (2026)
- [Alignment Is All You Need For X-to-4D Generation](https://mlsanity.com/paper/2607.02516/) (2026)

## Abstract (arXiv)

Role-playing agents (RPAs) have attracted growing interest for their ability to simulate immersive and interactive characters. However, existing approaches primarily focus on static role profiles, overlooking the dynamic perceptual abilities inherent to humans. To bridge this gap, we introduce the concept of dynamic role profiles by incorporating video modality into RPAs. To support this, we construct Role-playing-Video60k, a large-scale, high-quality dataset comprising 60k videos and 700k corresponding dialogues. Based on this dataset, we develop a comprehensive RPA framework that combines adaptive temporal sampling with both dynamic and static role profile representations. Specifically, the dynamic profile is created by adaptively sampling video frames and feeding them to the LLM in temporal order, while the static profile consists of (1) character dialogues from training videos during fine-tuning, and (2) a summary context from the input video during inference. This joint integration enables RPAs to generate greater responses. Furthermore, we propose a robust evaluation method covering eight metrics. Experimental results demonstrate the effectiveness of our framework, highlighting the importance of dynamic role profiles in developing RPAs.