# RoleMRC: A Fine-Grained Composite Benchmark for Role-Playing and Instruction-Following

arXiv: https://arxiv.org/abs/2502.11387 · HTML: https://mlsanity.com/paper/2502.11387/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Feb 2025.

## Built on by (most cited first, 18 of 18)

- [Reward Modeling from Natural Language Human Feedback](https://mlsanity.com/paper/2601.07349/) (2026)
- [IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation](https://mlsanity.com/paper/2603.04738/) (2026)
- [PersonaEval: Are LLM Evaluators Human Enough to Judge Role-Play?](https://mlsanity.com/paper/2508.10014/) (2025)
- [Two Heads Are Better Than One: Dual-Model Verbal Reflection at Inference-Time](https://mlsanity.com/paper/2502.19230/) (2025)
- [MOA: Multi-Objective Alignment for Role-Playing Agents](https://mlsanity.com/paper/2512.09756/) (2025)
- [Character-R1: Enhancing Role-Aware Reasoning in Role-Playing Agents via RLVR](https://mlsanity.com/paper/2601.04611/) (2026)
- [Sparse Activation Editing for Reliable Instruction Following in Narratives](https://mlsanity.com/paper/2505.16505/) (2025)
- [RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systems](https://mlsanity.com/paper/2512.10575/) (2025)
- [Video2Roleplay: A Multimodal Dataset and Framework for Video-Guided Role-playing Agents](https://mlsanity.com/paper/2509.15233/) (2025)
- [FURINA: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collaboration Pipeline](https://mlsanity.com/paper/2510.06800/) (2025)
- [RoleCDE:Benchmarking and Mitigating Role-Alignment Trade-offs in Role-Playing Agents](https://mlsanity.com/paper/2606.01552/) (2026)
- [VoxRole: A Comprehensive Benchmark for Evaluating Speech-Based Role-Playing Agents](https://mlsanity.com/paper/2509.03940/) (2025)
- [Learning a Single Token to Replace Long System Prompts in LLMs](https://mlsanity.com/paper/2511.23271/) (2025)
- [Understanding Generalization in Role-Playing Models via Information Theory](https://mlsanity.com/paper/2512.17270/) (2025)
- [TRACE Bench: Task-driven Roleplay Agentic Checklist Evaluation](https://mlsanity.com/paper/2608.11236/) (2026)
- [Reward-Decomposed Reinforcement Learning for Immersive Video Role-Playing](https://mlsanity.com/paper/2605.04733/) (2026)
- [BOOKMARKS: Efficient Active Storyline Memory for Role-playing](https://mlsanity.com/paper/2605.14169/) (2026)
- [Persona-Pruner: Sculpting Lightweight Models for Role-Playing](https://mlsanity.com/paper/2606.14695/) (2026)

## Abstract (arXiv)

Role-playing is important for Large Language Models (LLMs) to follow diverse instructions while maintaining role identity and the role's pre-defined ability limits. Existing role-playing datasets mostly contribute to controlling role style and knowledge boundaries, but overlook role-playing in instruction-following scenarios. We introduce a fine-grained role-playing and instruction-following composite benchmark, named RoleMRC, including: (1) Multi-turn dialogues between ideal roles and humans, including free chats or discussions upon given passages; (2) Role-playing machine reading comprehension, involving response, refusal, and attempts according to passage answerability and role ability; (3) More complex scenarios with nested, multi-turn and prioritized instructions. The final RoleMRC features a 10.2k role profile meta-pool, 37.9k well-synthesized role-playing instructions, and 1.4k testing samples. We develop a pipeline to quantitatively evaluate the fine-grained role-playing and instruction-following capabilities of several mainstream LLMs, as well as models that are fine-tuned on our data. Moreover, cross-evaluation on external role-playing datasets confirms that models fine-tuned on RoleMRC enhances instruction-following without compromising general role-playing and reasoning capabilities. We also probe the neural-level activation maps of different capabilities over post-tuned LLMs. Access to our RoleMRC, RoleMRC-mix and Codes: https://github.com/LuJunru/RoleMRC.