# CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulation

arXiv: https://arxiv.org/abs/2502.09082 · HTML: https://mlsanity.com/paper/2502.09082/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 13 Feb 2025.

## Builds on (most cited first, 30 of 30)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [Toxicity in ChatGPT: Analyzing Persona-assigned Language Models](https://mlsanity.com/paper/2304.05335/) (2023)
- [Character-LLM: A Trainable Agent for Role-Playing](https://mlsanity.com/paper/2310.10158/) (2023)
- [Role-Play with Large Language Models](https://mlsanity.com/paper/2305.16367/) (2023)
- [Recursively Summarizing Books with Human Feedback](https://mlsanity.com/paper/2109.10862/) (2021)
- [Self-Preference Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2410.21819/) (2024)
- [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://mlsanity.com/paper/2404.18231/) (2024)
- [InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews](https://mlsanity.com/paper/2310.17976/) (2023)
- [RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models](https://mlsanity.com/paper/2310.00746/) (2023)
- [CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation](https://mlsanity.com/paper/2401.01275/) (2024)
- [BooookScore: A systematic exploration of book-length summarization in the era of LLMs](https://mlsanity.com/paper/2310.00785/) (2023)
- [Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment](https://mlsanity.com/paper/2401.12474/) (2024)
- [Evaluating Large Language Model Biases in Persona-Steered Generation](https://mlsanity.com/paper/2405.20253/) (2024)
- [ChatHaruhi: Reviving Anime Character in Reality via Large Language Model](https://mlsanity.com/paper/2308.09597/) (2023)
- [Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works](https://mlsanity.com/paper/2404.12726/) (2024)
- [CharacterGLM: Customizing Chinese Conversational AI Characters with Large Language Models](https://mlsanity.com/paper/2311.16832/) (2023)
- [HoLLMwood: Unleashing the Creativity of Large Language Models in Screenwriting via Role Playing](https://mlsanity.com/paper/2406.11683/) (2024)
- [RoleEval: A Bilingual Role Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2312.16132/) (2023)
- [MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents](https://mlsanity.com/paper/2408.04203/) (2024)
- [Tachikuma: Understading Complex Interactions with Multi-Character and Novel Objects by Large Language Models](https://mlsanity.com/paper/2307.12573/) (2023)
- [BookWorm: A Dataset for Character Description and Analysis](https://mlsanity.com/paper/2410.10372/) (2024)

## Built on by (most cited first, 26 of 26)

- [Mind the Sim2Real Gap in User Simulation for Agentic Tasks](https://mlsanity.com/paper/2603.11245/) (2026)
- [Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoning](https://mlsanity.com/paper/2506.01748/) (2025)
- [NarraBench: A Comprehensive Framework for Narrative Benchmarking](https://mlsanity.com/paper/2510.09869/) (2025)
- [OdysSim: Building Foundation Models for Human Behavior Simulation](https://mlsanity.com/paper/2606.14199/) (2026)
- [CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization](https://mlsanity.com/paper/2508.09074/) (2025)
- [PersonaEval: Are LLM Evaluators Human Enough to Judge Role-Play?](https://mlsanity.com/paper/2508.10014/) (2025)
- [AdaMARP: An Adaptive Multi-Agent Interaction Framework for General Immersive Role-Playing](https://mlsanity.com/paper/2601.11007/) (2026)
- [The Chameleon's Limit: Investigating Persona Collapse and Homogenization in Large Language Models](https://mlsanity.com/paper/2604.24698/) (2026)
- [Guess What I am Thinking: A Benchmark for Inner Thought Reasoning of Role-Playing Language Agents](https://mlsanity.com/paper/2503.08193/) (2025)
- [RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systems](https://mlsanity.com/paper/2512.10575/) (2025)
- [Reinforcing Human Behavior Simulation via Verbal Feedback](https://mlsanity.com/paper/2605.20506/) (2026)
- [Can LLMs Grasp Implicit Cultural Values? Benchmarking LLMs' Cultural Intelligence with CQ-Bench](https://mlsanity.com/paper/2504.01127/) (2025)
- [Can LLMs Simulate Personas with Reversed Performance? A Systematic Investigation for Counterfactual Instruction Following in Math Reasoning Context](https://mlsanity.com/paper/2504.06460/) (2025)
- [FURINA: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collaboration Pipeline](https://mlsanity.com/paper/2510.06800/) (2025)
- [Compass-V2 Technical Report](https://mlsanity.com/paper/2504.15527/) (2025)
- [PRISON: Unmasking the Criminal Potential of Large Language Models](https://mlsanity.com/paper/2506.16150/) (2025)
- [VoxRole: A Comprehensive Benchmark for Evaluating Speech-Based Role-Playing Agents](https://mlsanity.com/paper/2509.03940/) (2025)
- [Social Welfare Function Leaderboard: When LLM Agents Allocate Social Welfare](https://mlsanity.com/paper/2510.01164/) (2025)
- [Implementing Long Text Style Transfer with LLMs through Dual-Layered Sentence and Paragraph Structure Extraction and Mapping](https://mlsanity.com/paper/2505.07888/) (2025)
- [Test-Time-Matching: Decouple Personality, Memory, and Linguistic Style in LLM-based Role-Playing Language Agent](https://mlsanity.com/paper/2507.16799/) (2025)
- [LLMs vs. Chinese Anime Enthusiasts: A Comparative Study on Emotionally Supportive Role-Playing](https://mlsanity.com/paper/2508.06388/) (2025)
- [Individualized Cognitive Simulation in Large Language Models: Evaluating Different Cognitive Representation Methods](https://mlsanity.com/paper/2510.20252/) (2025)
- [Reward-Decomposed Reinforcement Learning for Immersive Video Role-Playing](https://mlsanity.com/paper/2605.04733/) (2026)
- [From Blueprint to Reality: Modeling and Applying Putnam's Social Capital Theory with LLM-based Multi-agent Simulations](https://mlsanity.com/paper/2607.06080/) (2026)
- [Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2608.26735/) (2026)
- [ANIMASK: What the Model Contributes to Role Play in Simulated Story Worlds](https://mlsanity.com/paper/2609.16667/) (2026)

## Abstract (arXiv)

Role-playing language agents (RPLAs) have emerged as promising applications of large language models (LLMs). However, simulating established characters presents a challenging task for RPLAs, due to the lack of authentic character datasets and nuanced evaluation methods using such data. In this paper, we present CoSER, a collection of a high-quality dataset, open models, and an evaluation protocol towards effective RPLAs of established characters. The CoSER dataset covers 17,966 characters from 771 renowned books. It provides authentic dialogues with real-world intricacies, as well as diverse data types such as conversation setups, character experiences and internal thoughts. Drawing from acting methodology, we introduce given-circumstance acting for training and evaluating role-playing LLMs, where LLMs sequentially portray multiple characters in book scenes. Using our dataset, we develop CoSER 8B and CoSER 70B, i.e., advanced open role-playing LLMs built on LLaMA-3.1 models. Extensive experiments demonstrate the value of the CoSER dataset for RPLA training, evaluation and retrieval. Moreover, CoSER 70B exhibits state-of-the-art performance surpassing or matching GPT-4o on our evaluation and three existing benchmarks, i.e., achieving 75.80% and 93.47% accuracy on the InCharacter and LifeChoice benchmarks respectively.