mlsanityData to 14 Oct 2026

CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization

arXiv 2508.09074 ↗First posted 12 Aug 2025cs.CLcited by 10

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 2 more
Abstract (arXiv)

Reinforcement Learning Fine-Tuning (RLFT) has achieved notable success in tasks with objectively verifiable answers (e.g., code generation, mathematical reasoning), yet struggles with open-ended subjective tasks like role-playing dialogue. Traditional reward modeling approaches, which rely on independent sample-wise scoring, face dual challenges: subjective evaluation criteria and unstable reward signals.Motivated by the insight that human evaluation inherently combines explicit criteria with implicit comparative judgments, we propose Comparative Policy Optimization (CPO). CPO redefines the reward evaluation paradigm by shifting from sample-wise scoring to comparative group-wise scoring.Building on the same principle, we introduce the CharacterArena evaluation framework, which comprises two stages:(1) Contextualized Multi-turn Role-playing Simulation, and (2) Trajectory-level Comparative Evaluation. By operationalizing subjective scoring via objective trajectory comparisons, CharacterArena minimizes contextual bias and enables more robust and fair performance evaluation. Empirical results on CharacterEval, CharacterBench, and CharacterArena confirm that CPO effectively mitigates reward ambiguity and leads to substantial improvements in dialogue quality.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 23 of its 38 references that are in the map. First posted on arXiv 12 Aug 2025.

Builds on

23 of 38 references.

References · 23

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2017Decoupled Weight Decay Regularizationcited by 1,831
  8. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  9. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  10. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 339
  11. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  12. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 239
11 more
  1. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  2. 2023InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviewscited by 179
  3. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domainscited by 156
  4. 2024CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluationcited by 143
  5. 2024Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignmentcited by 104
  6. 2024Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Workscited by 38
  7. 2024CharacterBox: Evaluating the Role-Playing Capabilities of LLMs in Text-Based Virtual Worldscited by 38
  8. 2024Character is Destiny: Can Role-Playing Language Agents Make Persona-Driven Decisions?cited by 34
  9. 2025CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulationcited by 26
  10. 2023BatchEval: Towards Human-like Text Evaluationcited by 17
  11. 2025Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguitycited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026HER: Human-like Reasoning and Reinforcement Learning for LLM Role-playingcited by 10
  2. 2026EmoHarbor: Evaluating Personalized Emotional Support by Simulating the User's Internal Worldcited by 3
  3. 2026Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effectscited by 3
  4. 2026DynSess: Dynamic Session-Level Evaluation and Optimization Framework for Role-Playing Agentscited by 2
  5. 2026Listening to the Echo: User-Reaction Aware Policy Optimization via Scalar-Verbal Hybrid Reinforcement Learningcited by 1
  6. 2026Groupwise Agentic Grading and Advantage Redistribution for Code Agent RLcited by 1
  7. 2026Beyond Overlap Metrics: Rewarding Reasoning and Preferences for Faithful Multi-Role Dialogue Summarizationcited by 0
  8. 2026From Memory to Behavior: A Behavior-Aware Role-Playing Framework for Social Media Influencerscited by 0
  9. 2026TRACER: Trajectory-Aligned Learning for Multi-Turn User Simulationcited by 0
  10. 2026OSPD: On-Policy Self-Distillation for Persona-Consistent Dialoguecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systems
  2. 2026RoleCDE:Benchmarking and Mitigating Role-Alignment Trade-offs in Role-Playing Agents
  3. 2026Character Beyond Speech: Leveraging Role-Playing Evaluation in Audio Large Language Models via Reinforcement Learning
  4. 2026CRPO: Character-centric Group Relative Policy Optimization for Role-aware Reasoning in Role-playing Agents
  5. 2025Understanding Generalization in Role-Playing Models via Information Theory
  6. 2026Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
  7. 2026Reward-Decomposed Reinforcement Learning for Immersive Video Role-Playing
  8. 2025SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agents