# HER: Human-like Reasoning and Reinforcement Learning for LLM Role-playing

arXiv: https://arxiv.org/abs/2601.21459 · HTML: https://mlsanity.com/paper/2601.21459/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 26 of 26)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [Recursively Summarizing Books with Human Feedback](https://mlsanity.com/paper/2109.10862/) (2021)
- [MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention](https://mlsanity.com/paper/2506.13585/) (2025)
- [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://mlsanity.com/paper/2404.18231/) (2024)
- [RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models](https://mlsanity.com/paper/2310.00746/) (2023)
- [CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation](https://mlsanity.com/paper/2401.01275/) (2024)
- [BooookScore: A systematic exploration of book-length summarization in the era of LLMs](https://mlsanity.com/paper/2310.00785/) (2023)
- [ChatHaruhi: Reviving Anime Character in Reality via Large Language Model](https://mlsanity.com/paper/2308.09597/) (2023)
- [CharacterGLM: Customizing Chinese Conversational AI Characters with Large Language Models](https://mlsanity.com/paper/2311.16832/) (2023)
- [Character is Destiny: Can Role-Playing Language Agents Make Persona-Driven Decisions?](https://mlsanity.com/paper/2404.12138/) (2024)
- [OpenCharacter: Training Customizable Role-Playing LLMs with Large-Scale Synthetic Personas](https://mlsanity.com/paper/2501.15427/) (2025)
- [RoleEval: A Bilingual Role Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2312.16132/) (2023)
- [MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents](https://mlsanity.com/paper/2408.04203/) (2024)
- [Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoning](https://mlsanity.com/paper/2506.01748/) (2025)
- [RAIDEN-R1: Improving Role-awareness of LLMs via GRPO with Verifiable Reward](https://mlsanity.com/paper/2505.10218/) (2025)
- [CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization](https://mlsanity.com/paper/2508.09074/) (2025)
- [CogDual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based Rewards](https://mlsanity.com/paper/2507.17147/) (2025)
- [Echo-N1: Affective RL Frontier](https://mlsanity.com/paper/2512.00344/) (2025)
- [MOA: Multi-Objective Alignment for Role-Playing Agents](https://mlsanity.com/paper/2512.09756/) (2025)

## Built on by (most cited first, 10 of 10)

- [RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation](https://mlsanity.com/paper/2605.20204/) (2026)
- [Reinforcing Human Behavior Simulation via Verbal Feedback](https://mlsanity.com/paper/2605.20506/) (2026)
- [DynSess: Dynamic Session-Level Evaluation and Optimization Framework for Role-Playing Agents](https://mlsanity.com/paper/2605.29256/) (2026)
- [ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time?](https://mlsanity.com/paper/2606.05553/) (2026)
- [Individual Turing Test: A Case Study of LLM-based Simulation Using Longitudinal Personal Data](https://mlsanity.com/paper/2603.01289/) (2026)
- [Agentopia: Long-Term Life Simulation and Learning in Agent Societies](https://mlsanity.com/paper/2606.07513/) (2026)
- [Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization](https://mlsanity.com/paper/2606.27025/) (2026)
- [EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learning](https://mlsanity.com/paper/2609.12459/) (2026)
- [Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation](https://mlsanity.com/paper/2607.27816/) (2026)
- [PersMem: Internalizing Personality into Dual-Pathway Memory for LLM Agents](https://mlsanity.com/paper/2609.34372/) (2026)

## Abstract (arXiv)

LLM role-playing, i.e., using LLMs to simulate specific personas, has emerged as a key capability in various applications, such as companionship, content creation and digital games. While current models effectively capture character tones and knowledge, simulating the inner thoughts behind their behaviors remains a challenge. Towards cognitive simulation in LLM role-play, previous efforts mainly suffer from two deficiencies: lacking data with high-quality reasoning traces, and lacking reliable reward signals aligned with human preferences. In this paper, we propose HER, a unified framework for cognitive-level persona simulation. HER introduces dual-layer thinking, which distinguishes characters' first-person thinking from LLMs' third-person thinking. To bridge these gaps, we curate reasoning-augmented role-playing data via reverse engineering, and construct human-aligned principles and reward models. Leveraging these resources, we train HER models based on Qwen3-32B via supervised and reinforcement learning. Extensive experiments validate the effectiveness of our approach. Notably, our models significantly outperform the Qwen3-32B baseline, achieving a 30.26 improvement on the CoSER benchmark and a 14.97% gain on the Minimax Role-Play Bench. Our datasets, principles, and models are released to facilitate future research.