# MOA: Multi-Objective Alignment for Role-Playing Agents

arXiv: https://arxiv.org/abs/2512.09756 · HTML: https://mlsanity.com/paper/2512.09756/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 40 references that are in the map.
- First posted on arXiv 10 Dec 2025.

## Builds on (most cited first, 29 of 29)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Absolute Zero: Reinforced Self-play Reasoning with Zero Data](https://mlsanity.com/paper/2505.03335/) (2025)
- [Character-LLM: A Trainable Agent for Role-Playing](https://mlsanity.com/paper/2310.10158/) (2023)
- [Demystifying Long Chain-of-Thought Reasoning in LLMs](https://mlsanity.com/paper/2502.03373/) (2025)
- [Learning to Reason under Off-Policy Guidance](https://mlsanity.com/paper/2504.14945/) (2025)
- [On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2302.12095/) (2023)
- [Common 7B Language Models Already Possess Strong Math Capabilities](https://mlsanity.com/paper/2403.04706/) (2024)
- [Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment](https://mlsanity.com/paper/2401.12474/) (2024)
- [Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations](https://mlsanity.com/paper/2306.04618/) (2023)
- [PersonaGym: Evaluating Persona Agents and LLMs](https://mlsanity.com/paper/2407.18416/) (2024)
- [CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models](https://mlsanity.com/paper/2503.22342/) (2025)
- [The Oscars of AI Theater: A Survey on Role-Playing with Language Models](https://mlsanity.com/paper/2407.11484/) (2024)
- [NaturalConv: A Chinese Dialogue Dataset Towards Multi-turn Topic-driven Conversation](https://mlsanity.com/paper/2103.02548/) (2021)
- [OpenCharacter: Training Customizable Role-Playing LLMs with Large-Scale Synthetic Personas](https://mlsanity.com/paper/2501.15427/) (2025)
- [RoleMRC: A Fine-Grained Composite Benchmark for Role-Playing and Instruction-Following](https://mlsanity.com/paper/2502.11387/) (2025)
- [Reasoning Does Not Necessarily Improve Role-Playing Ability](https://mlsanity.com/paper/2502.16940/) (2025)
- [Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoning](https://mlsanity.com/paper/2506.01748/) (2025)
- [RAIDEN-R1: Improving Role-awareness of LLMs via GRPO with Verifiable Reward](https://mlsanity.com/paper/2505.10218/) (2025)
- [Exploring Forgetting in Large Language Model Pre-Training](https://mlsanity.com/paper/2410.17018/) (2024)
- [Entropy Regularizing Activation: Boosting Continuous Control, Large Language Models, and Image Classification with Activation as Entropy Constraints](https://mlsanity.com/paper/2510.08549/) (2025)

## Built on by (most cited first, 5 of 5)

- [MemPO: Self-Memory Policy Optimization for Long-Horizon Agents](https://mlsanity.com/paper/2603.00680/) (2026)
- [HER: Human-like Reasoning and Reinforcement Learning for LLM Role-playing](https://mlsanity.com/paper/2601.21459/) (2026)
- [DREAM: LLM-based Dynamic Role-playing via Event-Aware Memory Graph](https://mlsanity.com/paper/2608.05170/) (2026)
- [CRPO: Character-centric Group Relative Policy Optimization for Role-aware Reasoning in Role-playing Agents](https://mlsanity.com/paper/2605.25511/) (2026)
- [TRACER: Trajectory-Aligned Learning for Multi-Turn User Simulation](https://mlsanity.com/paper/2609.28690/) (2026)

## Abstract (arXiv)

Role-playing agents (RPAs) require balancing multiple objectives, such as instruction following, persona consistency, and stylistic fidelity, which are not always perfectly aligned across different dimensions. While prior work has primarily relied on supervised fine-tuning or reinforcement learning with scalarized rewards, these approaches do not explicitly address the coordination of multiple reward dimensions during optimization. We present MOA (Multi-Objective Alignment), a reinforcement-learning framework that enables multi-dimensional, fine-grained rubric optimization for general RPAs. MOA introduces a novel multi-objective optimization strategy that trains simultaneously on multiple fine-grained rubrics to boost optimization performance. Additionally, to improve both output diversity and generation quality, we employ thought-augmented rollouts with off-policy guidance. Experiments on PersonaGym and RoleMRC show that MOA consistently improves multi-dimensional role-playing performance over supervised and standard RL baselines. Under identical evaluation protocols, an 8B model trained with MOA reaches performance competitive with strong closed-source models across multiple evaluation dimensions. These results suggest that MOA provides a practical framework for training more capable general-purpose role-playing agents.