mlsanityData to 14 Oct 2026

Sotopia-RL: Reward Design for Social Intelligence

arXiv 2508.03905 ↗First posted 5 Aug 2025cs.CLcited by 24

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Social intelligence has become a critical capability for large language models (LLMs), enabling them to engage effectively in real-world social tasks such as collaboration and negotiation. Reinforcement learning (RL) is a natural fit for training socially intelligent agents because it allows models to learn sophisticated strategies directly through social interactions without requiring human annotations. However, there are two unique parts about social intelligence tasks: (1) the quality of individual utterances in social interactions is not strictly related to final success; (2) social interactions require multi-dimensional rubrics for success. Therefore, we argue that it is necessary to design rewards for building utterance-level multi-dimensional reward models to facilitate RL training for social intelligence tasks. To address these challenges, we propose Sotopia-RL, a novel framework that refines coarse episode-level feedback into utterance-level, multi-dimensional rewards. Utterance-level credit assignment attributes outcomes to individual utterances, while multi-dimensional rewards capture the full richness of social interactions and reduce reward hacking. Experiments in Sotopia, an open-ended social learning environment, demonstrate that Sotopia-RL achieves state-of-the-art social goal completion scores (7.17 on Sotopia-hard and 8.31 on Sotopia-full), significantly outperforming existing approaches. Ablation studies confirm the necessity of both utterance-level credit assignment and multi-dimensional reward design for RL training.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 61 references that are in the map. First posted on arXiv 5 Aug 2025.

Builds on

48 of 61 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Information-Theoretic Reward Decomposition for Generalizable RLHF8 Apr
  2. 2024Language Models as Efficient Reward Function Searchers for Custom-Environment Multi-Objective Reinforcement4 Sep
  3. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  4. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 43

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  10. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  11. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  12. 2023Large Language Models are not Fair Evaluatorscited by 1,085
31 more
  1. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  2. 2020Aligning AI With Shared Human Valuescited by 723
  3. 2025Process Reinforcement through Implicit Rewardscited by 504
  4. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
  5. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  6. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
  7. 2023Reward Design with Language Modelscited by 261
  8. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
  9. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  10. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domainscited by 156
  11. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 124
  12. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 93
  13. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 90
  14. 2025Process Reward Models for LLM Agents: Practical Framework and Directionscited by 85
  15. 2024Embodied LLM Agents Learn to Cooperate in Organized Teamscited by 82
  16. 2023Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agentscited by 77
  17. 2023Automating Customer Service using LangChain: Building custom open-source GPT Chatbot for organizationscited by 61
  18. 2024Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulationcited by 46
  19. 2020Emergent Social Learning via Multi-agent Reinforcement Learningcited by 44
  20. 2024Enhancing LLM-Based Feedback: Insights from Intelligent Tutoring Systems and the Learning Sciencescited by 37
  21. 2024InterPreT: Interactive Predicate Learning from Language Feedback for Generalizable Task Planningcited by 31
  22. 2025Language Model Personalization via Reward Factorizationcited by 31
  23. 2025SDPO: Segment-Level Direct Preference Optimization for Social Agentscited by 30
  24. 2024Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questionscited by 22
  25. 2022Empathetic Dialogue Generation via Sensitive Emotion Recognition and Sensible Knowledge Selectioncited by 19
  26. 2025SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agentscited by 16
  27. 2024Dialogue Action Tokens: Steering Language Models in Goal-Directed Dialogue with a Multi-Turn Plannercited by 13
  28. 2025SocialEval: Evaluating Social Intelligence of Large Language Modelscited by 12
  29. 2020Utterance-level Dialogue Understanding: An Empirical Studycited by 10
  30. 2025EPO: Explicit Policy Optimization for Strategic Reasoning in LLMs via Reinforcement Learningcited by 8
  31. 2025UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimalitycited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 20.

24 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2026CooperBench: Why Coding Agents Cannot be Your Teammates Yetcited by 23
  3. 2026Reward Modeling from Natural Language Human Feedbackcited by 11
  4. 2025Infusing Theory of Mind into Socially Intelligent LLM Agentscited by 8
  5. 2026On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agentscited by 6
  6. 2026One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RLcited by 6
  7. 2025Reducing Belief Deviation in Reinforcement Learning for Active Reasoningcited by 4
  8. 2026Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interactioncited by 4
  9. 2026SAVOIR: Learning Social Savoir-Faire via Shapley-based Reward Attributioncited by 4
  10. 2026Reinforcing Human Behavior Simulation via Verbal Feedbackcited by 4
  11. 2026Proactive Service Agents: A Unified Decision Framework, Methods, and Evaluationcited by 2
  12. 2026Momentum-Coupled Rubric Adaptation for Detailed Image Captioningcited by 2
12 more
  1. 2026Cliff: Learning Process Rewards from the First Mistakecited by 1
  2. 2026SocialRL: Refining LLMs' Social Intelligence through Multi-turn Reinforcement Learning and Reward Designcited by 1
  3. 2025ToMPO: Training LLM Strategic Decision Making from a Multi-Agent Perspectivecited by 0
  4. 2026ALSO: Adversarial Online Strategy Optimization for Social Agentscited by 0
  5. 2026CRPO: Character-centric Group Relative Policy Optimization for Role-aware Reasoning in Role-playing Agentscited by 0
  6. 2026Enhancing Social Intelligence in LLMs with Hierarchical Reasoning and Utterance-Level Goal Rewardingcited by 0
  7. 2026From Passive Delegates to Strategic Negotiators: Reinforcing Social Reasoning in Small Language Models with SocialRLcited by 0
  8. 2026Harness-Search: Guiding Long-Horizon Search through Multi-Agent Coordinationcited by 0
  9. 2026Learning to Simulate Individuals from Macro Social Signalscited by 0
  10. 2026RewardWeaver: Long-Horizon Interactive Learning for Language Agents via Self-Evolving Reward Adaptationcited by 0
  11. 2026Understanding and Reducing Misalignment in Persuasion Optimizationcited by 0
  12. 2026What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worldscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Social-R1: Towards Human-like Social Reasoning in LLMs
  2. 2025LIFELONG SOTOPIA: Evaluating Social Intelligence of Language Agents Over Lifelong Social Interactions
  3. 2026One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligence
  4. 2025SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning
  5. 2024Self-Evolved Reward Learning for LLMs
  6. 2024Towards Socially and Morally Aware RL agent: Reward Design With LLM
  7. 2026From Solo to Social Learning: Characterizing Recursive Social Improvement in LLMs
  8. 2025RLNVR: Reinforcement Learning from Non-Verified Real-World Rewards