mlsanityData to 7 Oct 2026

From Passive Delegates to Strategic Negotiators: Reinforcing Social Reasoning in Small Language Models with SocialRL

arXiv 2608.13787 ↗First posted 13 Aug 2026cs.AI cs.CL cs.LGcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

AI agents increasingly act on their users' behalf, handling tasks such as scheduling meetings, comparing offers, and haggling over prices. These principal-driven tasks routinely place the agent across from a counterpart (another user's agent, a seller, a recruiter) whose goals may conflict with its principal's. Yet the dispositions that make an assistant pleasant can make it a poor delegate: a friendly, helpful frontier model may disclose its principal's private information unprompted and concede at the first sign of resistance. We present SocialRL, a general recipe that trains social reasoning directly, and apply it to a 4B model across six domains: Deal-or-No-Deal, CaSiNo, Craigslist, Job Interview, Calendar, and Marketplace. Every domain is trained in-domain under the same recipe, and every policy is evaluated on all six. We find that (1) in-domain training reaches the frontier: on held-out scenarios the 4B matches or exceeds the GPT-5 family per domain, closing 73-122% of the baseline-to-frontier gap on the negotiation games, with 78% of buyer openings anchoring below target versus 3% untrained; (2) cross-domain transfer follows game structure: structurally paired games lift each other, a broad multi-issue donor lifts nearly all domains, and structurally isolated games transfer nothing; (3) guided by this transfer structure, two strategies, cascade RL and multi-teacher on-policy distillation (OPD), consolidate the per-domain specialists into a single unified 4B that reaches 0.627 average utility across all six environments, matching or exceeding GPT-4.1 (0.625), GPT-5.1 (0.619), and GPT-5.2 (0.613); (4) an explicit theory-of-mind scaffold helps only through training: distilling the ToM trace, rather than actions alone, lifts utility on every environment and generalizes better across them, and of the two ToM skills, only next-action prediction predicts negotiation outcomes.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 45 of its 56 references that are in the map. First posted on arXiv 13 Aug 2026.

Builds on

45 of 56 references.

References · 45

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  4. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  6. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  7. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  8. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  9. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  10. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
  11. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  12. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
33 more
  1. 2018Decoupling Strategy and Generation in Negotiation Dialoguescited by 156
  2. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learningcited by 94
  3. 2024How Well Can LLMs Negotiate? NegotiationArena Platform and Analysiscited by 93
  4. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 89
  5. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 83
  6. 2025Authenticated Delegation and Authorized AI Agentscited by 63
  7. 2021CaSiNo: A Corpus of Campsite Negotiation Dialogues for Automatic Negotiation Systemscited by 60
  8. 2026MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Trainingcited by 60
  9. 2024Game-theoretic LLM: Agent Workflow for Negotiation Gamescited by 55
  10. 2026Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learningcited by 51
  11. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 45
  12. 2025Fara-7B: An Efficient Agentic Model for Computer Usecited by 34
  13. 2025MAGPIE: A benchmark for Multi-AGent contextual PrIvacy Evaluationcited by 27
  14. 2026EvoTool: Self-Evolving Tool-Use Policy Optimization in LLM Agents via Blame-Aware Mutation and Diversity-Aware Selectioncited by 22
  15. 2025Sotopia-RL: Reward Design for Social Intelligencecited by 20
  16. 2025What Is Your AI Agent Buying? Evaluation, Biases, Model Dependence, & Emerging Implications for Agentic E-Commercecited by 18
  17. 2025Towards Dynamic Theory of Mind: Evaluating LLM Adaptation to Temporal Evolution of Human Statescited by 17
  18. 2024Efficacy of Language Model Self-Play in Non-Zero-Sum Gamescited by 14
  19. 2025The Automated but Risky Game: Modeling and Benchmarking Agent-to-Agent Negotiations and Transactions in Consumer Marketscited by 12
  20. 2025Infusing Theory of Mind into Socially Intelligent LLM Agentscited by 7
  21. 2026One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligencecited by 7
  22. 2026CalBench: Evaluating Coordination-Privacy Trade-offs in Multi-Agent LLMscited by 6
  23. 2026TERMS-Bench: Diagnosing LLM Negotiation Agents Beyond Deal Ratecited by 6
  24. 2026Evaluating Theory of Mind and Internal Beliefs in LLM-Based Multi-Agent Systemscited by 5
  25. 2024Cascading Reinforcement Learningcited by 4
  26. 2026Instructing LLMs to Negotiate using Reinforcement Learning with Verifiable Rewardscited by 4
  27. 2026AgentLTL: A Trace-Verification Framework for Measuring, Enforcing, and Training Procedural Compliance in Tool-Using LLM Agentscited by 4
  28. 2026Learning to Negotiate: Multi-Agent Deliberation for Collective Value Alignment in LLMscited by 3
  29. 2026Adaptive Theory of Mind for LLM-based Multi-Agent Coordinationcited by 3
  30. 2026Training Language Models for Bilateral Trade with Private Informationcited by 3
  31. 2026Poly-OPD: Heterogeneous Multi-Teacher On-Policy Distillation for Capability-Selectable Flow Modelscited by 2
  32. 2026GIFT: Games as Informal Training for Generalizable LLMscited by 1
  33. 2026PieArena: Ranking and Profiling Language Agents in Realistic Negotiation Scenarioscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025The Price of Thought: A Multilingual Analysis of Reasoning, Performance, and Cost of Negotiation in Large Language Models
  2. 2025SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agents
  3. 2024AgreeMate: Teaching LLMs to Haggle
  4. 2025GTAlign: Game-Theoretic Alignment of LLM Assistants for Social Welfare
  5. 2026SocialRL: Refining LLMs' Social Intelligence through Multi-turn Reinforcement Learning and Reward Design
  6. 2026Do Small Language Models Learn to Negotiate? A Controlled Scaling Study of RL-Trained Sellers
  7. 2026Choose Your Agent: Tradeoffs in Adopting AI Advisors, Coaches, and Delegates in Multi-Party Negotiation
  8. 2025LLM Agents for Bargaining with Utility-based Feedback