mlsanityData to 7 Oct 2026

SDPO: Segment-Level Direct Preference Optimization for Social Agents

arXiv 2501.01821 ↗First posted 3 Jan 2025cs.AI cs.CLcited by 29

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Social agents powered by large language models (LLMs) can simulate human social behaviors but fall short in handling complex social dialogues. Direct Preference Optimization (DPO) has proven effective in aligning LLM behavior with human preferences across various agent tasks. However, standard DPO focuses solely on individual turns, which limits its effectiveness in multi-turn social interactions. Several DPO-based multi-turn alignment methods with session-level data have shown potential in addressing this problem.While these methods consider multiple turns across entire sessions, they are often overly coarse-grained, introducing training noise, and lack robust theoretical support. To resolve these limitations, we propose Segment-Level Direct Preference Optimization (SDPO), which dynamically select key segments within interactions to optimize multi-turn agent behavior. SDPO minimizes training noise and is grounded in a rigorous theoretical framework. Evaluations on the SOTOPIA benchmark demonstrate that SDPO-tuned agents consistently outperform both existing DPO-based methods and proprietary LLMs like GPT-4o, underscoring SDPO's potential to advance the social intelligence of LLM-based agents. We release our code and data at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/SDPO.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 14 of its 24 references that are in the map. First posted on arXiv 3 Jan 2025.

Builds on

14 of 24 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Direct Multi-Turn Preference Optimization for Language Agents21 Jun

Other references · 13

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  7. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  8. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  9. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 161
  10. 2024Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignmentcited by 104
  11. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 89
  12. 2024SocialBench: Sociality Evaluation of Role-Playing Conversational Agentscited by 48
1 more
  1. 2024Selective Preference Optimization via Token-Level Reward Function Estimationcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 15.

29 papers.

Sort:

By area:

  1. 2025A Survey of Direct Preference Optimizationcited by 35
  2. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 32
  3. 2025Sotopia-RL: Reward Design for Social Intelligencecited by 20
  4. 2025Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuningcited by 18
  5. 2025Adaptive Social Learning via Mode Policy Optimization for Language Agentscited by 15
  6. 2026A Survey on LLM-based Conversational User Simulationcited by 14
  7. 2025Infusing Theory of Mind into Socially Intelligent LLM Agentscited by 7
  8. 2025Pharmacist: Safety Alignment Data Curation for Large Language Models against Harmful Fine-tuningcited by 6
  9. 2025TO-GATE: Clarifying Questions and Summarizing Responses with Trajectory Optimization for Eliciting Human Preferencecited by 5
  10. 2025Fine-grained Video Dubbing Duration Alignment with Segment Supervised Preference Optimizationcited by 3
  11. 2025MASLegalBench: Benchmarking Multi-Agent Systems in Deductive Legal Reasoningcited by 3
  12. 2025The Stackelberg Speaker: Optimizing Persuasive Communication in Social Deduction Gamescited by 3
17 more
  1. 2026SAVOIR: Learning Social Savoir-Faire via Shapley-based Reward Attributioncited by 3
  2. 2025General Exploratory Bonus for Optimistic Exploration in RLHFcited by 2
  3. 2025See or Say Graphs: Agent-Driven Scalable Graph Structure Understanding with Vision-Language Modelscited by 2
  4. 2026Spend Search Where It Pays: Value-Guided Structured Sampling and Optimization for Generative Recommendationcited by 2
  5. 2026DynSess: Dynamic Session-Level Evaluation and Optimization Framework for Role-Playing Agentscited by 2
  6. 2025MATRIX: Multimodal Agent Tuning for Robust Tool-Use Reasoningcited by 1
  7. 2026From Utterance to Vividity: Training Expressive Subtitle Translation LLM via Adaptive Local Preference Optimizationcited by 1
  8. 2026Do Coding Agents Deceive Us? Detecting and Preventing Cheating via Capped Evaluation with Randomized Testscited by 1
  9. 2026Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agentscited by 1
  10. 2026Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challengescited by 1
  11. 2026SocialRL: Refining LLMs' Social Intelligence through Multi-turn Reinforcement Learning and Reward Designcited by 1
  12. 2026Open Problems in Differentiable Social Choice: Learning Mechanisms, Decisions, and Alignmentcited by 0
  13. 2026ALSO: Adversarial Online Strategy Optimization for Social Agentscited by 0
  14. 2026Trajectories That Segment Themselves: Agent-Declared Boundaries as a Training Unitcited by 0
  15. 2026When and What to Teach: Budget-Aware Online Adaptation for Web Agentscited by 0
  16. 2026Turn-level Multiscale Density Ratio Estimation for LLM Agentscited by 0
  17. 2026Towards Communication-Efficient Social Intelligence in Language Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  2. 2025DEPO: Dual-Efficiency Preference Optimization for LLM Agents
  3. 2026PB-GRPO: Learning Socially Adaptive LLM Agents from Persona-Driven Simulation with Preference-Batched GRPO
  4. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimization
  5. 2024Direct Preference Optimization with an Offset
  6. 2026PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs
  7. 2025Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective
  8. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs