Direct Diversity Optimization for Diverse Successful Trajectories in Preference Post-Training
On the map: LLM agents and tool use → Reinforcement learning for LLM agents
Abstract (arXiv)
LLM agents for sequential decision tasks are often post-trained with trajectory-level outcome labels, but such labels provide little supervision for preserving multiple successful branches from the same decision state. We study this problem as successful strategy coverage: how broadly a model realizes distinct successful strategies under a fixed rollout budget. We present Direct Diversity Optimization (DDO), an offline post-training method that combines Divergence-Tree Collection (DTC) with the Reference-Relative Target-Odds Objective (RTO). DTC constructs state-aligned branch sets rooted at shared decision states, and RTO trains the model to match reference-relative targets over successful alternatives. DDO achieves the strongest task success and successful strategy coverage among the compared post-training methods across BabyAI, BabaIsAI, and WebShop. It also achieves the highest recovery rate after local action replacement and higher task success and coverage than successful-only imitation and decoding-time diversification controls.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 20 of its 29 references that are in the map. First posted on arXiv 9 Sep 2026.
Builds on
20 of 29 references.
Closest ancestors · 1
Same direction, newest first
Other references · 19
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2020Learning to summarize from human feedbackcited by 2,562
- 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
- 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
- 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
- 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
- 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
- 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
7 more
- 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94
- 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 86
- 2020Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimizationcited by 53
- 2025Diverse Preference Optimizationcited by 48
- 2025Diverse Preference Learning for Capabilities and Alignmentcited by 25
- 2024Baba Is AI: Break the Rules to Beat the Benchmarkcited by 10
- 2024On Extending Direct Preference Optimization to Accommodate Tiescited by 8
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Explicit Trajectory Diversity for RL-Based Post-Training of LLM Agents
- 2025Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity
- 2026When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play
- 2026Beyond Mode Collapse: Distribution Matching for Diverse Reasoning
- 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning
- 2026LLMZero: Discovering Adaptive Training Strategies for RL Post-Training via LLM Agents
- 2026Many Ways to Succeed: Diversity-Driven RL Fine-Tuning for VLA Generalization
- 2023Diverse Policy Optimization for Structured Action Space