mlsanityData to 7 Oct 2026

DGPO: Beyond Pairwise Preferences with Directional Consistent Groupwise Optimization

arXiv 2605.10863 ↗First posted 11 May 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Although Large Language Models (LLMs) have made remarkable progress, current preference optimization methods still struggle to align directional consistency while preserving reasoning diversity. To address this limitation, we propose Directional-Groupwise Preference Optimization (DGPO), a lightweight framework that aggregates supervision signals at the group level and explicitly models direction-aware alignment through multi-candidate comparisons. DGPO organizes forward and reverse question-answer instances into structured sets and optimizes a margin-based likelihood objective that separates coherent reasoning paths from inconsistent alternatives. This group-wise formulation captures richer relative information than pairwise objectives and reinforces consistency across diverse reasoning pathways. Empirical results show that our constructed reverse data yields a 3.2% average improvement across five benchmarks, while DGPO further delivers consistent gains across multiple datasets and model families, achieving average accuracy improvements of up to 3.6%.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 48 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

31 of 48 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Robust Preference Optimization via Dynamic Target Margins4 Jun
  2. 2024β-DPO: Direct Preference Optimization with Dynamic β11 Jul
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  4. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 26

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  9. 2025s1: Simple test-time scalingcited by 1,311
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  11. 2025LIMO: Less is More for Reasoningcited by 488
  12. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
14 more
  1. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  2. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 182
  3. 2024OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modelingcited by 66
  4. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learningcited by 63
  5. 2025Atom of Thoughts for Markov LLM Test-Time Scalingcited by 54
  6. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 53
  7. 2023RCOT: Detecting and Rectifying Factual Inconsistency in Reasoning by Reversing Chain-of-Thoughtcited by 47
  8. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Explorationcited by 40
  9. 2023Forward-Backward Reasoning in Large Language Models for Mathematical Verificationcited by 39
  10. 2024Reverse Thinking Makes LLMs Stronger Reasonerscited by 29
  11. 2025Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failurecited by 13
  12. 2025Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweightingcited by 6
  13. 2025Reason from Future: Reverse Thought Chain Enhances LLM Reasoningcited by 5
  14. 2025When Inverse Data Outperforms: Exploring the Pitfalls of Mixed Data in Multi-Stage Fine-Tuningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization
  2. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks
  3. 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts
  4. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Models
  5. 2024MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization
  6. 2026HiPO: Hierarchical Preference Optimization for Adaptive Reasoning in LLMs
  7. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  8. 2025InfoPO: On Mutual Information Maximization for Large Language Model Alignment