mlsanityData to 14 Oct 2026

GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization

arXiv 2604.15602 ↗First posted 17 Apr 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference optimization is widely used to align Large Language Models (LLMs) with preference feedback. However, most existing methods train on a single positive-negative pair per prompt, discarding additional supervision available in preference datasets that typically contain multiple candidate responses. Motivated by this limitation, recent work explores group-wise preference optimization, which jointly contrasts multiple responses for the same prompt, but its empirical behavior and scalability remain underexplored due to the memory overhead of group-coupled objectives. In this work, we present a unified empirical and systems study of group-wise preference optimization and develop a memory-efficient implementation for group-coupled objectives. By instantiating first-order linearization with objective-specific per-response coefficients, our implementation preserves first-order gradients while decoupling samples during backpropagation, substantially reducing peak memory usage and enabling scalable training with larger groups. Across offline and online settings, we show that leveraging multiple responses consistently outperforms single-pair training. Furthermore, incorporating a negative log-likelihood (NLL) term on positive responses is critical for both performance gains and training stability.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 46 references that are in the map. First posted on arXiv 17 Apr 2026.

Builds on

33 of 46 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts5 Dec
  2. 2024Preference Optimization with Multi-Sample Comparisons16 Oct
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  4. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  5. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  6. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  7. 2023Preference Ranking Optimization for Human Alignment30 Jun
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2023Let's Verify Step by Stepcited by 3,789
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  11. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
13 more
  1. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  3. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  4. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  5. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  6. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  7. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 201
  8. 2025Generalizing Verifiable Instruction Followingcited by 194
  9. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 112
  10. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  11. 2024On Softmax Direct Preference Optimization for Recommendationcited by 64
  12. 2025Listwise Preference Alignment Optimization for Tail Item Recommendationcited by 2
  13. 2026RankGR: Rank-Enhanced Generative Retrieval with Listwise Direct Preference Optimization in Recommendationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Models
  2. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  3. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  4. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
  5. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization
  6. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
  7. 2025AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Models
  8. 2026DGPO: Beyond Pairwise Preferences with Directional Consistent Groupwise Optimization