mlsanityData to 14 Oct 2026

Multi-Response Preference Optimization with Augmented Ranking Dataset

arXiv 2412.07812 ↗First posted 10 Dec 2024cs.CL cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have been remarkable, with new models consistently surpassing their predecessors. These advancements are underpinned by extensive research on various training mechanisms. Among these, Preference Optimization has played a significant role in improving the performance of LLMs by incorporating human preferences into the training process. However, constructing preference optimization datasets is challenging and the optimization process is highly sensitive to the dataset quality. In this study, we propose a novel approach to augment Preference Optimization datasets. Additionally, we introduce a Multi-response-based Preference Optimization training method that enables the simultaneous learning of multiple responses.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 18 of its 26 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

18 of 26 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Aligning Language Models with Preferences through f-divergence Minimization16 Feb
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2023Mistral 7Bcited by 3,164
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  12. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
4 more
  1. 2024Self-Rewarding Language Modelscited by 681
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  3. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  4. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 82

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  2. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs
  3. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
  4. 2024A Survey on Human Preference Learning for Large Language Models
  5. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  6. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization
  7. 2026GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization
  8. 2024TSO: Self-Training with Scaled Preference Optimization