mlsanityData to 14 Oct 2026

Meta-Learning Objectives for Preference Optimization

arXiv 2411.06568 ↗First posted 10 Nov 2024cs.LG cs.AI stat.MLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Evaluating preference optimization (PO) algorithms on LLM alignment is a challenging task that presents prohibitive costs, noise, and several variables like model size and hyper-parameters. In this work, we show that it is possible to gain insights on the efficacy of PO algorithm on simpler benchmarks. We design a diagnostic suite of MuJoCo tasks and datasets, which we use to systematically evaluate PO algorithms, establishing a more controlled and cheaper benchmark. We then propose a novel family of PO algorithms based on mirror descent, which we call Mirror Preference Optimization (MPO). Through evolutionary strategies, we search this class to discover algorithms specialized to specific properties of preference datasets, such as mixed-quality or noisy data. We demonstrate that our discovered PO algorithms outperform all known algorithms in the targeted MuJoCo settings. Finally, based on the insights gained from our MuJoCo experiments, we design a PO algorithm that significantly outperform existing baselines in an LLM alignment task.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 47 references that are in the map. First posted on arXiv 10 Nov 2024.

Builds on

30 of 47 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization18 Jul
  2. 2024Discovering Preference Optimization Algorithms with and for Large Language Models12 Jun
  3. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coverage3 Jun
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  6. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  7. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  8. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  10. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep

Other references · 20

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2024Mixtral of Expertscited by 1,823
  7. 2024Self-Rewarding Language Modelscited by 681
  8. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  9. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  10. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  11. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  12. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
8 more
  1. 2022Discovered Policy Optimisationcited by 104
  2. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  3. 2020Discovering Reinforcement Learning Algorithmscited by 77
  4. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  5. 2022evosax: JAX-based Evolution Strategiescited by 41
  6. 2024Discovering Temporally-Aware Reinforcement Learning Algorithmscited by 18
  7. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
  8. 2021A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learningcited by 12

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Preference Optimization by Estimating the Ratio of the Data Distributioncited by 9
  2. 2025DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  2. 2025HEAL: A Hypothesis-Based Preference-Aware Analysis Framework
  3. 2026Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labels
  4. 2024AIPO: Improving Training Objective for Iterative Preference Optimization
  5. 2024TODO: Enhancing LLM Alignment with Ternary Preferences
  6. 2026EvoPref: Multi-Objective Evolutionary Optimization Discovers Diverse LLM Alignments Beyond Gradient Descent
  7. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization
  8. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimization