mlsanityData to 7 Oct 2026

Discovering Preference Optimization Algorithms with and for Large Language Models

arXiv 2406.08414 ↗First posted 12 Jun 2024cs.LGcited by 38

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Offline preference optimization is a key method for enhancing and controlling the quality of Large Language Model (LLM) outputs. Typically, preference optimization is approached as an offline supervised learning task using manually-crafted convex loss functions. While these methods are based on theoretical insights, they are inherently constrained by human creativity, so the large search space of possible loss functions remains under explored. We address this by performing LLM-driven objective discovery to automatically discover new state-of-the-art preference optimization algorithms without (expert) human intervention. Specifically, we iteratively prompt an LLM to propose and implement new preference optimization loss functions based on previously-evaluated performance metrics. This process leads to the discovery of previously-unknown and performant preference optimization algorithms. The best performing of these we call Discovered Preference Optimization (DiscoPOP), a novel algorithm that adaptively blends logistic and exponential losses. Experiments demonstrate the state-of-the-art performance of DiscoPOP and its successful transfer to held-out tasks.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 52 of its 84 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

52 of 84 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models29 May
  2. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  3. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  4. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  5. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning25 Dec
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct

Other references · 46

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2023Let's Verify Step by Stepcited by 3,598
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  8. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  9. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  10. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  12. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
34 more
  1. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  2. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  3. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  4. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  5. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  6. 2019Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradientscited by 294
  7. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  8. 2024Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Modelcited by 258
  9. 2021Neural Symbolic Regression that Scalescited by 185
  10. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  11. 2022Evolution through Large Modelscited by 131
  12. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
  13. 2022Discovered Policy Optimisationcited by 103
  14. 2021Symbolic Regression via Neural-Guided Genetic Programming Population Seedingcited by 97
  15. 2018Evolved Policy Gradientscited by 86
  16. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 77
  17. 2020Discovering Reinforcement Learning Algorithmscited by 75
  18. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  19. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  20. 2024Large Language Models As Evolution Strategiescited by 57
  21. 2021Evolving Reinforcement Learning Algorithmscited by 48
  22. 2022Discovering Evolution Strategies via Meta-Black-Box Optimizationcited by 44
  23. 2023Discovering Attention-Based Genetic Algorithms via Meta-Black-Box Optimizationcited by 33
  24. 2023L2MAC: Large Language Model Automatic Computer for Extensive Code Generationcited by 30
  25. 2023Deep Generative Symbolic Regressioncited by 27
  26. 2023Vanishing Gradients in Reinforcement Finetuning of Language Modelscited by 26
  27. 2020Meta-learning curiosity algorithmscited by 25
  28. 2022Neural Laplace: Learning diverse classes of differential equations in the Laplace domaincited by 24
  29. 2024Discovering Temporally-Aware Reinforcement Learning Algorithmscited by 18
  30. 2023Discovering General Reinforcement Learning Algorithms with Adversarial Environment Designcited by 16
  31. 2024ODE Discovery for Longitudinal Heterogeneous Treatment Effects Inferencecited by 12
  32. 2023Neural Laplace Control for Continuous-time Delayed Systemscited by 10
  33. 2024Large Language Models as In-context AI Generators for Quality-Diversitycited by 7
  34. 2023Arbitrary Order Meta-Learning with Simple Population-Based Evolutioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 17; 2026: 12.

38 papers.

Sort:

By area:

  1. 2024Automated Design of Agentic Systemscited by 360
  2. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 153
  3. 2026Towards End-to-End Automation of AI Researchcited by 150
  4. 2025AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-benchcited by 65
  5. 2025A Survey of Direct Preference Optimizationcited by 35
  6. 2025MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?cited by 27
  7. 2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 24
  8. 2024Automating the Search for Artificial Life with Foundation Modelscited by 20
  9. 2026Towards Execution-Grounded Automated AI Researchcited by 19
  10. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  11. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 12
  12. 2024Generative Adversarial Reviews: When LLMs Become the Criticcited by 12
26 more
  1. 2025InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusioncited by 11
  2. 2024Automating Traffic Model Enhancement with AI Research Agentcited by 8
  3. 2025Automated Capability Discovery via Foundation Model Self-Explorationcited by 8
  4. 2026Simple Baselines are Competitive with Code Evolutioncited by 6
  5. 2026TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Explorationcited by 6
  6. 2025Foundation Model Self-Play: Open-Ended Strategy Innovation via Foundation Modelscited by 5
  7. 2025How Should We Meta-Learn Reinforcement Learning Algorithms?cited by 5
  8. 2025LLM-Guided Search for Deletion-Correcting Codescited by 4
  9. 2025Automating quantum feature map design via large language modelscited by 4
  10. 2026Mutation Without Variation: Convergence Dynamics in LLM-Driven Program Evolutioncited by 4
  11. 2024A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detectioncited by 3
  12. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimizationcited by 3
  13. 2026Discovering Multiagent Learning Algorithms with Large Language Modelscited by 3
  14. 2026ScienceFlow: A long-horizon agent for ML research, scientific discovery and beyondcited by 3
  15. 2026Recursive self-improvement of AI research agentscited by 3
  16. 2024Meta-Learning Objectives for Preference Optimizationcited by 2
  17. 2024RAGulator: Lightweight Out-of-Context Detectors for Grounded Text Generationcited by 1
  18. 2025EvoVLMA: Evolutionary Vision-Language Model Adaptationcited by 1
  19. 2025GeoEvolve: Automating Geospatial Model Discovery via Multi-Agent Large Language Modelscited by 1
  20. 2026Learning to Orchestrate Agents under Uncertaintycited by 1
  21. 2025ABCDWaveNet: Advancing Robust Road Ponding Detection in Fog through Dynamic Frequency-Spatial Synergycited by 0
  22. 2025On Monotonicity in AI Alignmentcited by 0
  23. 2025Cultural Alien Sampler: Open-ended art generation balancing originality and coherencecited by 0
  24. 2026Displacement-Resistant Extensions of DPO with Nonconvex f-Divergencescited by 0
  25. 2026SOLAR: A Self-Optimizing Open-Ended Autonomous Agent for Lifelong Learning and Continual Adaptationcited by 0
  26. 2026Unbiased Alignment for Large Language Models with Noisy Preferencescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Intrinsic Mutual Information as a Modulator for Preference Optimization
  2. 2025Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning
  3. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization
  4. 2024Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness
  5. 2026Nearly Optimal Active Preference Learning and Its Application to LLM Alignment
  6. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  7. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
  8. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning