mlsanityData to 14 Oct 2026

A Zeroth-Order Paradigm for LLM Preference Alignment

arXiv 2609.19144 ↗First posted 16 Sep 2026cs.CL cs.AI cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a convergence guarantee for its basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. We further introduce online ComPO, which retains the offline comparison mechanism and uses unlabeled policy generations for reverse-KL control relative to a reference policy. Following the coverage perspective of preference fine-tuning, we establish a performance guarantee for a basic constrained scheme under local coverage and in-distribution pairwise reward accuracy. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models demonstrate improvements over existing direct alignment methods, including length-controlled win rates, with pair-level diagnostics providing evidence consistent with mitigating likelihood displacement.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 86 of its 100 references that are in the map. First posted on arXiv 16 Sep 2026.

Builds on

86 of 100 references.

Closest ancestors · 26

Same direction, newest first

  1. 2026Reward-free Alignment for Conflicting Objectives2 Feb
  2. 2025ComPO: Preference Alignment via Comparison Oracles8 May
  3. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment19 Dec
  4. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs10 Oct
  5. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization5 Oct
  6. 2024Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback4 Oct
  7. 2024The Crucial Role of Samplers in Online Direct Preference Optimization29 Sep
  8. 2024Learning Dynamics of LLM Finetuning15 Jul
  9. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coverage3 Jun
  10. 2024MallowsPO: Fine-Tune Your LLM with Preference Dispersions23 May
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  12. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
14 more
  1. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  2. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  3. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  4. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  5. 2024Direct Preference Optimization with an Offset16 Feb
  6. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  7. 2024Noise Contrastive Alignment of Language Models with Explicit Rewards8 Feb
  8. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  9. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  10. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  12. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  13. 2023Preference Ranking Optimization for Human Alignment30 Jun
  14. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
38 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  2. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  4. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  6. 2024Self-Rewarding Language Modelscited by 681
  7. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  8. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  9. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  10. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  11. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  12. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  13. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  14. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  15. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  16. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  17. 2024Iterative Reasoning Preference Optimizationcited by 246
  18. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 198
  19. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  20. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  21. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  22. 2019Sign-OPT: A Query-Efficient Hard-label Adversarial Attackcited by 120
  23. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
  24. 2015An Optimal Algorithm for Bandit and Zero-Order Convex Optimization with Two-Point Feedbackcited by 111
  25. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 111
  26. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  27. 2017Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agentscited by 107
  28. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  29. 2019ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimizationcited by 76
  30. 2017Stochastic Zeroth-order Optimization in High Dimensionscited by 63
  31. 2019Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimizationcited by 47
  32. 2020Zeroth-Order Regularized Optimization (ZORO): Approximately Sparse Gradients and Adaptive Samplingcited by 46
  33. 2016A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Ordercited by 44
  34. 2019Gradientless Descent: High-Dimensional Zeroth-Order Optimizationcited by 43
  35. 2022Gradient-Free Methods for Deterministic and Stochastic Nonsmooth Nonconvex Optimizationcited by 43
  36. 2022Preference Exploration for Efficient Bayesian Optimization with Multiple Outcomescited by 42
  37. 2023Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questionscited by 42
  38. 2020Accelerated Zeroth-Order and First-Order Momentum Methods from Mini to Minimax Optimizationcited by 40

50 most cited of 60 All 60 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  2. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimization
  3. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  4. 2025Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling
  5. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  6. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  7. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
  8. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models