mlsanityData to 14 Oct 2026

Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Function

arXiv 2506.03066 ↗First posted 3 Jun 2025cs.LG stat.MLcited by 4

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

The link function, which characterizes the relationship between the preference for two trajectories and their returns, is a crucial component in designing RL algorithms that learn from preference feedback. Most existing methods, both theoretical and empirical, assume that the link function is known (often a logistic function based on the Bradley-Terry model), which is arguably restrictive given the complex nature of preferences, especially those of humans. To avoid mis-specification, this paper studies preference-based RL with an unknown link function and proposes a novel zeroth-order policy optimization algorithm called Sign-SZPO. Unlike typical zeroth-order methods, which rely on the known link function to estimate the value function differences and form a gradient estimator, Sign-SZPO only estimates the sign of the value function difference. It then constructs a parameter update direction that is positively correlated with the true policy gradient, eliminating the need to know the link function exactly. Under mild conditions, Sign-SZPO provably converges to a stationary policy with a polynomial rate in the number of policy iterations and trajectories per iteration. Empirical evaluations further demonstrate the robustness of Sign-SZPO under link function mis-specifications.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 60 of its 89 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

60 of 89 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference25 Sep
  2. 2024Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis11 Jun

Other references · 58

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2021Convex optimizationcited by 2,515
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  9. 2019On the Convergence of Adam and Beyondcited by 841
  10. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  11. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  12. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
38 more
  1. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  2. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  3. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  4. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  6. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  7. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  8. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  9. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  10. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  11. 2023Reward Design with Language Modelscited by 261
  12. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  13. 2017Inverse Reward Designcited by 249
  14. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  15. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  16. 2023Nash Learning from Human Feedbackcited by 224
  17. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 198
  18. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  19. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  20. 2019Sign-OPT: A Query-Efficient Hard-label Adversarial Attackcited by 120
  21. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
  22. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 111
  23. 2017Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agentscited by 107
  24. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  25. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  26. 2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 102
  27. 2023Is RLHF More Difficult than Standard RL?cited by 96
  28. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  29. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimismcited by 69
  30. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  31. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  32. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  33. 2022Models of human preference for learning reward functionscited by 54
  34. 2021A Zeroth-Order Block Coordinate Descent Algorithm for Huge-Scale Black-Box Optimizationcited by 46
  35. 2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46
  36. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  37. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedbackcited by 40
  38. 2023Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oraclescited by 35

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Semiparametric Preference Optimization: Your Language Model is Secretly a Single-Index Modelcited by 2
  2. 2026Objective Decoupling in Social Reinforcement Learning: Recovering Ground Truth from Sycophantic Majoritiescited by 2
  3. 2026Offline Constrained RLHF with Multiple Preference Oraclescited by 1
  4. 2026Efficient Federated RLHF via Zeroth-Order Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling
  2. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  3. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  4. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
  5. 2023Direct Preference-based Policy Optimization without Reward Modeling
  6. 2023Beyond Reward: Offline Preference-guided Policy Optimization
  7. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
  8. 2025Length-Controlled Margin-Based Preference Optimization without Reference Model