mlsanityData to 7 Oct 2026

Conditions on Preference Relations that Guarantee the Existence of Optimal Policies

arXiv 2311.01990 ↗First posted 3 Nov 2023cs.LGcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Learning from Preferential Feedback (LfPF) plays an essential role in training Large Language Models, as well as certain types of interactive learning agents. However, a substantial gap exists between the theory and application of LfPF algorithms. Current results guaranteeing the existence of optimal policies in LfPF problems assume that both the preferences and transition dynamics are determined by a Markov Decision Process. We introduce the Direct Preference Process, a new framework for analyzing LfPF problems in partially-observable, non-Markovian environments. Within this framework, we establish conditions that guarantee the existence of optimal policies by considering the ordinal structure of the preferences. We show that a decision-making problem can have optimal policies -- that are characterized by recursive optimality equations -- even when no reward function can express the learning goal. These findings underline the need to explore preference-based learning strategies which do not assume that preferences are generated by reward.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 61 references that are in the map. First posted on arXiv 3 Nov 2023.

Builds on

26 of 61 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2023Beyond Reward: Offline Preference-guided Policy Optimization25 May
  5. 2023Direct Preference-based Policy Optimization without Reward Modeling30 Jan
  6. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  7. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  8. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  9. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 17

  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2016Deep Exploration via Bootstrapped DQNcited by 518
  4. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  5. 2023A Definition of Continual Reinforcement Learningcited by 98
  6. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  7. 2021On the Expressivity of Markov Rewardcited by 71
  8. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  9. 2021Reinforcement Learning, Bit by Bitcited by 59
  10. 2021Simple Agent, Complex Environment: Efficient Reinforcement Learning with Agent Statescited by 42
  11. 2022Settling the Reward Hypothesiscited by 33
  12. 2021On the Theory of Reinforcement Learning with Once-per-Episode Feedbackcited by 32
5 more
  1. 2020Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learningcited by 21
  2. 2020Quantifying Hypothesis Space Misspecification in Learning from Human-Robot Demonstrations and Physical Correctionscited by 19
  3. 2023Provably Feedback-Efficient Reinforcement Learning via Active Reward Learningcited by 15
  4. 2022Utility Theory for Sequential Decision Makingcited by 8
  5. 2021Abstractions of General Reinforcement Learningcited by 5

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024A Black Swan Hypothesis: The Role of Human Irrationality in AI Safetycited by 3
  2. 2025Beyond Scalar Rewards: An Axiomatic Framework for Lexicographic MDPscited by 2
  3. 2025Goals and the Structure of Experiencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  3. 2025Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Options
  4. 2025Greedy Sampling Is Provably Efficient for RLHF
  5. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences
  6. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  7. 2026A Regret Minimization Framework on Preference Learning in Large Language Models
  8. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences