A Theoretical Framework for Partially Observed Reward-States in RLHF
On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF
Abstract (arXiv)
The growing deployment of reinforcement learning from human feedback (RLHF) calls for a deeper theoretical investigation of its underlying models. The prevalent models of RLHF do not account for neuroscience-backed, partially-observed "internal states" that can affect human feedback, nor do they accommodate intermediate feedback during an interaction. Both of these can be instrumental in speeding up learning and improving alignment. To address these limitations, we model RLHF as reinforcement learning with partially observed reward-states (PORRL). We accommodate two kinds of feedback - cardinal and dueling feedback. We first demonstrate that PORRL subsumes a wide class of RL problems, including traditional RL, RLHF, and reward machines. For cardinal feedback, we present two model-based methods (POR-UCRL, POR-UCBVI). We give both cardinal regret and sample complexity guarantees for the methods, showing that they improve over naive history-summarization. We then discuss the benefits of a model-free method like GOLF with naive history-summarization in settings with recursive internal states and dense intermediate feedback. For this purpose, we define a new history aware version of the Bellman-eluder dimension and give a new guarantee for GOLF in our setting, which can be exponentially sharper in illustrative examples. For dueling feedback, we show that a naive reduction to cardinal feedback fails to achieve sublinear dueling regret. We then present the first explicit reduction that converts guarantees for cardinal regret to dueling regret. In both feedback settings, we show that our models and guarantees generalize and extend existing ones.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 52 references that are in the map. First posted on arXiv 5 Feb 2024.
Builds on
35 of 52 references.
Closest ancestors · 11
Same direction, newest first
- 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
- 2023Is RLHF More Difficult than Standard RL?25 Jun
- 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
- 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning29 May
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
- 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
- 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
- 2019Fine-Tuning Language Models from Human Preferences18 Sep
- 2018Scalable agent alignment via reward modeling: a research direction19 Nov
- 2017Deep reinforcement learning from human preferences12 Jun
Other references · 24
- 2023GPT-4 Technical Reportcited by 18,132
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2023Let's Verify Step by Stepcited by 3,598
- 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
- 2022Solving math word problems with process- and outcome-based feedbackcited by 871
- 2021Unsolved Problems in ML Safetycited by 291
- 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
- 2023AI Alignment: A Comprehensive Surveycited by 256
- 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
- 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
- 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
12 more
- 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
- 2020Sample-Efficient Reinforcement Learning of Undercomplete POMDPscited by 66
- 2022GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyondcited by 45
- 2022PAC Reinforcement Learning for Predictive State Representationscited by 42
- 2021On the Theory of Reinforcement Learning with Once-per-Episode Feedbackcited by 32
- 2022Learning in Observable POMDPs, without Computationally Intractable Oraclescited by 30
- 2022Partially Observable RL with B-Stability: Unified Structural Condition and Sharp Sample-Efficient Algorithmscited by 24
- 2022PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagementcited by 21
- 2022Reinforcement Learning from Partial Observation: Linear Function Approximation with Provable Sample Efficiencycited by 15
- 2023Lower Bounds for Learning in Revealing POMDPscited by 14
- 2022Optimistic MLE -- A Generic Model-based Algorithm for Partially Observable Sequential Decision Makingcited by 6
- 2021Parallelizing Contextual Banditscited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
- 2025Reinforcement Learning from Human Feedback
- 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
- 2025Reward Model Overoptimisation in Iterated RLHF
- 2023The History and Risks of Reinforcement Learning and Human Feedback
- 2024RLHF Workflow: From Reward Modeling to Online RLHF
- 2024Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis
- 2025One Framework to Rule Them All: Unifying RL-Based and RL-Free Methods in RLHF