mlsanityData to 7 Oct 2026

Reinforcement Learning from Multi-level and Episodic Human Feedback

arXiv 2504.14732 ↗First posted 20 Apr 2025cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Designing an effective reward function has long been a challenge in reinforcement learning, particularly for complex tasks in unstructured environments. To address this, various learning paradigms have emerged that leverage different forms of human input to specify or refine the reward function. Reinforcement learning from human feedback is a prominent approach that utilizes human comparative feedback, expressed as a preference for one behavior over another, to tackle this problem. In contrast to comparative feedback, we explore multi-level human feedback, which is provided in the form of a score at the end of each episode. This type of feedback offers more coarse but informative signals about the underlying reward function than binary feedback. Additionally, it can handle non-Markovian rewards, as it is based on the evaluation of an entire episode. We propose an algorithm to efficiently learn both the reward function and the optimal policy from this form of feedback. Moreover, we show that the proposed algorithm achieves sublinear regret and demonstrate its empirical effectiveness through extensive simulations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 55 references that are in the map. First posted on arXiv 20 Apr 2025.

Builds on

34 of 55 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Is RLHF More Difficult than Standard RL?25 Jun
  2. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning29 May
  3. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  4. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  5. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  5. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  6. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  7. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  8. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  9. 2020Improved Optimistic Algorithms for Logistic Banditscited by 117
  10. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  11. 2020Provably Efficient Reinforcement Learning for Discounted MDPs with Feature Mappingcited by 102
  12. 2015Contextual Dueling Banditscited by 101
16 more
  1. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  2. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  3. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  4. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  5. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  6. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  7. 2020Reinforcement Learning with Trajectory Feedbackcited by 40
  8. 2016On Lower Bounds for Regret in Reinforcement Learningcited by 39
  9. 2019Learning to Generalize from Sparse and Underspecified Rewardscited by 39
  10. 2021On the Theory of Reinforcement Learning with Once-per-Episode Feedbackcited by 32
  11. 2013Learning Trajectory Preferences for Manipulators via Iterative Improvementcited by 31
  12. 2020Self-Concordant Analysis of Generalized Linear Bandits with Forgettingcited by 29
  13. 2016Learning Dynamic Robot-to-Human Object Handover from Human Feedbackcited by 19
  14. 2020Regret Bounds for Discounted MDPscited by 19
  15. 2024Inverse Reinforcement Learning by Estimating Expertise of Demonstratorscited by 11
  16. 2020Towards Tractable Optimism in Model-Based Reinforcement Learningcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  2. 2025Multi-Task Reward Learning from Human Ratings
  3. 2026Learning Kernel-Based MDPs from Episodic Preferential Feedback
  4. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  5. 2025Reward Learning from Multiple Feedback Types
  6. 2025Reinforcement Learning from Human Feedback
  7. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  8. 2024A Theoretical Framework for Partially Observed Reward-States in RLHF