mlsanityData to 7 Oct 2026

Boosting Feedback Efficiency of Interactive Reinforcement Learning by Adaptive Learning from Scores

arXiv 2307.05405 ↗First posted 11 Jul 2023cs.RO cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Interactive reinforcement learning has shown promise in learning complex robotic tasks. However, the process can be human-intensive due to the requirement of a large amount of interactive feedback. This paper presents a new method that uses scores provided by humans instead of pairwise preferences to improve the feedback efficiency of interactive reinforcement learning. Our key insight is that scores can yield significantly more data than pairwise preferences. Specifically, we require a teacher to interactively score the full trajectories of an agent to train a behavioral policy in a sparse reward environment. To avoid unstable scores given by humans negatively impacting the training process, we propose an adaptive learning scheme. This enables the learning paradigm to be insensitive to imperfect or unreliable scores. We extensively evaluate our method for robotic locomotion and manipulation tasks. The results show that the proposed method can efficiently learn near-optimal policies by adaptive learning from scores while requiring less feedback compared to pairwise preference learning methods. The source codes are publicly available at https://github.com/SSKKai/Interactive-Scoring-IRL.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 48 references that are in the map. First posted on arXiv 11 Jul 2023.

Builds on

25 of 48 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning24 May
  2. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  3. 2021Learning Multimodal Rewards from Rankings27 Sep
  4. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 20

  1. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2016OpenAI Gymcited by 1,684
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  5. 2015Prioritized Experience Replaycited by 1,070
  6. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  7. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  8. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  9. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  10. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  11. 2017Inverse Reward Designcited by 246
  12. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
8 more
  1. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  2. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  3. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  4. 2020Multi-expert learning of adaptive legged locomotioncited by 62
  5. 2021Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Reviewcited by 34
  6. 2021Correct Me if I am Wrong: Interactive Learning for Robotic Manipulationcited by 24
  7. 2022Feedback-efficient Active Preference Learning for Socially Aware Robot Navigationcited by 6
  8. 2021A Study on Dense and Sparse (Visual) Rewards in Robot Policy Learningcited by 3

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Grower-in-the-Loop Interactive Reinforcement Learning for Greenhouse Climate Controlcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Adaptive Querying for Reward Learning from Human Feedback
  2. 2023From "Thumbs Up" to "10 out of 10": Reconsidering Scalar Feedback in Interactive Reinforcement Learning
  3. 2023A State Augmentation based approach to Reinforcement Learning from Human Preferences
  4. 2025Reinforcement Learning from Multi-level and Episodic Human Feedback
  5. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  6. 2026Reward-DAgger: Robot-Gated Interactive Imitation Learning with General-Purpose Progress-Based Reward Models
  7. 2026Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policies
  8. 2026Coherent Off-Policy Improvement of Large Behavior Models with Learned Rewards