mlsanityData to 7 Oct 2026

Tell me why: Training preferences-based RL with human preferences and step-level explanations

arXiv 2405.14244 ↗First posted 23 May 2024cs.AI cs.HC cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Human-in-the-loop reinforcement learning allows the training of agents through various interfaces, even for non-expert humans. Recently, preference-based methods (PbRL), where the human has to give his preference over two trajectories, increased in popularity since they allow training in domains where more direct feedback is hard to formulate. However, the current PBRL methods have limitations and do not provide humans with an expressive interface for giving feedback. With this work, we propose a new preference-based learning method that provides humans with a more expressive interface to provide their preference over trajectories and a factual explanation (or annotation of why they have this preference). These explanations allow the human to explain what parts of the trajectory are most relevant for the preference. We allow the expression of the explanations over individual trajectory steps. We evaluate our method in various simulations using a simulated human oracle (with realistic restrictions), and our results show that our extended feedback can improve the speed of learning.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

18 of 24 references.

References · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  5. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  6. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  8. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  9. 2017Right for the Right Reasons: Training Differentiable Models by Constraining their Explanationscited by 293
  10. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  11. 2021Improving Deep Learning Interpretability by Saliency Guided Trainingcited by 68
  12. 2020Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentationcited by 42
6 more
  1. 2019Improving performance of deep learning models with axiomatic attribution priors and expected gradientscited by 32
  2. 2018Learning from Richer Human Guidance: Augmenting Comparison-Based Learning with Feature Queriescited by 16
  3. 2023RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedbackcited by 15
  4. 2020Efficiently Guiding Imitation Learning Agents with Human Gazecited by 6
  5. 2021Accelerating the Learning of TAMER with Counterfactual Explanationscited by 3
  6. 2023Iterative Reward Shaping using Human Feedback for Correcting Reward Misspecificationcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024OCALM: Object-Centric Assessment with Language Modelscited by 0