mlsanityData to 7 Oct 2026

Reward Learning with Trees: Methods and Evaluation

arXiv 2210.01007 ↗First posted 3 Oct 2022cs.LG cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Recent efforts to learn reward functions from human feedback have tended to use deep neural networks, whose lack of transparency hampers our ability to explain agent behaviour or verify alignment. We explore the merits of learning intrinsically interpretable tree models instead. We develop a recently proposed method for learning reward trees from preference labels, and show it to be broadly competitive with neural networks on challenging high-dimensional tasks, with good robustness to limited or corrupted data. Having found that reward tree learning can be done effectively in complex settings, we then consider why it should be used, demonstrating that the interpretable reward structure gives significant scope for traceability, verification and explanation.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 63 references that are in the map. First posted on arXiv 3 Oct 2022.

Builds on

27 of 63 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Preprocessing Reward Functions for Interpretability25 Mar
  2. 2022Safe Deep RL in 3D Environments using Human Feedback20 Jan
  3. 2021Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions20 Dec
  4. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  5. 2021Information Directed Reward Learning for Reinforcement Learning24 Feb
  6. 2019Learning Human Objectives by Evaluating Hypothetical Behavior5 Dec
  7. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 19

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  3. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  4. 2018Programmatically Interpretable Reinforcement Learningcited by 192
  5. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 176
  6. 2020Explainable Reinforcement Learning: A Surveycited by 139
  7. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  8. 2020Explainability in Deep Reinforcement Learningcited by 96
  9. 2021A Survey on Interpretable Reinforcement Learningcited by 70
  10. 2016Graying the black box: Understanding DQNscited by 68
  11. 2018Toward Interpretable Deep Reinforcement Learning with Linear Model U-Treescited by 65
  12. 2020Quantifying Differences in Reward Functionscited by 62
7 more
  1. 2018Establishing Appropriate Trust via Critical Statescited by 47
  2. 2018Contrastive Explanations for Reinforcement Learning in terms of Expected Consequencescited by 44
  3. 2020Understanding Learned Reward Functionscited by 37
  4. 2019Conservative Q-Improvement: Reinforcement Learning for an Interpretable Decision-Tree Policycited by 29
  5. 2018Object-Oriented Dynamics Predictorcited by 15
  6. 2019Improving User Specifications for Robot Behavior through Active Preference Learning: Framework and Evaluationcited by 7
  7. 2022Summarising and Comparing Agent Dynamics with Contrastive Spatiotemporal Abstractioncited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2.

2 papers.

  1. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  2. 2023Can Differentiable Decision Trees Enable Interpretable Reward Learning from Human Feedback?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025GRAM-R2: Self-Training Generative Foundation Reward Models for Reward Reasoning
  2. 2025GRAM: A Generative Foundation Reward Model for Reward Generalization
  3. 2023Learning Interpretable Models of Aircraft Handling Behaviour by Reinforcement Learning from Human Feedback
  4. 2025Capturing Individual Human Preferences with Reward Features
  5. 2025Reward Models in Deep Reinforcement Learning: A Survey
  6. 2024Reward Generalization in RLHF: A Topological Perspective
  7. 2023On The Fragility of Learned Reward Functions
  8. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards