mlsanityData to 7 Oct 2026

Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions

arXiv 2112.11230 ↗First posted 20 Dec 2021cs.LG cs.AIcited by 10

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

The potential of reinforcement learning (RL) to deliver aligned and performant agents is partially bottlenecked by the reward engineering problem. One alternative to heuristic trial-and-error is preference-based RL (PbRL), where a reward function is inferred from sparse human feedback. However, prior PbRL methods lack interpretability of the learned reward structure, which hampers the ability to assess robustness and alignment. We propose an online, active preference learning algorithm that constructs reward functions with the intrinsically interpretable, compositional structure of a tree. Using both synthetic and human-provided feedback, we demonstrate sample-efficient learning of tree-structured reward functions in several environments, then harness the enhanced interpretability to explore and debug for alignment.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 42 references that are in the map. First posted on arXiv 20 Dec 2021.

Builds on

15 of 42 references.

Closest ancestors · 3

Same direction, newest first

  1. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  2. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 12

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  3. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  4. 2017Inverse Reward Designcited by 246
  5. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  6. 2021A Survey on Interactive Reinforcement Learning: Design Principles and Open Challengescited by 41
  7. 2020Understanding Learned Reward Functionscited by 37
  8. 2015Upper-Confidence-Bound Algorithms for Active Learning in Multi-Armed Banditscited by 29
  9. 2020TripleTree: A Versatile Interpretable Representation of Black Box Agents and their Environmentscited by 18
  10. 2021APReL: A Library for Active Preference-based Reward Learning Algorithmscited by 14
  11. 2020Interactive Imitation Learning in State-Spacecited by 10
  12. 2015A graph interpretation of the least squares ranking methodcited by 1

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 5; 2024: 2.

10 papers.

  1. 2023Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directionscited by 123
  2. 2021A Survey on Interpretable Reinforcement Learningcited by 70
  3. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  4. 2022Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learningcited by 13
  5. 2024Informativeness of Reward Functions in Reinforcement Learningcited by 3
  6. 2022Reward Learning with Trees: Methods and Evaluationcited by 2
  7. 2023Incorporating Human Flexibility through Reward Preferences in Human-AI Teamingcited by 2
  8. 2024Explaining Learned Reward Functions with Counterfactual Trajectoriescited by 1
  9. 2023Learning Interpretable Models of Aircraft Handling Behaviour by Reinforcement Learning from Human Feedbackcited by 0
  10. 2023Can Differentiable Decision Trees Enable Interpretable Reward Learning from Human Feedback?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
  2. 2025Repairing Reward Functions with Feedback to Mitigate Reward Hacking
  3. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  4. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning
  5. 2023Direct Preference-based Policy Optimization without Reward Modeling
  6. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning
  7. 2025AutoRule: Reasoning Chain-of-thought Extracted Rule-based Rewards Improve Preference Learning
  8. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning