mlsanityData to 14 Oct 2026

Learning Interpretable Models of Aircraft Handling Behaviour by Reinforcement Learning from Human Feedback

arXiv 2305.16924 ↗First posted 26 May 2023cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We propose a method to capture the handling abilities of fast jet pilots in a software model via reinforcement learning (RL) from human preference feedback. We use pairwise preferences over simulated flight trajectories to learn an interpretable rule-based model called a reward tree, which enables the automated scoring of trajectories alongside an explanatory rationale. We train an RL agent to execute high-quality handling behaviour by using the reward tree as the objective, and thereby generate data for iterative preference collection and further refinement of both tree and agent. Experiments with synthetic preferences show reward trees to be competitive with uninterpretable neural network reward models on quantitative and qualitative evaluations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 74 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

31 of 74 references.

Closest ancestors · 9

Same direction, newest first

  1. 2022Preprocessing Reward Functions for Interpretability25 Mar
  2. 2022Safe Deep RL in 3D Environments using Human Feedback20 Jan
  3. 2021Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions20 Dec
  4. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  5. 2021Information Directed Reward Learning for Reinforcement Learning24 Feb
  6. 2019Learning Human Objectives by Evaluating Hypothetical Behavior5 Dec
  7. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  8. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  9. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 22

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  4. 2018Programmatically Interpretable Reinforcement Learningcited by 193
  5. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 177
  6. 2020Explainable Reinforcement Learning: A Surveycited by 141
  7. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  8. 2020Explainability in Deep Reinforcement Learningcited by 97
  9. 2021A Survey on Interpretable Reinforcement Learningcited by 73
  10. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  11. 2016Graying the black box: Understanding DQNscited by 68
  12. 2018Toward Interpretable Deep Reinforcement Learning with Linear Model U-Treescited by 65
10 more
  1. 2020Quantifying Differences in Reward Functionscited by 63
  2. 2018Establishing Appropriate Trust via Critical Statescited by 48
  3. 2018Contrastive Explanations for Reinforcement Learning in terms of Expected Consequencescited by 44
  4. 2020Understanding Learned Reward Functionscited by 37
  5. 2020Data-Driven Aerospace Engineering: Reframing the Industry with Machine Learningcited by 27
  6. 2022A Survey on Reinforcement Learning in Aviation Applicationscited by 19
  7. 2018Object-Oriented Dynamics Predictorcited by 15
  8. 2019Improving User Specifications for Robot Behavior through Active Preference Learning: Framework and Evaluationcited by 7
  9. 2020Learning Intrinsic Symbolic Rewards in Reinforcement Learningcited by 3
  10. 2022Summarising and Comparing Agent Dynamics with Contrastive Spatiotemporal Abstractioncited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning Acrobatic Flight from Preferences
  2. 2022Reward Learning with Trees: Methods and Evaluation
  3. 2023Can Differentiable Decision Trees Enable Interpretable Reward Learning from Human Feedback?
  4. 2022Relative Behavioral Attributes: Filling the Gap between Symbolic Goal Specification and Reward Learning from Human Preferences
  5. 2023On The Fragility of Learned Reward Functions
  6. 2021Dueling RL: Reinforcement Learning with Trajectory Preferences
  7. 2024Beyond Human Preferences: Exploring Reinforcement Learning Trajectory Evaluation and Improvement through LLMs
  8. 2025Capturing Individual Human Preferences with Reward Features