mlsanityData to 14 Oct 2026

Learning from Preferences and Mixed Demonstrations in General Settings

arXiv 2508.14027 ↗First posted 19 Aug 2025cs.LGcited by 1

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Reinforcement learning is a general method for learning in sequential settings, but it can often be difficult to specify a good reward function when the task is complex. In these cases, preference feedback or expert demonstrations can be used instead. However, existing approaches utilising both together are often ad-hoc, rely on domain-specific properties, or won't scale. We develop a new framing for learning from human data, reward-rational partial orderings over observations, designed to be flexible and scalable. Based on this we introduce a practical algorithm, LEOPARD: Learning Estimated Objectives from Preferences And Ranked Demonstrations. LEOPARD can learn from a broad range of data, including negative demonstrations, to efficiently learn reward functions across a wide range of domains. We find that when a limited amount of preference and demonstration feedback is available, LEOPARD outperforms existing baselines by a significant margin. Furthermore, we use LEOPARD to investigate learning from many types of feedback compared to just a single one, and find that combining feedback types is often beneficial.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 25 of its 41 references that are in the map. First posted on arXiv 19 Aug 2025.

Builds on

25 of 41 references.

References · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  6. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  7. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  8. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  9. 2017Inverse Reward Designcited by 249
  10. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 120
  11. 2016The Off-Switch Gamecited by 97
  12. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
13 more
  1. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  2. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  3. 2021Learning Multimodal Rewards from Rankingscited by 41
  4. 2024Towards Scalable Automated Alignment of LLMs: A Surveycited by 35
  5. 2019Preferences Implicit in the State of the Worldcited by 34
  6. 2016Learning Preferences for Manipulation Tasks from Online Coactive Feedbackcited by 31
  7. 2019Deep Bayesian Reward Learning from Preferencescited by 28
  8. 2020Bayesian Robust Optimization for Imitation Learningcited by 28
  9. 2023ChatGPT: Applications, Opportunities, and Threatscited by 28
  10. 2021Learning Reward Functions from Scale Feedbackcited by 24
  11. 2022Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interactioncited by 19
  12. 2022A Primer on Maximum Causal Entropy Inverse Reinforcement Learningcited by 13
  13. 2021Combining Reward Information from Multiple Sourcescited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Understanding Goal Generalisation in Sequential Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning Kernel-Based MDPs from Episodic Preferential Feedback
  2. 2025Reinforcement Learning from Multi-level and Episodic Human Feedback
  3. 2025Capturing Individual Human Preferences with Reward Features
  4. 2025Search-Based Credit Assignment for Offline Preference-Based Reinforcement Learning
  5. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferences
  6. 2025Reward Learning from Multiple Feedback Types
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  8. 2026LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback