mlsanityData to 7 Oct 2026

Active Reward Learning from Online Preferences

arXiv 2302.13507 ↗First posted 27 Feb 2023cs.LG cs.AI cs.ROcited by 12

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Robot policies need to adapt to human preferences and/or new environments. Human experts may have the domain knowledge required to help robots achieve this adaptation. However, existing works often require costly offline re-training on human feedback, and those feedback usually need to be frequent and too complex for the humans to reliably provide. To avoid placing undue burden on human experts and allow quick adaptation in critical real-world situations, we propose designing and sparingly presenting easy-to-answer pairwise action preference queries in an online fashion. Our approach designs queries and determines when to present them to maximize the expected value derived from the queries' information. We demonstrate our approach with experiments in simulation, human user studies, and real robot experiments. In these settings, our approach outperforms baseline techniques while presenting fewer queries to human experts. Experiment videos, code and appendices are found at https://sites.google.com/view/onlineactivepreferences.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 63 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

33 of 63 references.

Closest ancestors · 5

Same direction, newest first

  1. 2021Learning Multimodal Rewards from Rankings27 Sep
  2. 2021Preference-based Learning of Reward Function Features3 Mar
  3. 2020Active Preference-Based Gaussian Process Regression for Reward Learning6 May
  4. 2018DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback28 Oct
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 28

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  4. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  5. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  6. 2017Inverse Reward Designcited by 246
  7. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  8. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  9. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  10. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 98
  11. 2018EnsembleDAgger: A Bayesian Approach to Safe Imitation Learningcited by 96
  12. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
16 more
  1. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  2. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  3. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 67
  4. 2021LazyDAgger: Reducing Context Switching in Interactive Imitation Learningcited by 50
  5. 2015Shared Autonomy via Hindsight Optimizationcited by 46
  6. 2019Batch Active Learning Using Determinantal Point Processescited by 42
  7. 2021Learning Reward Functions from Scale Feedbackcited by 24
  8. 2019Preference-Based Learning for Exoskeleton Gait Optimizationcited by 23
  9. 2020Active Preference Learning using Maximum Regretcited by 21
  10. 2021Learning Human Objectives from Sequences of Physical Correctionscited by 21
  11. 2020ROIAL: Region of Interest Active Learning for Characterizing Exoskeleton Gait Preference Landscapescited by 19
  12. 2018Simplifying Reward Design through Divide-and-Conquercited by 12
  13. 2019Learning an Urban Air Mobility Encounter Model from Expert Preferencescited by 12
  14. 2020BabyAI 1.1cited by 8
  15. 2021Here's What I've Learned: Asking Questions that Reveal Reward Learningcited by 8
  16. 2020Active Reward Learning for Co-Robotic Vision Based Exploration in Bandwidth Limited Environmentscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 3; 2026: 1.

12 papers.

  1. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  2. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  3. 2024Batch Active Learning of Reward Functions from Human Preferencescited by 8
  4. 2025MILE: Model-based Intervention Learningcited by 7
  5. 2025PABBO: Preferential Amortized Black-Box Optimizationcited by 7
  6. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  7. 2023Contextual Bandits and Imitation Learning via Preference-Based Active Queriescited by 2
  8. 2025Robust Reward Alignment via Hypothesis Space Batch Cuttingcited by 2
  9. 2024MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Interventioncited by 1
  10. 2024A Decision-driven Methodology for Designing Uncertainty-aware AI Self-Assessmentcited by 1
  11. 2026Grounding Hierarchical Vision-Language-Action Models Through Explicit Language-Action Alignmentcited by 1
  12. 2023Direct Preference-Based Evolutionary Multi-Objective Optimization with Dueling Banditcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Adaptive Querying for Reward Learning from Human Feedback
  2. 2022Learning from Physical Human Feedback: An Object-Centric One-Shot Adaptation Method
  3. 2022Feedback-efficient Active Preference Learning for Socially Aware Robot Navigation
  4. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning
  5. 2024PREDILECT: Preferences Delineated with Zero-Shot Language-based Reasoning in Reinforcement Learning
  6. 2025Search-Based Credit Assignment for Offline Preference-Based Reinforcement Learning
  7. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  8. 2025Capturing Individual Human Preferences with Reward Features