# APReL: A Library for Active Preference-based Reward Learning Algorithms

arXiv: https://arxiv.org/abs/2108.07259 · HTML: https://mlsanity.com/paper/2108.07259/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 16 Aug 2021.

## Builds on (most cited first, 19 of 19)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [Third-Person Imitation Learning](https://mlsanity.com/paper/1703.01703/) (2017)
- [Batch Active Preference-Based Learning of Reward Functions](https://mlsanity.com/paper/1810.04303/) (2018)
- [Active Preference-Based Gaussian Process Regression for Reward Learning](https://mlsanity.com/paper/2005.02575/) (2020)
- [Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences](https://mlsanity.com/paper/2006.14091/) (2020)
- [Asking Easy Questions: A User-Friendly Approach to Active Reward Learning](https://mlsanity.com/paper/1910.04365/) (2019)
- [Learning Reward Functions by Integrating Human Demonstrations and Preferences](https://mlsanity.com/paper/1906.08928/) (2019)
- [Batch Active Learning Using Determinantal Point Processes](https://mlsanity.com/paper/1906.07975/) (2019)
- [Preference-Based Learning for Exoskeleton Gait Optimization](https://mlsanity.com/paper/1909.12316/) (2019)
- [Active Preference Learning using Maximum Regret](https://mlsanity.com/paper/2005.04067/) (2020)
- [Learning Human Objectives from Sequences of Physical Corrections](https://mlsanity.com/paper/2104.00078/) (2021)
- [ROIAL: Region of Interest Active Learning for Characterizing Exoskeleton Gait Preference Landscapes](https://mlsanity.com/paper/2011.04812/) (2020)
- [The Green Choice: Learning and Influencing Human Decisions on Shared Roads](https://mlsanity.com/paper/1904.02209/) (2019)
- [Learning an Urban Air Mobility Encounter Model from Expert Preferences](https://mlsanity.com/paper/1907.05575/) (2019)
- [Interactive Robot Training for Non-Markov Tasks](https://mlsanity.com/paper/2003.02232/) (2020)
- [Preference-based Learning of Reward Function Features](https://mlsanity.com/paper/2103.02727/) (2021)
- [Here's What I've Learned: Asking Questions that Reveal Reward Learning](https://mlsanity.com/paper/2107.01995/) (2021)

## Built on by (most cited first, 14 of 14)

- [imitation: Clean Imitation Learning Implementations](https://mlsanity.com/paper/2211.11972/) (2022)
- [Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback](https://mlsanity.com/paper/2402.02423/) (2024)
- [Learning Reward for Robot Skills Using Large Language Models via Self-Alignment](https://mlsanity.com/paper/2405.07162/) (2024)
- [Rating-based Reinforcement Learning](https://mlsanity.com/paper/2307.16348/) (2023)
- [RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback](https://mlsanity.com/paper/2308.04332/) (2023)
- [Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions](https://mlsanity.com/paper/2112.11230/) (2021)
- [Batch Active Learning of Reward Functions from Human Preferences](https://mlsanity.com/paper/2402.15757/) (2024)
- [Learning Reward for Physical Skills using Large Language Model](https://mlsanity.com/paper/2310.14092/) (2023)
- [Optimizing Algorithms From Pairwise User Preferences](https://mlsanity.com/paper/2308.04571/) (2023)
- [Does Your AI Agent Get You? A Personalizable Framework for Approximating Human Models from Argumentation-based Dialogue Traces](https://mlsanity.com/paper/2502.16376/) (2025)
- [Predicting Human Perceptions of Robot Performance During Navigation Tasks](https://mlsanity.com/paper/2310.11590/) (2023)
- [ProVox: Personalization and Proactive Planning for Situated Human-Robot Collaboration](https://mlsanity.com/paper/2506.12248/) (2025)
- [DP-Dueling: Learning from Preference Feedback without Compromising User Privacy](https://mlsanity.com/paper/2403.15045/) (2024)
- [Cieran: Designing Sequential Colormaps via In-Situ Active Preference Learning](https://mlsanity.com/paper/2402.15997/) (2024)

## Abstract (arXiv)

Reward learning is a fundamental problem in human-robot interaction to have robots that operate in alignment with what their human user wants. Many preference-based learning algorithms and active querying techniques have been proposed as a solution to this problem. In this paper, we present APReL, a library for active preference-based reward learning algorithms, which enable researchers and practitioners to experiment with the existing techniques and easily develop their own algorithms for various modules of the problem. APReL is available at https://github.com/Stanford-ILIAD/APReL.