# Integrating Human Expertise in Continuous Spaces: A Novel Interactive Bayesian Optimization Framework with Preference Expected Improvement

arXiv: https://arxiv.org/abs/2401.12662 · HTML: https://mlsanity.com/paper/2401.12662/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 23 Jan 2024.

## Builds on (most cited first, 16 of 16)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training](https://mlsanity.com/paper/2106.05091/) (2021)
- [Interactive Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1701.06049/) (2017)
- [Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces](https://mlsanity.com/paper/1709.10163/) (2017)
- [Few-Shot Preference Learning for Human-in-the-Loop RL](https://mlsanity.com/paper/2212.03363/) (2022)
- [A Survey on Interpretable Reinforcement Learning](https://mlsanity.com/paper/2112.13112/) (2021)
- [Preference Exploration for Efficient Bayesian Optimization with Multiple Outcomes](https://mlsanity.com/paper/2203.11382/) (2022)
- [DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback](https://mlsanity.com/paper/1810.11748/) (2018)
- [A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges](https://mlsanity.com/paper/2105.12949/) (2021)
- [Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Models](https://mlsanity.com/paper/2301.04741/) (2023)
- [Interactive Learning with Corrective Feedback for Policies based on Deep Neural Networks](https://mlsanity.com/paper/1810.00466/) (2018)
- [Interactive Machine Learning: A State of the Art Review](https://mlsanity.com/paper/2207.06196/) (2022)
- [On Preference Learning Based on Sequential Bayesian Optimization with Pairwise Comparison](https://mlsanity.com/paper/2103.13192/) (2021)

## Abstract (arXiv)

Interactive Machine Learning (IML) seeks to integrate human expertise into machine learning processes. However, most existing algorithms cannot be applied to Realworld Scenarios because their state spaces and/or action spaces are limited to discrete values. Furthermore, the interaction of all existing methods is restricted to deciding between multiple proposals. We therefore propose a novel framework based on Bayesian Optimization (BO). Interactive Bayesian Optimization (IBO) enables collaboration between machine learning algorithms and humans. This framework captures user preferences and provides an interface for users to shape the strategy by hand. Additionally, we've incorporated a new acquisition function, Preference Expected Improvement (PEI), to refine the system's efficiency using a probabilistic model of the user preferences. Our approach is geared towards ensuring that machines can benefit from human expertise, aiming for a more aligned and effective learning process. In the course of this work, we applied our method to simulations and in a real world task using a Franka Panda robot to show human-robot collaboration.