# Improving User Experience in Preference-Based Optimization of Reward Functions for Assistive Robots

arXiv: https://arxiv.org/abs/2411.11182 · HTML: https://mlsanity.com/paper/2411.11182/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 18 of its 49 references that are in the map.
- First posted on arXiv 17 Nov 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 18 of 18)

- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Batch Active Preference-Based Learning of Reward Functions](https://mlsanity.com/paper/1810.04303/) (2018)
- [Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences](https://mlsanity.com/paper/2002.09089/) (2020)
- [Correcting Robot Plans with Natural Language Feedback](https://mlsanity.com/paper/2204.05186/) (2022)
- [Asking Easy Questions: A User-Friendly Approach to Active Reward Learning](https://mlsanity.com/paper/1910.04365/) (2019)
- [Learning Multimodal Rewards from Rankings](https://mlsanity.com/paper/2109.12750/) (2021)
- [Efficient Model Learning for Human-Robot Collaborative Tasks](https://mlsanity.com/paper/1405.6341/) (2014)
- [Shared Autonomy with Learned Latent Actions](https://mlsanity.com/paper/2005.03210/) (2020)
- [Here's What I've Learned: Asking Questions that Reveal Reward Learning](https://mlsanity.com/paper/2107.01995/) (2021)
- [Long-Term Personalization of an In-Home Socially Assistive Robot for Children With Autism Spectrum Disorders](https://mlsanity.com/paper/1911.07992/) (2019)
- [Using Design Metaphors to Understand User Expectations of Socially Interactive Robot Embodiments](https://mlsanity.com/paper/2201.10671/) (2022)
- [LLM-Powered Socially Assistive Robot-Delivered Cognitive Behavioral Therapy Exercises: an Exploratory Study with University Students](https://mlsanity.com/paper/2402.17937/) (2024)
- [Design and Evaluation of a Socially Assistive Robot Schoolwork Companion for College Students with ADHD](https://mlsanity.com/paper/2401.06289/) (2024)
- [A metric for characterizing the arm nonuse workspace in poststroke individuals using a robot arm](https://mlsanity.com/paper/2401.06974/) (2024)
- [Build Your Own Robot Friend: An Open-Source Learning Module for Accessible and Engaging AI Education](https://mlsanity.com/paper/2402.01647/) (2024)
- [Design and Evaluation of a Hair Combing System Using a General-Purpose Robotic Arm](https://mlsanity.com/paper/2108.01233/) (2021)
- [Designing a Socially Assistive Robot to Support Older Adults with Low Vision](https://mlsanity.com/paper/2401.03329/) (2024)
- [Evaluating and Personalizing User-Perceived Quality of Text-to-Speech Voices for Delivering Mindfulness Meditation with Different Physical Embodiments](https://mlsanity.com/paper/2401.03581/) (2024)

## Built on by (most cited first, 1 of 1)

- [QuickLAP: Quick Language-Action Preference Learning for Semi-Autonomous Agents](https://mlsanity.com/paper/2511.17855/) (2025)

## Abstract (arXiv)

Assistive robots interact with humans and must adapt to different users' preferences to be effective. An easy and effective technique to learn non-expert users' preferences is through rankings of robot behaviors, for example, robot movement trajectories or gestures. Existing techniques focus on generating trajectories for users to rank that maximize the outcome of the preference learning process. However, the generated trajectories do not appear to reflect the user's preference over repeated interactions. In this work, we design an algorithm to generate trajectories for users to rank that we call Covariance Matrix Adaptation Evolution Strategies with Information Gain (CMA-ES-IG). CMA-ES-IG prioritizes the user's experience of the preference learning process. We show that users find our algorithm more intuitive and easier to use than previous approaches across both physical and social robot tasks. This project's code is hosted at github.com/interaction-lab/CMA-ES-IG