Improving User Experience in Preference-Based Optimization of Reward Functions for Assistive Robots
On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF
Abstract (arXiv)
Assistive robots interact with humans and must adapt to different users' preferences to be effective. An easy and effective technique to learn non-expert users' preferences is through rankings of robot behaviors, for example, robot movement trajectories or gestures. Existing techniques focus on generating trajectories for users to rank that maximize the outcome of the preference learning process. However, the generated trajectories do not appear to reflect the user's preference over repeated interactions. In this work, we design an algorithm to generate trajectories for users to rank that we call Covariance Matrix Adaptation Evolution Strategies with Information Gain (CMA-ES-IG). CMA-ES-IG prioritizes the user's experience of the preference learning process. We show that users find our algorithm more intuitive and easier to use than previous approaches across both physical and social robot tasks. This project's code is hosted at github.com/interaction-lab/CMA-ES-IG
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 18 of its 49 references that are in the map. First posted on arXiv 17 Nov 2024.
Builds on
18 of 49 references.
Closest ancestors · 1
Same direction, newest first
Other references · 17
- 2016The CMA Evolution Strategy: A Tutorialcited by 364
- 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
- 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
- 2022Correcting Robot Plans with Natural Language Feedbackcited by 68
- 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 67
- 2014Efficient Model Learning for Human-Robot Collaborative Taskscited by 35
- 2020Shared Autonomy with Learned Latent Actionscited by 34
- 2021Here's What I've Learned: Asking Questions that Reveal Reward Learningcited by 8
- 2019Long-Term Personalization of an In-Home Socially Assistive Robot for Children With Autism Spectrum Disorderscited by 6
- 2022Using Design Metaphors to Understand User Expectations of Socially Interactive Robot Embodimentscited by 6
- 2024LLM-Powered Socially Assistive Robot-Delivered Cognitive Behavioral Therapy Exercises: an Exploratory Study with University Studentscited by 6
- 2024Design and Evaluation of a Socially Assistive Robot Schoolwork Companion for College Students with ADHDcited by 5
5 more
- 2024A metric for characterizing the arm nonuse workspace in poststroke individuals using a robot armcited by 3
- 2024Build Your Own Robot Friend: An Open-Source Learning Module for Accessible and Engaging AI Educationcited by 3
- 2021Design and Evaluation of a Hair Combing System Using a General-Purpose Robotic Armcited by 2
- 2024Designing a Socially Assistive Robot to Support Older Adults with Low Visioncited by 2
- 2024Evaluating and Personalizing User-Perceived Quality of Text-to-Speech Voices for Delivering Mindfulness Meditation with Different Physical Embodimentscited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functions
- 2022Feedback-efficient Active Preference Learning for Socially Aware Robot Navigation
- 2023Active Reward Learning from Online Preferences
- 2025Efficiently Generating Expressive Quadruped Behaviors via Language-Guided Preference Learning
- 2022Learning from Physical Human Feedback: An Object-Centric One-Shot Adaptation Method
- 2023Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferences
- 2022Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interaction
- 2026Freeform Preference Learning for Robotic Manipulation