# Improving through Interaction: Searching Behavioral Representation Spaces with CMA-ES-IG

arXiv: https://arxiv.org/abs/2603.09011 · HTML: https://mlsanity.com/paper/2603.09011/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 40 of its 109 references that are in the map.
- First posted on arXiv 9 Mar 2026.

## Builds on (most cited first, 30 of 40)

- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Core Challenges of Social Robot Navigation: A Survey](https://mlsanity.com/paper/2103.05668/) (2021)
- [Batch Active Preference-Based Learning of Reward Functions](https://mlsanity.com/paper/1810.04303/) (2018)
- [Covariance Matrix Adaptation for the Rapid Illumination of Behavior Space](https://mlsanity.com/paper/1912.02400/) (2019)
- [Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences](https://mlsanity.com/paper/2002.09089/) (2020)
- [Asking Easy Questions: A User-Friendly Approach to Active Reward Learning](https://mlsanity.com/paper/1910.04365/) (2019)
- [Shared Autonomy via Deep Reinforcement Learning](https://mlsanity.com/paper/1802.01744/) (2018)
- [Quality-Diversity Optimization: a novel branch of stochastic optimization](https://mlsanity.com/paper/2012.04322/) (2020)
- [Autonomous skill discovery with Quality-Diversity and Unsupervised Descriptors](https://mlsanity.com/paper/1905.11874/) (2019)
- [CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrations](https://mlsanity.com/paper/2505.04999/) (2025)
- [Inference-Time Policy Steering through Human Interactions](https://mlsanity.com/paper/2411.16627/) (2024)
- [Efficient Model Learning for Human-Robot Collaborative Tasks](https://mlsanity.com/paper/1405.6341/) (2014)
- [Shared Autonomy with Learned Latent Actions](https://mlsanity.com/paper/2005.03210/) (2020)
- [Object Handovers: a Review for Robotics](https://mlsanity.com/paper/2007.12952/) (2020)
- [Robotic Vision for Human-Robot Interaction and Collaboration: A Survey and Systematic Review](https://mlsanity.com/paper/2307.15363/) (2023)
- [Comparison of High-Dimensional Bayesian Optimization Algorithms on BBOB](https://mlsanity.com/paper/2303.00890/) (2023)
- [Trajectory Improvement and Reward Learning from Comparative Language Feedback](https://mlsanity.com/paper/2410.06401/) (2024)
- [Human-Robot Mutual Adaptation in Shared Autonomy](https://mlsanity.com/paper/1701.07851/) (2017)
- [VoicePilot: Harnessing LLMs as Speech Interfaces for Physically Assistive Robots](https://mlsanity.com/paper/2404.04066/) (2024)
- [SIRL: Similarity-based Implicit Representation Learning](https://mlsanity.com/paper/2301.00810/) (2023)
- [Learning Models for Shared Control of Human-Machine Systems with Unknown Dynamics](https://mlsanity.com/paper/1808.08268/) (2018)
- [Here's What I've Learned: Asking Questions that Reveal Reward Learning](https://mlsanity.com/paper/2107.01995/) (2021)
- [The Sense of Agency in Assistive Robotics Using Shared Autonomy](https://mlsanity.com/paper/2501.07462/) (2025)
- [Long-Term Personalization of an In-Home Socially Assistive Robot for Children With Autism Spectrum Disorders](https://mlsanity.com/paper/1911.07992/) (2019)
- [Using Design Metaphors to Understand User Expectations of Socially Interactive Robot Embodiments](https://mlsanity.com/paper/2201.10671/) (2022)
- [LLM-Powered Socially Assistive Robot-Delivered Cognitive Behavioral Therapy Exercises: an Exploratory Study with University Students](https://mlsanity.com/paper/2402.17937/) (2024)
- [Design and Evaluation of a Socially Assistive Robot Schoolwork Companion for College Students with ADHD](https://mlsanity.com/paper/2401.06289/) (2024)
- [Personalizing User Engagement Dynamics in a Non-Verbal Communication Game for Cerebral Palsy](https://mlsanity.com/paper/2107.07446/) (2021)
- [Optimizing Algorithms From Pairwise User Preferences](https://mlsanity.com/paper/2308.04571/) (2023)

## Abstract (arXiv)

Robots that interact with humans must adapt to individual users' preferences to operate effectively in human-centered environments. An intuitive and effective technique to learn non-expert users' preferences is through rankings of robot behaviors, e.g., trajectories, gestures, or voices. Existing techniques primarily focus on generating queries that optimize preference learning outcomes, such as sample efficiency or final preference estimation accuracy. However, the focus on outcome overlooks key user expectations in the process of providing these rankings, which can negatively impact users' adoption of robotic systems. This work proposes the Covariance Matrix Adaptation Evolution Strategies with Information Gain (CMA-ES-IG) algorithm. CMA-ES-IG explicitly incorporates user experience considerations into the preference learning process by suggesting perceptually distinct and informative trajectories for users to rank. We demonstrate these benefits through both simulated studies and real-robot experiments. CMA-ES-IG, compared to state-of-the-art alternatives, (1) scales more effectively to higher-dimensional preference spaces, (2) maintains computational tractability for high-dimensional problems, (3) is robust to noisy or inconsistent user feedback, and (4) is preferred by non-expert users in identifying their preferred robot behaviors. This project's code is available at github.com/interaction-lab/CMA-ES-IG