# Optimizing Algorithms From Pairwise User Preferences

arXiv: https://arxiv.org/abs/2308.04571 · HTML: https://mlsanity.com/paper/2308.04571/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 40 references that are in the map.
- First posted on arXiv 8 Aug 2023.

## Builds on (most cited first, 19 of 19)

- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting](https://mlsanity.com/paper/1603.08785/) (2016)
- [Core Challenges of Social Robot Navigation: A Survey](https://mlsanity.com/paper/2103.05668/) (2021)
- [Intel RealSense Stereoscopic Depth Cameras](https://mlsanity.com/paper/1705.05548/) (2017)
- [A Literature Survey of Benchmark Functions For Global Optimization Problems](https://mlsanity.com/paper/1308.4008/) (2013)
- [Federated Bayesian Optimization via Thompson Sampling](https://mlsanity.com/paper/2010.10154/) (2020)
- [Preferential Bayesian Optimization](https://mlsanity.com/paper/1704.03651/) (2017)
- [Active Preference-Based Gaussian Process Regression for Reward Learning](https://mlsanity.com/paper/2005.02575/) (2020)
- [Preference Exploration for Efficient Bayesian Optimization with Multiple Outcomes](https://mlsanity.com/paper/2203.11382/) (2022)
- [Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oracles](https://mlsanity.com/paper/2303.03751/) (2023)
- [SocNavBench: A Grounded Simulation Testing Framework for Evaluating Social Navigation](https://mlsanity.com/paper/2103.00047/) (2021)
- [Preferential Batch Bayesian Optimization](https://mlsanity.com/paper/2003.11435/) (2020)
- [APReL: A Library for Active Preference-based Reward Learning Algorithms](https://mlsanity.com/paper/2108.07259/) (2021)
- [A Federated Data-Driven Evolutionary Algorithm](https://mlsanity.com/paper/2102.08288/) (2021)
- [Optimal Bounds for Noisy Sorting](https://mlsanity.com/paper/2302.12440/) (2023)
- [Learning How to Optimize Black-Box Functions With Extreme Limits on the Number of Function Evaluations](https://mlsanity.com/paper/2103.10321/) (2021)
- [Customized Handling of Unintended Interface Operation in Assistive Robots](https://mlsanity.com/paper/2007.02092/) (2020)
- [Distributed Evolution Strategies for Black-box Stochastic Optimization](https://mlsanity.com/paper/2204.04450/) (2022)
- [Discovering Multiple Algorithm Configurations](https://mlsanity.com/paper/2303.07434/) (2023)

## Built on by (most cited first, 4 of 4)

- [Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation](https://mlsanity.com/paper/2312.13469/) (2023)
- [Demonstration-Enhanced Adaptable Multi-Objective Robot Navigation](https://mlsanity.com/paper/2404.04857/) (2024)
- [Improving through Interaction: Searching Behavioral Representation Spaces with CMA-ES-IG](https://mlsanity.com/paper/2603.09011/) (2026)
- [ACME: A Multi-Cultural, Multi-Embodiment Social-Navigation Dataset](https://mlsanity.com/paper/2607.21964/) (2026)

## Abstract (arXiv)

Typical black-box optimization approaches in robotics focus on learning from metric scores. However, that is not always possible, as not all developers have ground truth available. Learning appropriate robot behavior in human-centric contexts often requires querying users, who typically cannot provide precise metric scores. Existing approaches leverage human feedback in an attempt to model an implicit reward function; however, this reward may be difficult or impossible to effectively capture. In this work, we introduce SortCMA to optimize algorithm parameter configurations in high dimensions based on pairwise user preferences. SortCMA efficiently and robustly leverages user input to find parameter sets without directly modeling a reward. We apply this method to tuning a commercial depth sensor without ground truth, and to robot social navigation, which involves highly complex preferences over robot behavior. We show that our method succeeds in optimizing for the user's goals and perform a user study to evaluate social navigation results.