# Efficient Controller Learning from Human Preferences and Numerical Data Via Multi-Modal Surrogate Models

arXiv: https://arxiv.org/abs/2603.24138 · HTML: https://mlsanity.com/paper/2603.24138/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 25 Mar 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 18 of 18)

- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Multi-Information Source Optimization](https://mlsanity.com/paper/1603.00389/) (2016)
- [Safe Controller Optimization for Quadrotors with Gaussian Processes](https://mlsanity.com/paper/1509.01066/) (2015)
- [Preferential Bayesian Optimization](https://mlsanity.com/paper/1704.03651/) (2017)
- [Preference Exploration for Efficient Bayesian Optimization with Multiple Outcomes](https://mlsanity.com/paper/2203.11382/) (2022)
- [Automatic LQR Tuning Based on Gaussian Process Global Optimization](https://mlsanity.com/paper/1605.01950/) (2016)
- [Overview of Gaussian process based multi-fidelity techniques with variable relationship between fidelities](https://mlsanity.com/paper/2006.16728/) (2020)
- [qEUBO: A Decision-Theoretic Acquisition Function for Preferential Bayesian Optimization](https://mlsanity.com/paper/2303.15746/) (2023)
- [Performance-oriented model learning for data-driven MPC design](https://mlsanity.com/paper/1904.10839/) (2019)
- [Multi-Fidelity Bayesian Optimization with Unreliable Information Sources](https://mlsanity.com/paper/2210.13937/) (2022)
- [Stability-informed Bayesian Optimization for MPC Cost Function Learning](https://mlsanity.com/paper/2404.12187/) (2024)
- [Safe and Stable Closed-Loop Learning for Neural-Network-Supported Model Predictive Control](https://mlsanity.com/paper/2409.10171/) (2024)
- [Preference-based MPC calibration](https://mlsanity.com/paper/2003.11294/) (2020)
- [Simulation-Aided Policy Tuning for Black-Box Robot Learning](https://mlsanity.com/paper/2411.14246/) (2024)
- [A Hierarchical Surrogate Model for Efficient Multi-Task Parameter Learning in Closed-Loop Control](https://mlsanity.com/paper/2508.12738/) (2025)
- [Exploiting Prior Knowledge in Preferential Learning of Individualized Autonomous Vehicle Driving Styles](https://mlsanity.com/paper/2503.15407/) (2025)
- [Efficient Learning of Vehicle Controller Parameters via Multi-Fidelity Bayesian Optimization: From Simulation to Experiment](https://mlsanity.com/paper/2506.08719/) (2025)
- [Guided Multi-Fidelity Bayesian Optimization for Data-driven Controller Tuning with Digital Twins](https://mlsanity.com/paper/2509.17952/) (2025)

## Built on by (most cited first, 1 of 1)

- [Exploiting Hierarchical Controller Structure in Contextual Parameter Learning for Humanoid Loco-Manipulation](https://mlsanity.com/paper/2610.04609/) (2026)

## Abstract (arXiv)

Tuning control policies manually to meet high-level objectives is often time-consuming. Bayesian optimization provides a data-efficient framework for automating this process using numerical evaluations of an objective function. However, many systems, particularly those involving humans, require optimization based on subjective criteria. Preferential Bayesian optimization addresses this by learning from pairwise comparisons instead of quantitative measurements, but relying solely on preference data can be inefficient. We propose a multi-fidelity, multi-modal Bayesian optimization framework that integrates low-fidelity numerical data with high-fidelity human preferences. Our approach employs Gaussian process surrogate models with both hierarchical, autoregressive and non-hierarchical, coregionalization-based structures, enabling efficient learning from mixed-modality data. We illustrate the framework by tuning an autonomous vehicle's trajectory planner, showing that combining numerical and preference data significantly reduces the need for experiments involving the human decision maker while effectively adapting driving style to individual preferences.