mlsanityData to 7 Oct 2026

Efficient Controller Learning from Human Preferences and Numerical Data Via Multi-Modal Surrogate Models

arXiv 2603.24138 ↗First posted 25 Mar 2026cs.LG cs.SY eess.SYcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Tuning control policies manually to meet high-level objectives is often time-consuming. Bayesian optimization provides a data-efficient framework for automating this process using numerical evaluations of an objective function. However, many systems, particularly those involving humans, require optimization based on subjective criteria. Preferential Bayesian optimization addresses this by learning from pairwise comparisons instead of quantitative measurements, but relying solely on preference data can be inefficient. We propose a multi-fidelity, multi-modal Bayesian optimization framework that integrates low-fidelity numerical data with high-fidelity human preferences. Our approach employs Gaussian process surrogate models with both hierarchical, autoregressive and non-hierarchical, coregionalization-based structures, enabling efficient learning from mixed-modality data. We illustrate the framework by tuning an autonomous vehicle's trajectory planner, showing that combining numerical and preference data significantly reduces the need for experiments involving the human decision maker while effectively adapting driving style to individual preferences.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 18 of its 35 references that are in the map. First posted on arXiv 25 Mar 2026.

Builds on

18 of 35 references.

References · 18

  1. 2018Pyro: Deep Universal Probabilistic Programmingcited by 379
  2. 2016Multi-Information Source Optimizationcited by 83
  3. 2015Safe Controller Optimization for Quadrotors with Gaussian Processescited by 77
  4. 2017Preferential Bayesian Optimizationcited by 72
  5. 2022Preference Exploration for Efficient Bayesian Optimization with Multiple Outcomescited by 41
  6. 2016Automatic LQR Tuning Based on Gaussian Process Global Optimizationcited by 29
  7. 2020Overview of Gaussian process based multi-fidelity techniques with variable relationship between fidelitiescited by 24
  8. 2023qEUBO: A Decision-Theoretic Acquisition Function for Preferential Bayesian Optimizationcited by 23
  9. 2019Performance-oriented model learning for data-driven MPC designcited by 22
  10. 2022Multi-Fidelity Bayesian Optimization with Unreliable Information Sourcescited by 17
  11. 2024Stability-informed Bayesian Optimization for MPC Cost Function Learningcited by 8
  12. 2024Safe and Stable Closed-Loop Learning for Neural-Network-Supported Model Predictive Controlcited by 6
6 more
  1. 2020Preference-based MPC calibrationcited by 5
  2. 2024Simulation-Aided Policy Tuning for Black-Box Robot Learningcited by 4
  3. 2025Exploiting Prior Knowledge in Preferential Learning of Individualized Autonomous Vehicle Driving Stylescited by 1
  4. 2025Efficient Learning of Vehicle Controller Parameters via Multi-Fidelity Bayesian Optimization: From Simulation to Experimentcited by 1
  5. 2025A Hierarchical Surrogate Model for Efficient Multi-Task Parameter Learning in Closed-Loop Controlcited by 1
  6. 2025Guided Multi-Fidelity Bayesian Optimization for Data-driven Controller Tuning with Digital Twinscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient Multi-Modal Planning with Reward-Guided Preference Optimization for Autonomous Driving
  2. 2021Dream to Explore: Adaptive Simulations for Autonomous Systems
  3. 2024DIPPER: Direct Preference Optimization to Accelerate Primitive-Enabled Hierarchical Reinforcement Learning
  4. 2024Preferential Multi-Objective Bayesian Optimization
  5. 2022Conditional Predictive Behavior Planning with Inverse Reinforcement Learning for Human-like Autonomous Driving
  6. 2025A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approach
  7. 2025Learning Personalized Driving Styles via Reinforcement Learning from Human Feedback
  8. 2025Predictive Preference Learning from Human Interventions