# Enhancing Preference-based Linear Bandits via Human Response Time

arXiv: https://arxiv.org/abs/2409.05798 · HTML: https://mlsanity.com/paper/2409.05798/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 86 references that are in the map.
- First posted on arXiv 9 Sep 2024.

## Builds on (most cited first, 23 of 23)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Teaching language models to support answers with verified quotes](https://mlsanity.com/paper/2203.11147/) (2022)
- [On the Complexity of Best Arm Identification in Multi-Armed Bandit Models](https://mlsanity.com/paper/1407.4443/) (2014)
- [Bandits with Knapsacks](https://mlsanity.com/paper/1305.2545/) (2013)
- [Preference-based Online Learning with Dueling Bandits: A Survey](https://mlsanity.com/paper/1807.11398/) (2018)
- [Sequential Experimental Design for Transductive Linear Bandits](https://mlsanity.com/paper/1906.08399/) (2019)
- [Gamification of Pure Exploration for Linear Bandits](https://mlsanity.com/paper/2007.00953/) (2020)
- [High-Dimensional Experimental Design and Kernel Bandits](https://mlsanity.com/paper/2105.05806/) (2021)
- [MNL-Bandit: A Dynamic Learning Approach to Assortment Selection](https://mlsanity.com/paper/1706.03880/) (2017)
- [Minimax Optimal Fixed-Budget Best Arm Identification in Linear Bandits](https://mlsanity.com/paper/2105.13017/) (2021)
- [Improved Confidence Bounds for the Linear Logistic Model and Applications to Linear Bandits](https://mlsanity.com/paper/2011.11222/) (2020)
- [Preference-Based Learning for Exoskeleton Gait Optimization](https://mlsanity.com/paper/1909.12316/) (2019)
- [Learning Reward Functions from Scale Feedback](https://mlsanity.com/paper/2110.00284/) (2021)
- [Fixed-Budget Best-Arm Identification in Structured Bandits](https://mlsanity.com/paper/2106.04763/) (2021)
- [Content-driven Music Recommendation: Evolution, State of the Art, and Challenges](https://mlsanity.com/paper/2107.11803/) (2021)
- [On Elimination Strategies for Bandit Fixed-Confidence Identification](https://mlsanity.com/paper/2205.10936/) (2022)
- [From "Thumbs Up" to "10 out of 10": Reconsidering Scalar Feedback in Interactive Reinforcement Learning](https://mlsanity.com/paper/2311.10284/) (2023)
- [Optimal Exploration is no harder than Thompson Sampling](https://mlsanity.com/paper/2310.06069/) (2023)
- [Instance-Sensitive Algorithms for Pure Exploration in Multinomial Logit Bandit](https://mlsanity.com/paper/2012.01499/) (2020)
- [Human Goal Recognition as Bayesian Inference: Investigating the Impact of Actions, Timing, and Goal Solvability](https://mlsanity.com/paper/2402.10510/) (2024)

## Built on by (most cited first, 3 of 3)

- [Understanding LoRA as Knowledge Memory: An Empirical Analysis](https://mlsanity.com/paper/2603.01097/) (2026)
- [Preference Learning with Response Time: Robust Losses and Guarantees](https://mlsanity.com/paper/2505.22820/) (2025)
- [In-Context Reward Adaptation for Robust Preference Modeling](https://mlsanity.com/paper/2605.30323/) (2026)

## Abstract (arXiv)

Interactive preference learning systems infer human preferences by presenting queries as pairs of options and collecting binary choices. Although binary choices are simple and widely used, they provide limited information about preference strength. To address this, we leverage human response times, which are inversely related to preference strength, as an additional signal. We propose a computationally efficient method that combines choices and response times to estimate human utility functions, grounded in the EZ diffusion model from psychology. Theoretical and empirical analyses show that for queries with strong preferences, response times complement choices by providing extra information about preference strength, leading to significantly improved utility estimation. We incorporate this estimator into preference-based linear bandits for fixed-budget best-arm identification. Simulations on three real-world datasets demonstrate that using response times significantly accelerates preference learning compared to choice-only approaches. Additional materials, such as code, slides, and talk video, are available at https://shenlirobot.github.io/pages/NeurIPS24.html