mlsanityData to 14 Oct 2026

Online LLM Personalization from Pairwise Preferences Across Users

arXiv 2610.17203 ↗First posted 12 Oct 2026cs.LG cs.CL stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models can answer the same question in different styles, but users' preferred styles are hidden and vary across users and questions. We study inference-time personalization through online selection from a fixed set of prompts, each representing a persona. Users arrive sequentially, each asks a short sequence of questions, and the system displays two responses per question and observes one pairwise choice. We formulate this interaction as a variant of the contextual dueling bandit problem in which the learner starts without preference data, adapts within each session, and exploits information shared across users. The objective is to minimize cumulative strong regret, summing the utility gaps of both displayed responses relative to the user's best response among the available options. We make two contributions. First, we introduce the problem and release a reproducible preference benchmark and dataset built around a tutoring case study, containing educational questions, LLM-generated answers in six teaching styles, and pairwise judgments from LLM-simulated students with hidden preference profiles. We fit Bradley-Terry models to these judgments to obtain reference utilities for regret evaluation. Second, we propose a baseline that learns a mixture prior over users from completed sessions and adapts to each new user within the session. On students and topics never used in development, it removes 44-48% of random pairs' excess regret and achieves over half the performance of a per-user linear oracle fitted in hindsight. We release code, prompts, and the benchmark.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 43 of its 61 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

43 of 61 references.

References · 43

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  4. 2023Whose Opinions Do Language Models Reflect?cited by 773
  5. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  7. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
  8. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  9. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
  10. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  11. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  12. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
31 more
  1. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 234
  2. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
  3. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 144
  4. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 132
  5. 2014Online Clustering of Banditscited by 102
  6. 2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 102
  7. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 74
  8. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 73
  9. 2024"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Modelscited by 69
  10. 2021Meta-Thompson Samplingcited by 63
  11. 2024Active Preference Optimization for Sample Efficient RLHFcited by 62
  12. 2024PersonalLLM: Tailoring LLMs to Individual Preferencescited by 60
  13. 2024Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluationcited by 58
  14. 2023When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Modelscited by 53
  15. 2020Meta-learning with Stochastic Linear Banditscited by 46
  16. 2020Latent Bandits Revisitedcited by 44
  17. 2021Bayesian decision-making under misspecified priors with applications to meta-learningcited by 44
  18. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 38
  19. 2022Stochastic Contextual Dueling Bandits under Linear Stochastic Transitivity Modelscited by 37
  20. 2021Hierarchical Bayesian Banditscited by 35
  21. 2021No Regrets for Learning the Prior in Banditscited by 30
  22. 2024Feel-Good Thompson Sampling for Contextual Dueling Banditscited by 27
  23. 2024Towards the Pedagogical Steering of Large Language Models for Tutoring: A Case Study with Modeling Productive Failurecited by 24
  24. 2021Thompson Sampling with a Mixture Priorcited by 16
  25. 2021Metalearning Linear Bandits by Prior Updatecited by 16
  26. 2024Personality-aware Student Simulation for Conversational Intelligent Tutoring Systemscited by 12
  27. 2025T-POP: Test-Time Personalization with Online Preference Feedbackcited by 7
  28. 2025Inference-Time Personalized Alignment with a Few User Preference Queriescited by 6
  29. 2026One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignmentcited by 5
  30. 2025Online Clustering of Dueling Banditscited by 3
  31. 2013Relative Upper Confidence Bound for the K-Armed Dueling Bandit Problemcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026HyperTrace: Hypothesis-Based Preference Tracing for Online LLM Personalization
  2. 2026Cold-Start Personalization via Training-Free Priors from Structured World Models
  3. 2024Comparison-based Active Preference Learning for Multi-dimensional Personalization
  4. 2023Active Preference Inference using Language Models and Probabilistic Reasoning
  5. 2026APM: Evaluating Style Personalization in LLMs with Arbitrary Preference Mappings
  6. 2025FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Users
  7. 2025Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas
  8. 2024ComPO: Community Preferences for Language Model Personalization