mlsanityData to 14 Oct 2026

Inference-Time Personalized Alignment with a Few User Preference Queries

arXiv 2511.02966 ↗First posted 4 Nov 2025cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

We study the problem of aligning a generative model's response with a user's preferences. Recent works have proposed several different formulations for personalized alignment; however, they either require a large amount of user preference queries or require that the preference be explicitly specified as a text input. In this paper, we propose a novel inference-time personalized alignment method, UserAlign, that elicits the user's preferences with a few queries as pairwise response comparisons. In particular, UserAlign builds on the theoretical framework of best-arm identification in logistic bandits and selects a personalized response from a fixed pool of the model's generated responses. The key idea is to consider the user's feedback consistent and noise-free, and incorporate it into the theoretical framework to identify the best response quickly. Experimental results across several tasks, involving personalized text and image generation, showcase the effectiveness of UserAlign in achieving personalized alignment.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 39 of its 56 references that are in the map. First posted on arXiv 4 Nov 2025.

Builds on

39 of 56 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  10. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  11. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  12. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
27 more
  1. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
  2. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 132
  3. 2020Improved Optimistic Algorithms for Logistic Banditscited by 118
  4. 2024ARGS: Alignment as Reward-Guided Searchcited by 103
  5. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 94
  6. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  7. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Modelcited by 79
  8. 2017Preferential Bayesian Optimizationcited by 74
  9. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 74
  10. 2024Active Preference Optimization for Sample Efficient RLHFcited by 62
  11. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  12. 2024Active Preference Learning for Large Language Modelscited by 58
  13. 2020Instance-Wise Minimax-Optimal Algorithms for Logistic Banditscited by 57
  14. 2024DeAL: Decoding-time Alignment for Large Language Modelscited by 55
  15. 2024Efficient Exploration for LLMscited by 45
  16. 2017Fully adaptive algorithm for pure exploration in linear banditscited by 43
  17. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 43
  18. 2024Reinforcement Learning from Human Feedback with Active Queriescited by 43
  19. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 42
  20. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 38
  21. 2024Bayesian Reward Models for LLM Alignmentcited by 35
  22. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMscited by 29
  23. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 27
  24. 2023Improved Regret Bounds of (Multinomial) Logistic Bandits via Regret-to-Confidence-Set Conversioncited by 24
  25. 2024Deep Bayesian Active Learning for Preference Modeling in Large Language Modelscited by 19
  26. 2024Sample-Efficient Alignment for LLMscited by 16
  27. 2025On-the-fly Preference Alignment via Principle-Guided Decodingcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Efficient Personalization of Generative User Interfacescited by 0
  2. 2026Coherence Maximization Improves Pluralistic Alignmentcited by 0
  3. 2026From Static Personal Values to Contextualized Personalization: Bayesian Personalized Value Alignment for LLMscited by 0
  4. 2026The Missing Coefficients: Bayesian Pairwise Merging for Model Personalizationcited by 0
  5. 2026NepPlural: A Human-Anchored Benchmark for Persona Annotation in Nepali Migration Discoursecited by 0
  6. 2026Online LLM Personalization from Pairwise Preferences Across Userscited by 0