mlsanityData to 7 Oct 2026

Drift: Decoding-time Personalized Alignments with Implicit User Preferences

arXiv 2502.14289 ↗First posted 20 Feb 2025cs.CLcited by 13

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Personalized alignments for individual users have been a long-standing goal in large language models (LLMs). We introduce Drift, a novel framework that personalizes LLMs at decoding time with implicit user preferences. Traditional Reinforcement Learning from Human Feedback (RLHF) requires thousands of annotated examples and expensive gradient updates. In contrast, Drift personalizes LLMs in a training-free manner, using only a few dozen examples to steer a frozen model through efficient preference modeling. Our approach models user preferences as a composition of predefined, interpretable attributes and aligns them at decoding time to enable personalized generation. Experiments on both a synthetic persona dataset (Perspective) and a real human-annotated dataset (PRISM) demonstrate that Drift significantly outperforms RLHF baselines while using only 50-100 examples. Our results and analysis show that Drift is both computationally efficient and interpretable.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 46 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

32 of 46 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024PAD: Personalized Alignment of LLMs at Decoding-Time5 Oct
  2. 2024Aligning to Thousands of Preferences via System Message Generalization28 May
  3. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
  4. 2023Aligning Language Models to User Opinions24 May

Other references · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  8. 2018Hierarchical Neural Story Generationcited by 1,182
  9. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  10. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  11. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
  12. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
16 more
  1. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  2. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  3. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  4. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  5. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  6. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  7. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  8. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 87
  9. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  10. 2023Controlled Text Generation via Language Model Arithmeticcited by 51
  11. 2022Critic-Guided Decoding for Controlled Text Generationcited by 45
  12. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Modelscited by 43
  13. 2024Implicit Personalization in Language Models: A Systematic Studycited by 18
  14. 2023LifeTox: Unveiling Implicit Toxicity in Life Advicecited by 14
  15. 2024AdvisorQA: Towards Helpful and Harmless Advice-seeking Question Answering with Collective Intelligencecited by 12
  16. 2024PersonalSum: A User-Subjective Guided Personalized Summarization Dataset for Large Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 7.

13 papers.

Sort:

By area:

  1. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
  2. 2025Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generationcited by 29
  3. 2025CoSteer: Collaborative Decoding-Time Personalization via Local Delta Steeringcited by 12
  4. 2026Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directionscited by 10
  5. 2025Avoidance Decoding for Diverse Multi-Branch Story Generationcited by 6
  6. 2025FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Datacited by 5
  7. 2025On the Role of Preference Variance in Preference Optimizationcited by 3
  8. 2026EXACT: Explicit Attribute-Guided Decoding-Time Personalizationcited by 3
  9. 2026CARD: Cluster-level Adaptation with Reward-guided Decoding for Personalized Text Generationcited by 2
  10. 2026Do Implicit Personalization and Explicit Styles Conflict? PsPLUG: A Lightweight Plug-in for Balancing Personalization and Style in Customized LLMscited by 0
  11. 2026REAR: Test-time Preference Realignment through Reward Decompositioncited by 0
  12. 2026PrefReward: Learning User Preference Matrix for Personalized Text Generationcited by 0
1 more
  1. 2026Weightless Fine-Tuning: Personalizing LLMs via Logit-Space Transportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment
  2. 2024Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
  3. 2025Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment
  4. 2024Personalized Language Modeling from Personalized Human Feedback
  5. 2026Toward Robust Personalized Alignment for LLMs: Mitigating Persona Drift in Multi-Turn Dialogue
  6. 2024Aligning LLMs with Individual Preferences via Interaction
  7. 2025Is Active Persona Inference Necessary for Aligning Small Models to Personal Preferences?
  8. 2025Aligning Large Language Models via Fully Self-Synthetic Data