mlsanityData to 7 Oct 2026

PAD: Personalized Alignment of LLMs at Decoding-Time

arXiv 2410.04070 ↗First posted 5 Oct 2024cs.CL cs.AIcited by 58

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Aligning with personalized preferences, which vary significantly across cultural, educational, and political differences, poses a significant challenge due to the computational costs and data demands of traditional alignment methods. In response, this paper presents Personalized Alignment at Decoding-time (PAD), a novel framework designed to align LLM outputs with diverse personalized preferences during the inference phase, eliminating the need for additional training. By introducing a unique personalized reward modeling strategy, this framework decouples the text generation process from personalized preferences, facilitating the generation of generalizable token-level personalized rewards. The PAD algorithm leverages these rewards to guide the decoding process, dynamically tailoring the base model's predictions to personalized preferences. Extensive experimental results demonstrate that PAD not only outperforms existing training-based alignment methods in terms of aligning with diverse preferences but also shows significant generalizability to preferences unseen during training and scalability across different base models. This work advances the capability of LLMs to meet user needs in real-time applications, presenting a substantial step forward in personalized LLM alignment.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 57 of its 72 references that are in the map. First posted on arXiv 5 Oct 2024.

Builds on

57 of 72 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Aligning to Thousands of Preferences via System Message Generalization28 May
  2. 2024A Roadmap to Pluralistic Alignment7 Feb
  3. 2024Personalized Language Modeling from Personalized Human Feedback6 Feb
  4. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
  5. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values11 Oct
  6. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models23 Aug
  7. 2023Aligning Language Models to User Opinions24 May
  8. 2020Learning to summarize from human feedback2 Sep

Other references · 49

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  9. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  10. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  11. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  12. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
37 more
  1. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  2. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  3. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  4. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  5. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  6. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  7. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editingcited by 182
  8. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  9. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  10. 2023Controlled Decoding from Language Modelscited by 133
  11. 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 115
  12. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 105
  13. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  14. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  15. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  16. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 87
  17. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
  18. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73
  19. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  20. 2023Fast Model Debias with Machine Unlearningcited by 70
  21. 2024Decoding-time Realignment of Language Modelscited by 67
  22. 2024DeAL: Decoding-time Alignment for Large Language Modelscited by 55
  23. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 54
  24. 2023See and Think: Embodied Agent in Virtual Environmentcited by 54
  25. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  26. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Modelscited by 43
  27. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 42
  28. 2024Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Modelscited by 32
  29. 2024Learnable Privacy Neurons Localization in Language Modelscited by 26
  30. 2024CityCraft: A Real Crafter for 3D City Generationcited by 25
  31. 2024Hierarchical Auto-Organizing System for Open-Ended Multi-Agent Navigationcited by 24
  32. 2023CityGen: Infinite and Controllable City Layout Generationcited by 22
  33. 2024BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMscited by 21
  34. 2024Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Modelcited by 19
  35. 2024MORL-Prompt: An Empirical Analysis of Multi-Objective Reinforcement Learning for Discrete Prompt Optimizationcited by 16
  36. 2024Large Language Model Bias Mitigation from the Perspective of Knowledge Editingcited by 16
  37. 2024STEVE Series: Step-by-Step Construction of Agent Systems in Minecraftcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 30; 2026: 27.

58 papers.

Sort:

By area:

  1. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  2. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  3. 2025Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalizationcited by 44
  4. 2024FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMscited by 36
  5. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 34
  6. 2025Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generationcited by 29
  7. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignmentcited by 28
  8. 2025Language Model Personalization via Reward Factorizationcited by 27
  9. 2025A Survey of Personalization: From RAG to Agentcited by 26
  10. 2025Latent Inter-User Difference Modeling for LLM Personalizationcited by 20
  11. 2025Datasets and Recipes for Video Temporal Grounding via Reinforcement Learningcited by 18
  12. 2026Learning Personalized Agents from Human Feedbackcited by 16
38 more
  1. 2025Drift: Decoding-time Personalized Alignments with Implicit User Preferencescited by 13
  2. 2025The Era of Real-World Human Interaction: RL from User Conversationscited by 13
  3. 2025CoSteer: Collaborative Decoding-Time Personalization via Local Delta Steeringcited by 12
  4. 2025Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgmentcited by 11
  5. 2025PersonaAgent: Bridging Memory and Action for Personalized LLM Agentscited by 11
  6. 2025MT-RewardTree: A Comprehensive Framework for Advancing LLM-Based Machine Translation via Reward Modelingcited by 9
  7. 2025Capturing Individual Human Preferences with Reward Featurescited by 8
  8. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  9. 2025BiasFilter: An Inference-Time Debiasing Framework for Large Language Modelscited by 6
  10. 2025T-POP: Test-Time Personalization with Online Preference Feedbackcited by 6
  11. 2026SpecSteer: Synergizing Local Context and Global Reasoning for Efficient Personalized Generationcited by 6
  12. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Modelscited by 5
  13. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewardscited by 5
  14. 2025Multi-Value Alignment for LLMs via Value Decorrelation and Extrapolationcited by 5
  15. 2025BiasGuard: A Reasoning-enhanced Bias Detection Tool For Large Language Modelscited by 4
  16. 2026Synthetic Interaction Data for Scalable Personalization in Large Language Modelscited by 4
  17. 2025Towards Proactive Personalization through Profile Customization for Individual Users in Dialoguescited by 3
  18. 2026GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimizationcited by 3
  19. 2025A Survey on Training-free Alignment of Large Language Modelscited by 2
  20. 2026Text as a Universal Interface for Transferable Personalizationcited by 2
  21. 2026Enhancing Persona Following at Decoding Time via Dynamic Importance Estimation for Role-Playing Agentscited by 2
  22. 2026VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Modelscited by 2
  23. 2026Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuningcited by 2
  24. 2026Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable Personalizationcited by 2
  25. 2026Preference-Aware Rubric Learning for Personalized Evaluationcited by 2
  26. 2025Detection, Classification, and Mitigation of Gender Bias in Large Language Modelscited by 1
  27. 2025Alignment-Aware Decodingcited by 1
  28. 2026LLMdoctor: Token-Level Flow-Guided Preference Optimization for Efficient Test-Time Alignment of Large Language Modelscited by 1
  29. 2026Limits of n-gram Style Control for LLMs via Logit-Space Injectioncited by 1
  30. 2026Spectral Souping: A Unified Framework for Online Preference Alignmentcited by 1
  31. 2026Toward Robust Personalized Alignment for LLMs: Mitigating Persona Drift in Multi-Turn Dialoguecited by 1
  32. 2025Aligning LLMs on a Budget: Inference-Time Alignment with Heuristic Reward Modelscited by 0
  33. 2025YNTP-100: A Benchmark for Your Next Token Prediction with 100 Peoplecited by 0
  34. 2026ECLIPTICA -- A Framework for Switchable LLM Alignment via CITA - Contrastive Instruction-Tuned Alignmentcited by 0
  35. 2026Do Implicit Personalization and Explicit Styles Conflict? PsPLUG: A Lightweight Plug-in for Balancing Personalization and Style in Customized LLMscited by 0
  36. 2026Light Alignment Improves LLM Safety via Model Self-Reflection with a Single Neuroncited by 0
  37. 2026VISA: Value Injection via Shielded Adaptation for Personalized LLM Alignmentcited by 0
  38. 2026Reward Weighted Classifier-Free Guidance as Policy Improvement in Autoregressive Modelscited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
  2. 2025HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation
  3. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
  4. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications
  5. 2025Personalized LLM Decoding via Contrasting Personal Preference
  6. 2025The Reward Model Selection Crisis in Personalized Alignment
  7. 2026One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment
  8. 2026Personalizing LLMs with Binary Feedback: A Preference-Corrected Optimization Framework