mlsanityData to 14 Oct 2026

Decoding-Time Language Model Alignment with Multiple Objectives

arXiv 2406.18853 ↗First posted 27 Jun 2024cs.LGcited by 94

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning language models (LMs) to human preferences has emerged as a critical pursuit, enabling these models to better serve diverse user needs. Existing methods primarily focus on optimizing LMs for a single reward function, limiting their adaptability to varied objectives. Here, we propose multi-objective decoding (MOD), a decoding-time algorithm that outputs the next token from a linear combination of predictions of all base models, for any given weightings over different objectives. We exploit a common form among a family of f-divergence regularized alignment approaches (such as PPO, DPO, and their variants) to identify a closed-form solution by Legendre transform, and derive an efficient decoding strategy. Theoretically, we show why existing approaches can be sub-optimal even in natural settings and obtain optimality guarantees for our method. Empirical results demonstrate the effectiveness of the algorithm. For example, compared to a parameter-merging baseline, MOD achieves 12.8% overall reward improvement when equally optimizing towards 3 objectives. Moreover, we experiment with MOD on combining three fully-finetuned LLMs of different model sizes, each aimed at different objectives such as safety, coding, and general user preference. Unlike traditional methods that require careful curation of a mixture of datasets to achieve comprehensive improvement, we can quickly experiment with preference weightings using MOD to find the best combination of models. Our best combination reduces toxicity on Toxigen to nearly 0% and achieves 7.9--33.3% improvement across other three metrics (textiti.e., Codex@1, GSM-COT, BBH-COT).

Key facts. Cited by 94 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 50 of its 67 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

50 of 67 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment29 Feb
  2. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  3. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  4. 2024Decoding-time Realignment of Language Models5 Feb
  5. 2023Align on the Fly: Adapting Chatbot Behavior to Established Norms26 Dec
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  7. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
30 more
  1. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  3. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  4. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  5. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  6. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  7. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  8. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  9. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 161
  10. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
  11. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  12. 2023Controlled Decoding from Language Modelscited by 135
  13. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  14. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 111
  15. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  16. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  17. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 90
  18. 2024Tuning Language Models by Proxycited by 89
  19. 2021Controlled Text Generation as Continuous Optimization with Multiple Constraintscited by 70
  20. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  21. 2022Gradient-Based Constrained Sampling from Language Modelscited by 64
  22. 2022PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithmcited by 58
  23. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  24. 2024Fine-tuning of diffusion models via stochastic control: entropy regularization and beyondcited by 42
  25. 2023Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RLcited by 36
  26. 2024Offset Unlearning for Large Language Modelscited by 36
  27. 2021Controllable Generation from Pre-trained Language Models via Inverse Promptingcited by 29
  28. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  29. 2023Merging Decision Transformers: Weight Averaging for Forming Multi-Task Policiescited by 15
  30. 2023Free from Bellman Completeness: Trajectory Stitching via Model-based Return-conditioned Supervised Learningcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 53; 2026: 33.

94 papers.

Sort:

By area:

  1. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  2. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  3. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 71
  4. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  5. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 60
  6. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  7. 2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 46
  8. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 43
  9. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 42
  10. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 38
  11. 2024Personality Alignment of Large Language Modelscited by 36
  12. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 36
38 more
  1. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 28
  2. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 28
  3. 2024The Crucial Role of Samplers in Online Direct Preference Optimizationcited by 27
  4. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 27
  5. 2025ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoningcited by 25
  6. 2024Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Modelcited by 22
  7. 2025Robust Multi-Objective Preference Alignment with Online DPOcited by 22
  8. 2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 18
  9. 2025Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignmentcited by 16
  10. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  11. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 14
  12. 2025Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generationcited by 14
  13. 2025Drift: Decoding-time Personalized Alignments with Implicit User Preferencescited by 13
  14. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Modelscited by 13
  15. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 13
  16. 2025CoSteer: Collaborative Decoding-Time Personalization via Local Delta Steeringcited by 13
  17. 2025Fast Large Language Model Collaborative Decoding via Speculationcited by 12
  18. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  19. 2025Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHFcited by 11
  20. 2025Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgmentcited by 11
  21. 2026Reward-free Alignment for Conflicting Objectivescited by 11
  22. 2025On-the-fly Preference Alignment via Principle-Guided Decodingcited by 10
  23. 2025MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignmentcited by 10
  24. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
  25. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 9
  26. 2025Bounded Rationality for LLMs: Satisficing Alignment at Inference-Timecited by 9
  27. 2026Token-Level LLM Collaboration via FusionRoutecited by 9
  28. 2025Capturing Individual Human Preferences with Reward Featurescited by 8
  29. 2025ProxyThinker: Test-Time Guidance through Small Visual Reasonerscited by 8
  30. 2025CoBRA: Programming Cognitive Bias in Social Agents Using Classic Social Science Experimentscited by 8
  31. 2025Test-Time Scaling in Diffusion LLMs via Hidden Semi-Autoregressive Expertscited by 8
  32. 2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Modelscited by 7
  33. 2025Sampling from Your Language Model One Byte at a Timecited by 7
  34. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 7
  35. 2025GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMscited by 6
  36. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewardscited by 6
  37. 2025Inference-Time Personalized Alignment with a Few User Preference Queriescited by 6
  38. 2026EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learningcited by 6

50 most cited of 94 All 94 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024DeAL: Decoding-time Alignment for Large Language Models
  2. 2025Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding
  3. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignment
  4. 2024Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Prompts
  5. 2024A Thorough Examination of Decoding Methods in the Era of LLMs
  6. 2025Collab: Controlled Decoding using Mixture of Agents for LLM Alignment
  7. 2024Aligning Large Language Models with Representation Editing: A Control Perspective
  8. 2025Pareto Multi-Objective Alignment for Language Models