Data to 7 Oct 2026

Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards

arXiv 2402.18571 ↗First posted 28 Feb 2024cs.LG cs.AI cs.CLcited by 145

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Fine-grained control over large language models (LLMs) remains a significant challenge, hindering their adaptability to diverse user needs. While Reinforcement Learning from Human Feedback (RLHF) shows promise in aligning LLMs, its reliance on scalar rewards often limits its ability to capture diverse user preferences in real-world applications. To address this limitation, we introduce the Directional Preference Alignment (DPA) framework. Unlike the scalar-reward RLHF, DPA incorporates multi-objective reward modeling to represent diverse preference profiles. Additionally, DPA models user preferences as directions (i.e., unit vectors) in the reward space to achieve user-dependent preference control. Our method involves training a multi-objective reward model and then fine-tuning the LLM with a preference-conditioned variant of Rejection Sampling Finetuning (RSF), an RLHF method adopted by Llama 2. This method enjoys a better performance trade-off across various reward objectives. In comparison with the scalar-reward RLHF, DPA offers users intuitive control over LLM generation: they can arithmetically specify their desired trade-offs (e.g., more helpfulness with less verbosity). We also validate the effectiveness of DPA with real-world alignment experiments on Mistral-7B. Our method provides straightforward arithmetic control over the trade-off between helpfulness and verbosity while maintaining competitive performance with strong baselines such as Direct Preference Optimization (DPO).

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 82 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

53 of 82 references.

Closest ancestors · 6

Same direction, newest first

Other references · 47

2020Language Models are Few-Shot Learnerscited by 32,199
2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
2023GPT-4 Technical Reportcited by 18,132
2017Proximal Policy Optimization Algorithmscited by 14,441
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
2017Deep reinforcement learning from human preferencescited by 3,846
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634

35 more

2023Mistral 7Bcited by 3,065
2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
2019Fine-Tuning Language Models from Human Preferencescited by 2,124
2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
2024Self-Rewarding Language Modelscited by 654
2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
2023Nash Learning from Human Feedbackcited by 220
2022Optimizing Prompts for Text-to-Image Generationcited by 204
2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 107
2023Is RLHF More Difficult than Standard RL?cited by 96
2018Batch Active Preference-Based Learning of Reward Functionscited by 87
2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
2023LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Modelscited by 58
2023Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspectivecited by 51
2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Modelscited by 23
2023Moral Machine or Tyranny of the Majority?cited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 52; 2025: 58; 2026: 35.

145 papers.

Sort:

By area:

2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 105
2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105

38 more

2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimizationcited by 75
2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 55
2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 53
2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 53
2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 44
2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 42
2024Personality Alignment of Large Language Modelscited by 36
2025Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Explorationcited by 33
2024Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Modelscited by 32
2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Docited by 32
2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
2024Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedbackcited by 31
2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedbackcited by 30
2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
2024Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Modelscited by 27
2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Trainingcited by 27
2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 27
2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimizationcited by 26
2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Modelscited by 22
2025Robust Multi-Objective Preference Alignment with Online DPOcited by 21
2024MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximationcited by 20
2024Inference-Time Language Model Alignment via Integrated Value Guidancecited by 19
2025Direct Alignment with Heterogeneous Preferencescited by 19
2025Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personascited by 17
2024ROPO: Robust Preference Optimization for Large Language Modelscited by 16
2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimizationcited by 16
2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 15
2025Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignmentcited by 15
2025Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactionscited by 14
2024Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learningcited by 13
2024T-REG: Preference Optimization with Token-Level Reward Regularizationcited by 13

50 most cited of 145 All 145 (JSON)

Similar, not cited

Close in wording, no citation either way