mlsanityData to 7 Oct 2026

Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment

arXiv 2402.19085 ↗First posted 29 Feb 2024cs.CL cs.AI cs.SYcited by 105

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Alignment in artificial intelligence pursues the consistency between model responses and human preferences as well as values. In practice, the multifaceted nature of human preferences inadvertently introduces what is known as the "alignment tax" -a compromise where enhancements in alignment within one objective (e.g.,harmlessness) can diminish performance in others (e.g.,helpfulness). However, existing alignment techniques are mostly unidirectional, leading to suboptimal trade-offs and poor flexibility over various objectives. To navigate this challenge, we argue the prominence of grounding LLMs with evident preferences. We introduce controllable preference optimization (CPO), which explicitly specifies preference scores for different objectives, thereby guiding the model to generate responses that meet the requirements. Our experimental analysis reveals that the aligned models can provide responses that match various preferences among the "3H" (helpfulness, honesty, harmlessness) desiderata. Furthermore, by introducing diverse data and alignment goals, we surpass baseline methods in aligning with single objectives, hence mitigating the impact of the alignment tax and achieving improvements in multi-objective alignment.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 48 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

28 of 48 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  2. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  3. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs3 Feb
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  11. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
11 more
  1. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  2. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  3. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  4. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  5. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  6. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  7. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  8. 2022FaithDial: A Faithful Benchmark for Information-Seeking Dialoguecited by 108
  9. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 107
  10. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Modelscited by 61
  11. 2023On Uncertainty Calibration and Selective Generation in Probabilistic Neural Summarization: A Benchmark Studycited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 32; 2025: 50; 2026: 23.

105 papers.

Sort:

By area:

  1. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  2. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  3. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  4. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 105
  5. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  6. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  7. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
  8. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  9. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  10. 2025STAR-1: Safer Alignment of Reasoning LLMs with 1K Datacited by 67
  11. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  12. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
38 more
  1. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Modelscited by 43
  2. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 42
  3. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
  4. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 35
  5. 2025A Survey of Direct Preference Optimizationcited by 35
  6. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  7. 2025Language Model Personalization via Reward Factorizationcited by 27
  8. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 27
  9. 2024Learning to Summarize from LLM-generated Feedbackcited by 24
  10. 2024Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalizationcited by 23
  11. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  12. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  13. 2025Robust Multi-Objective Preference Alignment with Online DPOcited by 21
  14. 2025Rejected Dialects: Biases Against African American Language in Reward Modelscited by 20
  15. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  16. 2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 15
  17. 2025Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignmentcited by 15
  18. 2024Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learningcited by 13
  19. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Modelscited by 13
  20. 2024Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspectivecited by 12
  21. 2025Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgmentcited by 11
  22. 2025Preemptive Detection and Steering of LLM Misalignment via Latent Reachabilitycited by 10
  23. 2026Reward-free Alignment for Conflicting Objectivescited by 10
  24. 2024Regularized Conditional Diffusion Model for Multi-Task Preference Alignmentcited by 9
  25. 2024Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Promptscited by 9
  26. 2024Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Processcited by 8
  27. 2024SudoLM: Learning Access Control of Parametric Knowledge with Authorization Alignmentcited by 8
  28. 2025ARMOR: Aligning Secure and Safe Large Language Models via Meticulous Reasoningcited by 8
  29. 2025Talk to Your Slides: High-Efficiency Slide Editing via Language-Driven Structured Data Manipulationcited by 7
  30. 2025OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignmentcited by 7
  31. 2025Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignmentcited by 7
  32. 2024BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalizationcited by 6
  33. 20242D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervisioncited by 6
  34. 2025Adaptive Helpfulness-Harmlessness Alignment with Preference Vectorscited by 6
  35. 2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Modelscited by 6
  36. 2025AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Modelscited by 6
  37. 2025Read the Scene, Not the Script: Outcome-Aware Safety for LLMscited by 6
  38. 2024COS-DPO: Conditioned One-Shot Multi-Objective Fine-Tuning Frameworkcited by 5

50 most cited of 105 All 105 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MixDPO: Modeling Preference Strength for Pluralistic Alignment
  2. 2025Beyond Ordinal Preferences: Why Alignment Needs Cardinal Human Feedback
  3. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?
  4. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback
  5. 2025Direct Alignment with Heterogeneous Preferences
  6. 2026Multi-Objective Exploration and Preference Optimization via Mutual Information
  7. 2024SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling
  8. 2024Geometric-Averaged Preference Optimization for Soft Preference Labels