mlsanityData to 7 Oct 2026

Panacea: Pareto Alignment via Preference Adaptation for LLMs

arXiv 2402.02030 ↗First posted 3 Feb 2024cs.CLcited by 78

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Current methods for large language model alignment typically use scalar human preference labels. However, this convention tends to oversimplify the multi-dimensional and heterogeneous nature of human preferences, leading to reduced expressivity and even misalignment. This paper presents Panacea, an innovative approach that reframes alignment as a multi-dimensional preference optimization problem. Panacea trains a single model capable of adapting online and Pareto-optimally to diverse sets of preferences without the need for further tuning. A major challenge here is using a low-dimensional preference vector to guide the model's behavior, despite it being governed by an overwhelmingly large number of parameters. To address this, Panacea is designed to use singular value decomposition (SVD)-based low-rank adaptation, which allows the preference vector to be simply injected online as singular values. Theoretically, we prove that Panacea recovers the entire Pareto front with common loss aggregation methods under mild conditions. Moreover, our experiments demonstrate, for the first time, the feasibility of aligning a single LLM to represent an exponentially vast spectrum of human preferences through various optimization methods. Our work marks a step forward in effectively and efficiently aligning models to diverse and intricate human preferences in a controllable and Pareto-optimal manner.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 67 references that are in the map. First posted on arXiv 3 Feb 2024.

Builds on

41 of 67 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023On Diversified Preferences of Large Language Model Alignment12 Dec
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 38

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2021Convex optimizationcited by 2,479
  10. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  11. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
26 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  4. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  5. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  6. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  7. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  8. 2023AI Alignment: A Comprehensive Surveycited by 256
  9. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  10. 2019Pareto Multi-Task Learningcited by 224
  11. 2023Nash Learning from Human Feedbackcited by 220
  12. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 166
  13. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  14. 2020Learning the Pareto Front with Hypernetworkscited by 122
  15. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 107
  16. 2022Multi-Objective GFlowNetscited by 100
  17. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 87
  18. 2023Non-Vacuous Generalization Bounds for Large Language Modelscited by 53
  19. 2020Controllable Pareto Multi-Task Learningcited by 51
  20. 2022PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithmcited by 51
  21. 2022Pareto Set Learning for Expensive Multi-Objective Optimizationcited by 51
  22. 2023Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspectivecited by 51
  23. 2023Sample-efficient Multi-objective Molecular Optimization with GFlowNetscited by 48
  24. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Modelcited by 37
  25. 2024Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generationcited by 35
  26. 2023Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferencescited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 34; 2026: 27.

78 papers.

Sort:

By area:

  1. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 105
  2. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  3. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 42
  4. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 39
  5. 2025A Survey of Direct Preference Optimizationcited by 35
  6. 2024Stepwise Alignment for Constrained Language Model Policy Optimizationcited by 30
  7. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMscited by 29
  8. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 27
  9. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  10. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Modelscited by 22
  11. 2024MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximationcited by 20
  12. 2025Understanding and Mitigating Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundarycited by 17
38 more
  1. 2025Enhancing Intent Understanding for Ambiguous prompt: A Human-Machine Co-Adaption Strategycited by 16
  2. 2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 15
  3. 2025Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignmentcited by 15
  4. 2024Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferencescited by 13
  5. 2024Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learningcited by 13
  6. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Modelscited by 13
  7. 2025Learning Safety Constraints for Large Language Modelscited by 13
  8. 2025CoSteer: Collaborative Decoding-Time Personalization via Local Delta Steeringcited by 12
  9. 2025Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learningcited by 11
  10. 2024LibMOON: A Gradient-based MultiObjective OptimizatioN Library in PyTorchcited by 10
  11. 2025Game Theory Meets LLM and Agentic AI: Reimagining Cybersecurity for the Age of Intelligent Threatscited by 10
  12. 2024Regularized Conditional Diffusion Model for Multi-Task Preference Alignmentcited by 9
  13. 2024Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Promptscited by 9
  14. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 8
  15. 2025OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignmentcited by 7
  16. 2024BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalizationcited by 6
  17. 2025Adaptive Helpfulness-Harmlessness Alignment with Preference Vectorscited by 6
  18. 2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Modelscited by 6
  19. 2025iDSE: Navigating Design Space Exploration in High-Level Synthesis Using LLMscited by 6
  20. 2025T-POP: Test-Time Personalization with Online Preference Feedbackcited by 6
  21. 2024Challenges and Future Directions of Data-Centric AI Alignmentcited by 5
  22. 2024Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimizationcited by 5
  23. 2025REWARD CONSISTENCY: Improving Multi-Objective Alignment from a Data-Centric Perspectivecited by 5
  24. 2024Prior Constraints-based Reward Model Training for Aligning Large Language Modelscited by 4
  25. 2024Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processescited by 4
  26. 2025Generative Psycho-Lexical Approach for Constructing Value Systems in Large Language Modelscited by 4
  27. 2025ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models using Pareto High-quality Datacited by 4
  28. 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 4
  29. 2025Vulnerability Mitigation for Safety-Aligned Language Models via Debiasingcited by 3
  30. 2025GRAVITY: A Framework for Personalized Text Generation via Profile-Grounded Synthetic Preferencescited by 3
  31. 2026UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignmentcited by 3
  32. 2026CHASE: Adversarial Red-Blue Teaming for Improving LLM Safety using Reinforcement Learningcited by 3
  33. 2024The Superalignment of Superhuman Intelligence with Large Language Modelscited by 2
  34. 2025e-SimFT: Alignment of Generative Models with Simulation Feedback for Pareto-Front Design Explorationcited by 2
  35. 2025PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimizationcited by 2
  36. 2025Review of Hallucination Understanding in Large Language and Vision Modelscited by 2
  37. 2025GTAlign: Game-Theoretic Alignment of LLM Assistants for Social Welfarecited by 2
  38. 2025Precise Attribute Intensity Control in Large Language Models via Targeted Representation Editingcited by 2

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
  2. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  3. 2025ComPO: Preference Alignment via Comparison Oracles
  4. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  5. 2026Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment
  6. 2026A Zeroth-Order Paradigm for LLM Preference Alignment
  7. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
  8. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques