mlsanityData to 7 Oct 2026

Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards

arXiv 2306.04488 ↗First posted 7 Jun 2023cs.LG cs.AI cs.CVcited by 232

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Foundation models are first pre-trained on vast unsupervised datasets and then fine-tuned on labeled data. Reinforcement learning, notably from human feedback (RLHF), can further align the network with the intended usage. Yet the imperfections in the proxy reward may hinder the training and lead to suboptimal results; the diversity of objectives in real-world tasks and human opinions exacerbate the issue. This paper proposes embracing the heterogeneity of diverse rewards by following a multi-policy strategy. Rather than focusing on a single a priori reward, we aim for Pareto-optimal generalization across the entire space of preferences. To this end, we propose rewarded soup, first specializing multiple networks independently (one for each proxy reward) and then interpolating their weights linearly. This succeeds empirically because we show that the weights remain linearly connected when fine-tuned on diverse rewards from a shared pre-trained initialization. We demonstrate the effectiveness of our approach for text-to-text (summarization, Q&A, helpful assistant, review), text-image (image captioning, text-to-image generation, visual grounding, VQA), and control (locomotion) tasks. We hope to enhance the alignment of deep models, and how they interact with the world in all its diversity.

Key facts. Cited by 232 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 139 of its 217 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

139 of 217 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  3. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  4. 2023Rewarding Chatbots for Real-World Engagement with Millions of Users10 Mar
  5. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  7. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 131

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2023GPT-4 Technical Reportcited by 18,132
  9. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  10. 2022Training language models to follow instructions with human feedbackcited by 15,712
  11. 2017Proximal Policy Optimization Algorithmscited by 14,441
  12. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
38 more
  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2016Deep Learning with Differential Privacycited by 2,682
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  13. 2020Learning to summarize from human feedbackcited by 2,562
  14. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  15. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  16. 2016Concrete Problems in AI Safetycited by 1,582
  17. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  18. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  19. 2014How transferable are features in deep neural networks?cited by 1,216
  20. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  21. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  22. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  23. 2022Editing Models with Task Arithmeticcited by 1,076
  24. 2016Modeling Context in Referring Expressionscited by 1,054
  25. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  26. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  27. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  28. 2021Ethical and social risks of harm from Language Modelscited by 957
  29. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  30. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  31. 2021Robust fine-tuning of zero-shot modelscited by 785
  32. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  33. 2023Whose Opinions Do Language Models Reflect?cited by 759
  34. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  35. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  36. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  37. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  38. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627

50 most cited of 131 All 131 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 88; 2025: 80; 2026: 56.

232 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  3. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  4. 2023Nash Learning from Human Feedbackcited by 220
  5. 2023Mitigating the Alignment Tax of RLHFcited by 181
  6. 2024A Roadmap to Pluralistic Alignmentcited by 175
  7. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  8. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  9. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  10. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  11. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
  12. 2024Representation Surgery for Multi-Task Model Mergingcited by 95
38 more
  1. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  2. 2023Deep Model Fusion: A Surveycited by 89
  3. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  4. 2025Test-time Alignment of Diffusion Models without Reward Over-optimizationcited by 89
  5. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  6. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
  7. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
  8. 2024Decoding-time Realignment of Language Modelscited by 67
  9. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  10. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  11. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  12. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  13. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  14. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 53
  15. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  16. 2023Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languagescited by 52
  17. 2024What Matters for Model Merging at Scale?cited by 52
  18. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  19. 2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 47
  20. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  21. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Modelscited by 43
  22. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  23. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  24. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 42
  25. 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Modelscited by 40
  26. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
  27. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 39
  28. 2024Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generationcited by 35
  29. 2024Consistency-diversity-realism Pareto fronts of conditional image generative modelscited by 35
  30. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 35
  31. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  32. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 34
  33. 2024Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Modelscited by 32
  34. 2024BadMerging: Backdoor Attacks Against Model Mergingcited by 32
  35. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedbackcited by 30
  36. 2024Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Modelscited by 27
  37. 2025Language Model Personalization via Reward Factorizationcited by 27
  38. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 27

50 most cited of 232 All 232 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimization
  2. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferences
  3. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning
  4. 2024Reward Generalization in RLHF: A Topological Perspective
  5. 2026Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment
  6. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  7. 2025Learning Pareto-Optimal Rewards from Noisy Preferences: A Framework for Multi-Objective Inverse Reinforcement Learning
  8. 2024MaxMin-RLHF: Alignment with Diverse Human Preferences