mlsanityData to 14 Oct 2026

Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment

arXiv 2402.10207 ↗First posted 15 Feb 2024cs.LG cs.AI cs.CLcited by 175

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We consider the problem of multi-objective alignment of foundation models with human preferences, which is a critical step towards helpful and harmless AI systems. However, it is generally costly and unstable to fine-tune large foundation models using reinforcement learning (RL), and the multi-dimensionality, heterogeneity, and conflicting nature of human preferences further complicate the alignment process. In this paper, we introduce Rewards-in-Context (RiC), which conditions the response of a foundation model on multiple rewards in its prompt context and applies supervised fine-tuning for alignment. The salient features of RiC are simplicity and adaptivity, as it only requires supervised fine-tuning of a single foundation model and supports dynamic adjustment for user preferences during inference time. Inspired by the analytical solution of an abstracted convex optimization problem, our dynamic inference-time adjustment method approaches the Pareto-optimal solution for multiple objectives. Empirical evidence demonstrates the efficacy of our method in aligning both Large Language Models (LLMs) and diffusion models to accommodate diverse rewards with only around 10% GPU hours compared with multi-objective RL baseline.

Key facts. Cited by 175 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 55 of its 73 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

55 of 73 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  2. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 52

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  12. 2017Deep reinforcement learning from human preferencescited by 3,912
38 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2021Convex optimizationcited by 2,515
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  7. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  8. 2017Hindsight Experience Replaycited by 1,136
  9. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  11. 2023Instruction Tuning with GPT-4cited by 686
  12. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  13. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  14. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  15. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  16. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 256
  17. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  18. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 242
  19. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  20. 2023Nash Learning from Human Feedbackcited by 224
  21. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  22. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 170
  23. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  24. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 111
  25. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  26. 2019Reward-Conditioned Policiescited by 102
  27. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 90
  28. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  29. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 82
  30. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  31. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 76
  32. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 62
  33. 2020Generalized Hindsight for Reinforcement Learningcited by 51
  34. 2023Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RLcited by 37
  35. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  36. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  37. 2019Policy Continuation with Hindsight Inverse Dynamicscited by 22
  38. 2023Tailoring Self-Rationalizers with Multi-Reward Distillationcited by 21

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 40; 2025: 80; 2026: 55.

175 papers.

Sort:

By area:

  1. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 221
  2. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 146
  3. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  4. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  6. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 107
  7. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  8. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 94
  9. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 60
  10. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  11. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
  12. 2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 46
38 more
  1. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Modelscited by 44
  2. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 43
  3. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  4. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 42
  5. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 39
  6. 2024Consistency-diversity-realism Pareto fronts of conditional image generative modelscited by 36
  7. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 36
  8. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
  9. 2025Language Model Personalization via Reward Factorizationcited by 31
  10. 2025SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMscited by 29
  11. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 28
  12. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
  13. 2024One-Shot Safety Alignment for Large Language Models via Optimal Dualizationcited by 24
  14. 2025Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviewscited by 24
  15. 2025Sotopia-RL: Reward Design for Social Intelligencecited by 24
  16. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  17. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  18. 2025Virtual Agent Economiescited by 21
  19. 2024MAP: Multi-Human-Value Alignment Palettecited by 20
  20. 2024Multi-Objective Alignment of Large Language Models Through Hypervolume Maximizationcited by 20
  21. 2025Pareto Multi-Objective Alignment for Language Modelscited by 20
  22. 2025Imbalanced Gradients in RL Post-Training of Multi-Task LLMscited by 20
  23. 2025From Selection to Generation: A Survey of LLM-based Active Learningcited by 19
  24. 2025Robust Reward Modeling via Causal Rubricscited by 19
  25. 2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 18
  26. 2025Understanding and Mitigating Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundarycited by 18
  27. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  28. 2024ROPO: Robust Preference Optimization for Large Language Modelscited by 16
  29. 2025SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agentscited by 16
  30. 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 16
  31. 2024Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learningcited by 15
  32. 2025FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Userscited by 15
  33. 2025Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generationcited by 14
  34. 2025Drift: Decoding-time Personalized Alignments with Implicit User Preferencescited by 13
  35. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Modelscited by 13
  36. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 13
  37. 2025The Era of Real-World Human Interaction: RL from User Conversationscited by 13
  38. 2024Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspectivecited by 12

50 most cited of 175 All 175 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Preference Learning for AI Alignment: a Causal Perspective
  2. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models
  3. 2025Beyond Monolithic Rewards: A Hybrid and Multi-Aspect Reward Optimization for MLLM Alignment
  4. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization
  5. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models
  6. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  7. 2024Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes
  8. 2024Towards Reliable Alignment: Uncertainty-aware RLHF