mlsanityData to 7 Oct 2026

Robust Multi-Objective Controlled Decoding of Large Language Models

arXiv 2503.08796 ↗First posted 11 Mar 2025cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

We introduce Robust Multi-Objective Decoding (RMOD), a novel inference-time algorithm that robustly aligns Large Language Models (LLMs) to multiple human objectives (e.g., instruction-following, helpfulness, safety) by maximizing the worst-case rewards. RMOD formulates the robust decoding problem as a maximin two-player game between adversarially computed reward weights and the sampling policy, solvable through a Nash equilibrium. We demonstrate that this game reduces to a convex optimization problem to identify the worst-case reward weights, with the optimal sampling policy analytically derived. For practical applications, we propose an efficient algorithm of RMOD tailored for contemporary LLMs, introducing minimal computational overhead compared to standard non-robust Controlled Decoding methods. Experimental results across a range of popular alignment datasets with up to 10 objectives show the effectiveness of RMOD and its distilled version, consistently outperforming baselines in worst-case rewards and win rates.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 63 of its 89 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

63 of 89 references.

References · 63

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  12. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
38 more
  1. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  2. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  3. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  4. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  5. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  6. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  7. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  8. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  9. 2023Mitigating the Alignment Tax of RLHFcited by 181
  10. 2024A Roadmap to Pluralistic Alignmentcited by 175
  11. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  12. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  13. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  14. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  15. 2023Controlled Decoding from Language Modelscited by 133
  16. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  17. 2023Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Predictioncited by 113
  18. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  19. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 107
  20. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  21. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  22. 2023Aligning Language Models to User Opinionscited by 98
  23. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  24. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  25. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  26. 2024Tuning Language Models by Proxycited by 88
  27. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 87
  28. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85
  29. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
  30. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 71
  31. 2024Decoding-time Realignment of Language Modelscited by 67
  32. 2022Gradient-Based Constrained Sampling from Language Modelscited by 64
  33. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
  34. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Modelscited by 61
  35. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  36. 2023On the steerability of large language models toward data-driven personascited by 53
  37. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 53
  38. 2022PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithmcited by 51

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  2. 2025Bounded Rationality for LLMs: Satisficing Alignment at Inference-Timecited by 8
  3. 2026Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspectivecited by 2
  4. 2025SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferencescited by 1
  5. 2025ProSocialAlign: Preference Conditioned Test Time Alignment in Language Modelscited by 1
  6. 2025MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selectioncited by 0
  7. 2026One Model for All: Multi-Objective Controllable Language Modelscited by 0
  8. 2026MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Modelscited by 0