mlsanityData to 7 Oct 2026

Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Prompts

arXiv 2408.05094 ↗First posted 9 Aug 2024cs.CLcited by 9

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The task of multi-objective alignment aims at balancing and controlling the different alignment objectives (e.g., helpfulness, harmlessness and honesty) of large language models to meet the personalized requirements of different users. However, previous methods tend to train multiple models to deal with various user preferences, with the number of trained models growing linearly with the number of alignment objectives and the number of different preferences. Meanwhile, existing methods are generally poor in extensibility and require significant re-training for each new alignment objective considered. Considering the limitation of previous approaches, we propose MCA (Multi-objective Contrastive Alignemnt), which constructs an expert prompt and an adversarial prompt for each objective to contrast at the decoding time and balances the objectives through combining the contrast. Our approach is verified to be superior to previous methods in obtaining a well-distributed Pareto front among different alignment objectives.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 56 references that are in the map. First posted on arXiv 9 Aug 2024.

Builds on

38 of 56 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment28 May
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models25 Mar
  4. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment29 Feb
  5. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  6. 2024Decoding-time Realignment of Language Models5 Feb
  7. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs3 Feb
  8. 2024Tradeoffs Between Alignment and Helpfulness in Language Models with Steering Methods29 Jan
  9. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 217 Nov
  10. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Training7 Nov
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  12. 2023LIMA: Less Is More for Alignment18 May
1 more
  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  7. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  8. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  9. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  10. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  11. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  12. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
13 more
  1. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  2. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  3. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  4. 2024A Roadmap to Pluralistic Alignmentcited by 175
  5. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  6. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  7. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  8. 2023Alleviating Hallucinations of Large Language Models through Induced Hallucinationscited by 78
  9. 2023Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decodingcited by 42
  10. 2024ROSE Doesn't Do That: Boosting the Safety of Instruction-Tuned Large Language Models with Reverse Prompt Contrastive Decodingcited by 32
  11. 2024Parameter-Efficient Detoxification with Contrastive Decodingcited by 18
  12. 2024Distillation Contrastive Decoding: Improving LLMs Reasoning with Contrastive Decoding and Distillationcited by 16
  13. 2024On the Transformations across Reward Model, Parameter Update, and In-Context Promptcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025Learning Safety Constraints for Large Language Modelscited by 13
  2. 2025Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimizationcited by 11
  3. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 8
  4. 2025ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models using Pareto High-quality Datacited by 4
  5. 2025A Survey on Training-free Alignment of Large Language Modelscited by 2
  6. 2026VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Modelscited by 2
  7. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  8. 2026Efficient Low-Resource Language Adaptation via Multi-Source Dynamic Logit Fusioncited by 0
  9. 2026Operational Proto-Introspection in Looped Language Models: Process-Quality Taps, Executable Branching, and the Readout-Control Boundarycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Swiss-Knife: A Framework for Reconfigurable Externalised Multi-Objective Alignment at Decode Time
  2. 2024Decoding-Time Language Model Alignment with Multiple Objectives
  3. 2026MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models
  4. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards
  5. 2024DeAL: Decoding-time Alignment for Large Language Models
  6. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment
  7. 2026Uncovering Cross-Objective Interference in Multi-Objective Alignment
  8. 2025Alignment-Aware Decoding