MINT: Min-Selection Preference Distillation for Balanced Multi-Objective Alignment
On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods
Abstract (arXiv)
Aligning a language agent to several objectives at once is a persistent failure mode of preference-based training: when objectives are combined additively, optimization collapses onto whichever is cheapest to improve and sacrifices the rest, so a support agent learns to sound warm while giving no real help. The root issue is that an additive reward has no notion of balance. We introduce Mint (MIN-selection preference disTillation), a one-line change to preference distillation: rather than ranking sampled candidates by a weighted sum of rewards, we rank them by their weakest objective, distilling the best-balanced candidate over the most lopsided one with an unchanged DPO objective. This is the p -> negative infinity limit of a generalized-mean family spanning additive to worst-case selection. Across cooperative emotional support and adversarial negotiation, min-selection lifts both objectives while sharply cutting their imbalance; on emotional support it raises the weaker axis from 0.37 to 0.64 (p < 10^-40), surpassing human experts and persisting across full multi-turn rollouts. A turn-by-turn analysis yields our central finding: min-selection corrects imbalance in proportion to how imbalanced the reference policy is, and its benefit endures over an interaction precisely as long as that imbalance does.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 12 of its 15 references that are in the map. First posted on arXiv 14 Aug 2026.
Builds on
12 of 15 references.
Closest ancestors · 5
Same direction, newest first
- 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
- 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
- 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
- 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
- 2022Training language models to follow instructions with human feedback4 Mar
Other references · 7
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
- 2021Towards Emotional Support Dialog Systemscited by 294
- 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
- 2018Decoupling Strategy and Generation in Negotiation Dialoguescited by 156
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
- 2025REWARD CONSISTENCY: Improving Multi-Objective Alignment from a Data-Centric Perspective
- 2025UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality
- 2026Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansion
- 2026Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment
- 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards
- 2024MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization
- 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models