mlsanityData to 7 Oct 2026

Multi-Objective Exploration and Preference Optimization via Mutual Information

arXiv 2607.01392 ↗First posted 1 Jul 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models with diverse and heterogeneous human values requires multi-objective alignment methods to effectively trade off conflicting preference dimensions. Current methods achieve this trade-off by training policies conditioned on preference vectors and leveraging online direct preference optimization. However, exploration uncertainty can cause the reward distributions of responses generated under different preference vectors to overlap, and the generated responses may fail to effectively align with the corresponding preference vectors. In this paper, we propose Multi-Objective Exploration and Preference Optimization via Mutual Information (MI-EPO), an information-theoretic framework. It unifies multi-objective exploration and alignment by maximizing the joint conditional mutual information among generated responses, preference feedback, and preference vectors. By incorporating a probabilistic routing mechanism, MI-EPO naturally decomposes objective alignment and preference-aware exploration, encouraging the model to generate responses that are distinguishable and aligned with different preference conditions. Experiments on safe alignment and helpful assistant tasks show that MI-EPO significantly improves the alignment between generated responses and preference vectors, makes the outputs more controllable, and achieves stable trade-offs across multiple objectives.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 49 references that are in the map. First posted on arXiv 1 Jul 2026.

Builds on

42 of 49 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignment10 Feb
  2. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models16 May
  3. 2025InfoPO: On Mutual Information Maximization for Large Language Model Alignment13 May
  4. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model6 May
  5. 2025REWARD CONSISTENCY: Improving Multi-Objective Alignment from a Data-Centric Perspective15 Apr
  6. 2025Robust Multi-Objective Preference Alignment with Online DPO1 Mar
  7. 2025Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment20 Feb
  8. 2024Decoding-Time Language Model Alignment with Multiple Objectives27 Jun
  9. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  10. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models25 Mar
  11. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  12. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
2 more
  1. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  8. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  9. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  10. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  11. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 87
  12. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
16 more
  1. 2022Pareto Set Learning for Expensive Multi-Objective Optimizationcited by 51
  2. 2024Smooth Tchebycheff Scalarization for Multi-Objective Optimizationcited by 50
  3. 2021Multi-Objective Meta Learningcited by 39
  4. 2021Conditional Contrastive Learning for Improving Fairness in Self-Supervised Learningcited by 25
  5. 2022Conditional Contrastive Learning with Kernelcited by 22
  6. 2024TeleChat Technical Reportcited by 15
  7. 2024Few for Many: Tchebycheff Set Scalarization for Many-Objective Optimizationcited by 15
  8. 2024Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferencescited by 13
  9. 2024Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Mergingcited by 13
  10. 2024Tele-FLM Technical Reportcited by 12
  11. 2026Real-Time Aligned Reward Model beyond Semanticscited by 12
  12. 202452B to 1T: Lessons Learned via Tele-FLM Seriescited by 11
  13. 2025Technical Report of TeleChat2, TeleChat2.5 and T1cited by 10
  14. 2024Efficient Pareto Manifold Learning with Low-Rank Structurecited by 9
  15. 2025Training Report of TeleChat3-MoEcited by 4
  16. 2026Hallucinations as Orthogonal Noise: Inference-Time Manifold Alignment via Dynamic Contextual Orthogonalizationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Models
  2. 2025Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
  3. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models
  4. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment
  5. 2026Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment
  6. 2025MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment
  7. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment
  8. 2024SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling