mlsanityData to 14 Oct 2026

InfoPO: On Mutual Information Maximization for Large Language Model Alignment

arXiv 2505.08507 ↗First posted 13 May 2025cs.LGcited by 7

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We study the post-training of large language models (LLMs) with human preference data. Recently, direct preference optimization and its variants have shown considerable promise in aligning language models, eliminating the need for reward models and online sampling. Despite these benefits, these methods rely on explicit assumptions about the Bradley-Terry (BT) model, which makes them prone to overfitting and results in suboptimal performance, particularly on reasoning-heavy tasks. To address these challenges, we propose a principled preference fine-tuning algorithm called InfoPO, which effectively and efficiently aligns large language models using preference data. InfoPO eliminates the reliance on the BT model and prevents the likelihood of the chosen response from decreasing. Extensive experiments confirm that InfoPO consistently outperforms established baselines on widely used open benchmarks, particularly in reasoning tasks.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 55 references that are in the map. First posted on arXiv 13 May 2025.

Builds on

40 of 55 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters2 Feb
  2. 2024How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspective14 Oct
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  4. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  5. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  6. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  7. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  8. 2024Noise Contrastive Alignment of Language Models with Explicit Rewards8 Feb
  9. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  11. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  12. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep

Other references · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
16 more
  1. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  2. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  3. 2018MINE: Mutual Information Neural Estimationcited by 683
  4. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  5. 2019On Variational Bounds of Mutual Informationcited by 559
  6. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  7. 2019Variational Information Distillation for Knowledge Transfercited by 324
  8. 2019On Mutual Information Maximization for Representation Learningcited by 316
  9. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
  10. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  11. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
  12. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  13. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  14. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  15. 2021Conditional Contrastive Learning for Improving Fairness in Self-Supervised Learningcited by 25
  16. 2023A General Offline Reinforcement Learning Framework for Interactive Recommendationcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  2. 2025What's on My Network? Using Large Language Models to Identify Real-World IoT Devices at Scalecited by 1
  3. 2025Negative-Guided Subject Fidelity Optimization for Zero-Shot Subject-Driven Generationcited by 0
  4. 2025Closing the Expression Gap in LLM Instructions via Socratic Questioningcited by 0
  5. 2026Intrinsic Mutual Information as a Modulator for Preference Optimizationcited by 0
  6. 2026Multi-Objective Exploration and Preference Optimization via Mutual Informationcited by 0
  7. 2026MASCRDM: Multi-Agent System for Compliance Risk Detection and Mitigation in Training Process of Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  3. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  4. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  5. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
  6. 2025ComPO: Preference Alignment via Comparison Oracles
  7. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  8. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models