mlsanityData to 7 Oct 2026

Maximizing Mutual Information Between Prompt and Response Improves LLM Performance With No Additional Data

arXiv 2603.19294 ↗First posted 10 Mar 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

While post-training has successfully improved large language models (LLMs) across a variety of domains, these gains heavily rely on human-labeled data or external verifiers. Existing data has already been exploited, and new data is expensive to collect. Moreover, true intelligence goes far beyond verifiable tasks. Therefore, we need self-improvement frameworks that are less dependent on external signals and more broadly applicable to both verifiable and non-verifiable domains. We propose **Mutual Information Preference Optimization (MIPO)**, a contrastive data augmentation method that constructs preference pairs by generating a positive response conditioning on the correct prompt, and a negative response by conditioning on a random, unrelated prompt. We show that using Direct Preference Optimization to learn from this paired data maximizes pointwise mutual information *under the base LLM* between prompts and model responses. Experiments with with 1-7B parameter Llama and Qwen instruct models show that MIPO achieves 3-16% gains (and 51% increase for Qwen2.5-1.5B-Instruct) on personalization compared to prompting baselines. Surprisingly, MIPO can also be useful in verifiable domains, such as math and multiple-choice question answering, yielding 1-20% gains *without any additional data or external supervision*. These results suggest a promising direction for self-improvement using intrinsic signals derived from contrastive data pairs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 53 of its 72 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

53 of 72 references.

References · 53

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  10. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  12. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
38 more
  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  3. 2019On Variational Bounds of Mutual Informationcited by 550
  4. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  5. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  6. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  7. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  8. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  9. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  10. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  11. 2025Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scalecited by 163
  12. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  13. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 142
  14. 2025Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)cited by 137
  15. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 124
  16. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  17. 2024OLMES: A Standard for Language Model Evaluationscited by 100
  18. 2025Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversitycited by 100
  19. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  20. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  21. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelscited by 64
  22. 2024Active Preference Optimization for Sample Efficient RLHFcited by 61
  23. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  24. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Promptscited by 36
  25. 2024Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talkcited by 34
  26. 2024I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigmcited by 27
  27. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Modelcited by 25
  28. 2025The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gainscited by 25
  29. 2024Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Modelcited by 22
  30. 2025CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactionscited by 20
  31. 2024Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labelscited by 18
  32. 2025Spectrum Tuning: Post-Training for Distributional Coverage and In-Context Steerabilitycited by 17
  33. 2022Contrastive Value Learning: Implicit Models for Simple Offline RLcited by 15
  34. 2025On a Connection Between Imitation Learning and RLHFcited by 13
  35. 2024Varying Shades of Wrong: Aligning LLMs with Wrong Answers Onlycited by 9
  36. 2025Learning to summarize user information for personalized reinforcement learning from human feedbackcited by 9
  37. 2023Automatic Pair Construction for Contrastive Post-trainingcited by 7
  38. 2025Difficulty-Based Preference Data Selection by DPO Implicit Reward Gapcited by 7

50 most cited of 53 All 53 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Systematic Examination of Preference Learning through the Lens of Instruction-Following
  2. 2024Self-Boosting Large Language Models with Synthetic Preference Data
  3. 2025Self-Supervised Prompt Optimization
  4. 2026Towards Self-Robust LLMs: Intrinsic Prompt Noise Resistance via CoIPO
  5. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  6. 2025SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization
  7. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  8. 2025Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning