mlsanityData to 14 Oct 2026

Noise Contrastive Alignment of Language Models with Explicit Rewards

arXiv 2402.05369 ↗First posted 8 Feb 2024cs.LG cs.CLcited by 85

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

User intentions are typically formalized as evaluation rewards to be maximized when fine-tuning language models (LMs). Existing alignment methods, such as Direct Preference Optimization (DPO), are mainly tailored for pairwise preference data where rewards are implicitly defined rather than explicitly given. In this paper, we introduce a general framework for LM alignment, leveraging Noise Contrastive Estimation (NCE) to bridge the gap in handling reward datasets explicitly annotated with scalar evaluations. Our framework comprises two parallel algorithms, NCA and InfoNCA, both enabling the direct extraction of an LM policy from reward data as well as preference data. Notably, we show that the DPO loss is a special case of our proposed InfoNCA objective under pairwise preference settings, thereby integrating and extending current alignment theories. By comparing NCA and InfoNCA, we demonstrate that the well-observed decreasing-likelihood trend of DPO/InfoNCA is caused by their focus on adjusting relative likelihood across different responses. In contrast, NCA optimizes the absolute likelihood for each response, thereby effectively preventing the chosen likelihood from decreasing. We evaluate our methods in both reward and preference settings with Mistral-8*7B and 7B models. Experiments suggest that InfoNCA/NCA surpasses various preference baselines when reward datasets are available. We also find NCA significantly outperforms DPO in complex reasoning tasks like math and coding.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 53 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

34 of 53 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  3. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  4. 2023Preference Ranking Optimization for Human Alignment30 Jun
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
17 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  6. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  7. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  9. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  10. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  11. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  12. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 318
  13. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  14. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 192
  15. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 149
  16. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  17. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 105

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 26; 2025: 44; 2026: 15.

85 papers.

Sort:

By area:

  1. 2025Process Reinforcement through Implicit Rewardscited by 504
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  3. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  4. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  5. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  6. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  7. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  8. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 72
  9. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 71
  10. 2024On Softmax Direct Preference Optimization for Recommendationcited by 64
  11. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 54
  12. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
38 more
  1. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 51
  2. 2024Binary Classifier Optimization for Large Language Model Alignmentcited by 45
  3. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
  4. 2025A Survey of Direct Preference Optimizationcited by 36
  5. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 31
  6. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 27
  7. 2024Modality-Fair Preference Optimization for Trustworthy MLLM Alignmentcited by 26
  8. 2024Aligning Target-Aware Molecule Diffusion Models with Exact Energy Optimizationcited by 25
  9. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameterscited by 25
  10. 2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 25
  11. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forwardcited by 24
  12. 2024CARMO: Dynamic Criteria Generation for Context-Aware Reward Modellingcited by 22
  13. 2024TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Treescited by 19
  14. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 19
  15. 2025UltraIF: Advancing Instruction Following from the Wildcited by 18
  16. 2025Visual Generation Without Guidancecited by 17
  17. 2024Toward Guidance-Free AR Visual Generation via Condition Contrastive Alignmentcited by 14
  18. 2025Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimizationcited by 14
  19. 2025Dual-IPO: Dual-Iterative Preference Optimization for Text-to-Video Generationcited by 13
  20. 2025ComPO: Preference Alignment via Comparison Oraclescited by 13
  21. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Controlcited by 12
  22. 2025A Survey on Mathematical Reasoning and Optimization with Large Language Modelscited by 12
  23. 2025Discrete Diffusion Trajectory Alignment via Stepwise Decompositioncited by 12
  24. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMscited by 10
  25. 2025Teaching Your Models to Understand Code via Focal Preference Alignmentcited by 10
  26. 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrastscited by 9
  27. 2024Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Modelcited by 9
  28. 2025Mining Intrinsic Rewards from LLM Hidden States for Efficient Best-of-N Samplingcited by 9
  29. 2024Length Desensitization in Direct Preference Optimizationcited by 8
  30. 2025Enhancing Reasoning Abilities of Small LLMs with Cognitive Alignmentcited by 7
  31. 2025InfoPO: On Mutual Information Maximization for Large Language Model Alignmentcited by 7
  32. 2025Bias Fitting to Mitigate Length Bias of Reward Model in RLHFcited by 7
  33. 2025Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimizationcited by 7
  34. 2025Efficient Switchable Safety Control in LLMs via Magic-Token-Guided Co-Trainingcited by 7
  35. 2025SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Modelscited by 7
  36. 2025Disentangling Length Bias In Preference Learning Via Response-Conditioned Modelingcited by 6
  37. 2025Direct Value Optimization: Improving Chain-of-Thought Reasoning in LLMs with Refined Valuescited by 6
  38. 2025Improving Protein Sequence Design through Designability Preference Optimizationcited by 6

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization
  2. 2025Towards Reliable, Uncertainty-Aware Alignment
  3. 2026Unbiased Alignment for Large Language Models with Noisy Preferences
  4. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
  5. 2024ROPO: Robust Preference Optimization for Large Language Models
  6. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  7. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  8. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment