mlsanityData to 7 Oct 2026

Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibration

arXiv 2402.16030 ↗First posted 25 Feb 2024cs.CL cs.AIcited by 13

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

While Reinforcement Learning from Human Feedback (RLHF) significantly enhances the generation quality of Large Language Models (LLMs), recent studies have raised concerns regarding the complexity and instability associated with the Proximal Policy Optimization (PPO) algorithm, proposing a series of order-based calibration methods as viable alternatives. This paper delves further into current order-based methods, examining their inefficiencies in utilizing reward values and addressing misalignment issues. Building upon these findings, we propose a novel Value-based CaliBration (VCB) method to better align LLMs with human preferences. Experimental results demonstrate that VCB surpasses existing alignment methods on AI assistant and summarization datasets, providing impressive generalizability, robustness, and stability in diverse settings.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 29 references that are in the map. First posted on arXiv 25 Feb 2024.

Builds on

22 of 29 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  2. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  3. 2023Preference Ranking Optimization for Human Alignment30 Jun
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 18

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
6 more
  1. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  2. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  3. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  4. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  5. 2020Revisiting Design Choices in Proximal Policy Optimizationcited by 38
  6. 2021Boosting the Speed of Entity Alignment 10*: Dual Attention Matching Network with Normalized Hard Sample Miningcited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 2; 2026: 5.

13 papers.

Sort:

By area:

  1. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 160
  2. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  3. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  4. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  5. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  6. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  7. 2025Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoningcited by 14
  8. 2025Pretrain Value, Not Reward: Decoupled Value Policy Optimizationcited by 6
  9. 2026Coverage Improvement and Fast Convergence of On-policy Preference Learningcited by 5
  10. 2026Diffusion Reinforcement Learning via Centered Reward Distillationcited by 3
  11. 2026Learning to Prompt: Improving Student Engagement with Adaptive LLM-based High-School Tutoringcited by 1
  12. 2026Pareto-Optimal Offline Reinforcement Learning via Smooth Tchebycheff Scalarizationcited by 0
1 more
  1. 2026Configurable Reward Model for Balanced Safety Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach
  2. 2024Value Augmented Sampling for Language Model Alignment and Personalization
  3. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment
  4. 2024Towards Efficient Exact Optimization of Language Model Alignment
  5. 2023On Diversified Preferences of Large Language Model Alignment
  6. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  7. 2025Doubly Robust Alignment for Large Language Models
  8. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback