mlsanityData to 7 Oct 2026

The Trickle-down Impact of Reward (In-)consistency on RLHF

arXiv 2309.16155 ↗First posted 28 Sep 2023cs.CL cs.LGcited by 29

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Standard practice within Reinforcement Learning from Human Feedback (RLHF) involves optimizing against a Reward Model (RM), which itself is trained to reflect human preferences for desirable generations. A notable subject that is understudied is the (in-)consistency of RMs -- whether they can recognize the semantic changes to different prompts and appropriately adapt their reward assignments -- and their impact on the downstream RLHF model. In this paper, we visit a series of research questions relevant to RM inconsistency: (1) How can we measure the consistency of reward models? (2) How consistent are the existing RMs and how can we improve them? (3) In what ways does reward inconsistency influence the chatbots resulting from the RLHF model training? We propose Contrast Instructions -- a benchmarking strategy for the consistency of RM. Each example in Contrast Instructions features a pair of lexically similar instructions with different ground truth responses. A consistent RM is expected to rank the corresponding instruction and response higher than other combinations. We observe that current RMs trained with the standard ranking objective fail miserably on Contrast Instructions compared to average humans. To show that RM consistency can be improved efficiently without using extra training budget, we propose two techniques ConvexDA and RewardFusion, which enhance reward consistency through extrapolation during the RM training and inference stage, respectively. We show that RLHF models trained with a more consistent RM yield more useful responses, suggesting that reward inconsistency exhibits a trickle-down effect on the downstream RLHF process.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 61 of its 82 references that are in the map. First posted on arXiv 28 Sep 2023.

Builds on

61 of 82 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  2. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  3. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  4. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  5. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  7. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  8. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 53

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2017Billion-scale similarity search with GPUscited by 2,477
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  11. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  12. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
38 more
  1. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  2. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  3. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  4. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  5. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  6. 2018Generating Natural Language Adversarial Examplescited by 422
  7. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  8. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  9. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 304
  10. 2023Pretraining Language Models with Human Preferencescited by 245
  11. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  12. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  13. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  14. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 203
  15. 2019Certified Robustness to Adversarial Word Substitutionscited by 164
  16. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfercited by 147
  17. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  18. 2019Synthetic QA Corpora Generation with Roundtrip Consistencycited by 143
  19. 2020Contextual Diversity for Active Learningcited by 139
  20. 2018Categorizing Variants of Goodhart's Lawcited by 132
  21. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  22. 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 107
  23. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  24. 2019Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systemscited by 99
  25. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 96
  26. 2019Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagationcited by 87
  27. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  28. 2020Logic-Guided Data Augmentation and Regularization for Consistent Question Answeringcited by 77
  29. 2021Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selectioncited by 75
  30. 2019Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanationscited by 71
  31. 2020Frequency-Guided Word Substitutions for Detecting Textual Adversarial Examplescited by 54
  32. 2021Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!cited by 51
  33. 2022SemAttack: Natural Textual Attacks via Different Semantic Spacescited by 42
  34. 2022Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inferencecited by 40
  35. 2019Natural Language Adversarial Defense through Synonym Encodingcited by 39
  36. 2022On the Evaluation Metrics for Paraphrase Generationcited by 32
  37. 2019Does Data Augmentation Lead to Positive Margin?cited by 18
  38. 2020A Sweet Rabbit Hole by DARCY: Using Honeypots to Detect Universal Trigger's Adversarial Attackscited by 14

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 11; 2025: 10; 2026: 4.

29 papers.

Sort:

By area:

  1. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  2. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  3. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  4. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  5. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 92
  6. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  7. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimizationcited by 67
  8. 2024Improving Reward Models with Synthetic Critiquescited by 44
  9. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
  10. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  11. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  12. 2024Transforming and Combining Rewards for Aligning Large Language Modelscited by 24
17 more
  1. 2024Tur[k]ingBench: A Challenge Benchmark for Web Agentscited by 18
  2. 2025Robust Reward Modeling via Causal Rubricscited by 18
  3. 2025Learning to summarize user information for personalized reinforcement learning from human feedbackcited by 9
  4. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 8
  5. 2024Rethinking the Role of Proxy Rewards in Language Model Alignmentcited by 7
  6. 2024Towards Reliable Alignment: Uncertainty-aware RLHFcited by 7
  7. 2024It Takes Two: On the Seamlessness between Reward and Policy Model in RLHFcited by 5
  8. 2025Information-Theoretic Reward Decomposition for Generalizable RLHFcited by 5
  9. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPOcited by 5
  10. 2026Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agentscited by 4
  11. 2025Examining Alignment of Large Language Models through Representative Heuristics: The Case of Political Stereotypescited by 2
  12. 2025Take Off the Training Wheels Progressive In-Context Learning for Effective Alignmentcited by 2
  13. 2025Tiny Reward Modelscited by 0
  14. 2025bi-GRPO: Bidirectional Optimization for Jailbreak Backdoor Injection on LLMscited by 0
  15. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 0
  16. 2026Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoderscited by 0
  17. 2026RLVR2: Reinforcement Learning with Verifiable Rubric-based Rankingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  2. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  3. 2024Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards
  4. 2024How to Evaluate Reward Models for RLHF
  5. 2024The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models
  6. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  7. 2025Reward Model Overoptimisation in Iterated RLHF
  8. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective