mlsanityData to 7 Oct 2026

Reward modeling and RLHF

1,029 papers2 this weekMedium confidence in this grouping: mostly one topic, with blurry edgesMedium confidence in this grouping: mostly one topic, with blurry edges · About

Keywords: reward, human, feedback, rlhf, reinforcement

Key facts. Reward modeling and RLHF: 1,029 papers on arXiv since 2017 (data through 7 Oct 2026). 505 of its 1,029 mapped papers (49%) cite “Deep reinforcement learning from human preferences” (2017, arXiv 1706.03741). 437 of its 1,029 mapped papers (42%) cite “Direct Preference Optimization: Your Language Model is Secretly a Reward Model” (2023, arXiv 2305.18290). 20 new papers 17 Sep–7 Oct.

What’s new

17 Sep – 7 Oct 2026

20 new papers, 17 Sep – 7 Oct; 2 this week.

  1. This week · 2
  2. A Scoping Review and Experimental Study on Reinforcement Learning from Human Feedback for Human-Robot Collaboration
  3. Learning a Ranking from Human Feedback in Log-Concave Random Utility Models
  4. Earlier · 18
  5. Groupwise Distortion Guarantees for Preference-Based Alignment
  6. Bandits via Additive Quantized Representations also fits: Reward hacking and misspecification
  7. Robust Nash Alignment under Preference Uncertainty also fits: LLM preference optimization methods
  8. Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedback
  9. Efficient Multi-Modal Planning with Reward-Guided Preference Optimization for Autonomous Driving
  10. ProgressCompass: Embodied Progress Reward Models Are Lost Without the Right Context also fits: Reward hacking and misspecification
  11. Using Context Is Not Enough: Test-Time Training for Personalized Reward Modeling
  12. ARS: Agentic Reward System for Robot Learning also fits: Reward hacking and misspecification
All 20
  1. RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedback also fits: Process reward models for reasoning
  2. Achieve What You Imagined: Learning to Align Actions with Visual Plans
  3. Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policies
  4. Direct Hidden-State Alignment: Mapping and Controlling Preference Expression in LLMs also fits: LLM preference optimization methods
  5. RMB: Reward Model Boosting Mitigates Reward Hacking
  6. Axiom Satisfiability of Linear Rewards in Alignment
  7. Saturation-Insensitive Dueling Bandits with General Function Approximation also fits: Reward hacking and misspecification
  8. Improving LLM Collaboration via Multi-Agent Preference Learning also fits: Process reward models for reasoning
  9. BiasReducer: Adaptive Bias Mitigation for Reward Models also fits: LLM preference optimization methods
  10. PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning

Foundations

Most cited by papers here

  1. Deep reinforcement learning from human preferences2017cited by 505 here
  2. Direct Preference Optimization: Your Language Model is Secretly a Reward Model2023cited by 437 here
  3. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback2022cited by 335 here
  4. Fine-Tuning Language Models from Human Preferences2019cited by 258 here
  5. Scaling Laws for Reward Model Overoptimization2022cited by 226 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2017Deep reinforcement learning from human preferencescited by 505 here
  2. 2018Reward learning from human preferences and demonstrations in Ataricited by 115 here
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 258 here
  4. 2020Learning to summarize from human feedbackcited by 324 here
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 107 here
  6. 2022Training language models to follow instructions with human feedbackcited by 583 here
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 335 here
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 226 here
  9. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 109 here
  10. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 136 here
  11. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 105 here
  12. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 82 here
  13. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 136 here
  14. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 95 here
10 more it builds on, not in this chain
  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 117 here
  2. 2021A General Language Assistant as a Laboratory for Alignmentcited by 88 here
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 151 here
  4. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 99 here
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 95 here
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 437 here
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 148 here
  8. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 89 here
  9. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 89 here
  10. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 81 here

The 24 most cited of 1,209.

Browse

30 of 1,029, ranked 6–35

  1. 6RewardBench: Evaluating Reward Models for Language Modeling2024136
  2. 7Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback2023136
  3. 8PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training2021117
  4. 9Reward learning from human preferences and demonstrations in Atari2018115
  5. 10Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons2023109
  6. 11WebGPT: Browser-assisted question-answering with human feedback2021107
  7. 12Reward Model Ensembles Help Mitigate Overoptimization2023105
  8. 13RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment202399
  9. 14Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts202495
  10. 15Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint202389
All 30
  1. 16Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking202382
  2. 17Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs202481
  3. 18RLHF Workflow: From Reward Modeling to Online RLHF202480
  4. 19SLiC-HF: Sequence Likelihood Calibration with Human Feedback202378
  5. 20A Long Way to Go: Investigating Length Correlations in RLHF202378
  6. 21Fine-Grained Human Feedback Gives Better Rewards for Language Model Training202376
  7. 22RRHF: Rank Responses to Align Language Models with Human Feedback without tears202370
  8. 23Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs202466
  9. 24WARM: On the Benefits of Weight Averaged Reward Models202465
  10. 25RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style202464
  11. 26RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback202363
  12. 27Scalable agent alignment via reward modeling: a research direction201860
  13. 28Nash Learning from Human Feedback202358
  14. 29ODIN: Disentangled Reward Mitigates Hacking in RLHF202452
  15. 30SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning202250
  16. 31A Minimaximalist Approach to Reinforcement Learning from Human Feedback202446
  17. 32Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs202446
  18. 33HelpSteer2: Open-source dataset for training top-performing reward models202446
  19. 34Is RLHF More Difficult than Standard RL?202346
  20. 35Self-Rewarding Language Models202445