mlsanityData to 7 Oct 2026

Fine-Grained Human Feedback Gives Better Rewards for Language Model Training

arXiv 2306.01693 ↗First posted 2 Jun 2023cs.CLcited by 399

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Language models (LMs) often exhibit undesirable text generation behaviors, including generating false, toxic, or irrelevant outputs. Reinforcement learning from human feedback (RLHF) - where human preference judgments on LM outputs are transformed into a learning signal - has recently shown promise in addressing these issues. However, such holistic feedback conveys limited information on long text outputs; it does not indicate which aspects of the outputs influenced user preference; e.g., which parts contain what type(s) of errors. In this paper, we use fine-grained human feedback (e.g., which sentence is false, which sub-sentence is irrelevant) as an explicit training signal. We introduce Fine-Grained RLHF, a framework that enables training and learning from reward functions that are fine-grained in two respects: (1) density, providing a reward after every segment (e.g., a sentence) is generated; and (2) incorporating multiple reward models associated with different feedback types (e.g., factual incorrectness, irrelevance, and information incompleteness). We conduct experiments on detoxification and long-form question answering to illustrate how learning with such reward functions leads to improved performance, supported by both automatic and human evaluation. Additionally, we show that LM behaviors can be customized using different combinations of fine-grained reward models. We release all data, collected human feedback, and codes at https://FineGrainedRLHF.github.io.

Key facts. Cited by 399 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 44 references that are in the map. First posted on arXiv 2 Jun 2023.

Builds on

37 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  3. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  4. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  9. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  10. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  11. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  12. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
21 more
  1. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  2. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  3. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  4. 2022Teaching language models to support answers with verified quotescited by 290
  5. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  6. 2023Pretraining Language Models with Human Preferencescited by 245
  7. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  8. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  9. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 231
  10. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  11. 2023Training Language Models with Language Feedback at Scalecited by 125
  12. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  13. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
  14. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  15. 2021Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Textcited by 89
  16. 2019ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisonscited by 83
  17. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 75
  18. 2023Automatic Evaluation of Attribution by Large Language Modelscited by 70
  19. 2022Towards Teachable Reasoning Systems: Using a Dynamic Memory of User Feedback for Continual System Improvementcited by 37
  20. 2022Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedbackcited by 35
  21. 2022When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labelscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 62; 2024: 169; 2025: 103; 2026: 65.

399 papers.

Sort:

By area:

  1. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  2. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  5. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  6. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  7. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  8. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  9. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  10. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  11. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  12. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 317
38 more
  1. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  2. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  3. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  4. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  5. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 254
  6. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  7. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  8. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  9. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  10. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  11. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  12. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  13. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  14. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  15. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  16. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  17. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  18. 2024Token-level Direct Preference Optimizationcited by 135
  19. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  20. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  21. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 124
  22. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  23. 2024Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-stepcited by 123
  24. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 121
  25. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  26. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  27. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  28. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 106
  29. 2024Rule Based Rewards for Language Model Safetycited by 106
  30. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 105
  31. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  32. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  33. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  34. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoningcited by 96
  35. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revisioncited by 96
  36. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  37. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  38. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 95

50 most cited of 399 All 399 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Interpreting Learned Feedback Patterns in Large Language Models
  2. 2026Reward Modeling from Natural Language Human Feedback
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedback
  5. 2022Training Language Models with Language Feedback
  6. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  7. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  8. 2023Chain of Hindsight Aligns Language Models with Feedback