mlsanityData to 7 Oct 2026

Understanding the Effects of RLHF on LLM Generalisation and Diversity

arXiv 2310.06452 ↗First posted 10 Oct 2023cs.LG cs.AI cs.CLcited by 386

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Large language models (LLMs) fine-tuned with reinforcement learning from human feedback (RLHF) have been used in some of the most widely deployed AI models to date, such as OpenAI's ChatGPT or Anthropic's Claude. While there has been significant work developing these methods, our understanding of the benefits and downsides of each stage in RLHF is still limited. To fill this gap, we present an extensive analysis of how each stage of the process (i.e. supervised fine-tuning (SFT), reward modelling, and RLHF) affects two key properties: out-of-distribution (OOD) generalisation and output diversity. OOD generalisation is crucial given the wide range of real-world scenarios in which these models are being used, while output diversity refers to the model's ability to generate varied outputs and is important for a variety of use cases. We perform our analysis across two base models on both summarisation and instruction following tasks, the latter being highly relevant for current LLM use cases. We find that RLHF generalises better than SFT to new inputs, particularly as the distribution shift between train and test becomes larger. However, RLHF significantly reduces output diversity compared to SFT across a variety of measures, implying a tradeoff in current LLM fine-tuning methods between generalisation and diversity. Our results provide guidance on which fine-tuning method should be used depending on the application, and show that more research is needed to improve the tradeoff between generalisation and diversity.

Key facts. Cited by 386 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 66 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

53 of 66 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  4. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followers10 Feb
  5. 2023Chain of Hindsight Aligns Language Models with Feedback6 Feb
  6. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  8. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  9. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
31 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  5. 2017Hindsight Experience Replaycited by 1,096
  6. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  7. 2022Red Teaming Language Models with Language Modelscited by 983
  8. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  11. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  12. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  13. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  14. 2023Instruction Tuning with GPT-4cited by 676
  15. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  16. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  17. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  18. 2022Teaching language models to support answers with verified quotescited by 290
  19. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  20. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  21. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 127
  22. 2023Emergent autonomous scientific research capabilities of large language modelscited by 127
  23. 2023Training Language Models with Language Feedback at Scalecited by 125
  24. 2020A Distributional Approach to Controlled Text Generationcited by 117
  25. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  26. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  27. 2020One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RLcited by 79
  28. 2022Semantic Diversity in Dialogue with Natural Language Inferencecited by 28
  29. 2022Rethinking and Refining the Distinct Metriccited by 27
  30. 2021Discovering Diverse Solutions in Deep Reinforcement Learning by Maximizing State-Action-Based Mutual Informationcited by 18
  31. 2022Robust Preference Learning for Storytelling via Contrastive Reinforcement Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 78; 2025: 133; 2026: 172.

386 papers.

Sort:

By area:

  1. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  2. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  3. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
  4. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  5. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  6. 2024A Roadmap to Pluralistic Alignmentcited by 175
  7. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  8. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  9. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  10. 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 130
  11. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 123
  12. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 103
38 more
  1. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  2. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  3. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
  4. 2025Outcome-based Exploration for LLM Reasoningcited by 79
  5. 2024Evaluating Large Language Model Biases in Persona-Steered Generationcited by 78
  6. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
  7. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
  8. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  9. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  10. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  11. 2024Learn Your Reference Model for Real Good Alignmentcited by 56
  12. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  13. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  14. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  15. 2024Self-Consistency Preference Optimizationcited by 50
  16. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
  17. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  18. 2025Diverse Preference Optimizationcited by 48
  19. 2025Base Models Beat Aligned Models at Randomness and Creativitycited by 48
  20. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  21. 2024Detection and Measurement of Syntactic Templates in Generated Textcited by 44
  22. 2025The Price of Format: Diversity Collapse in LLMscited by 43
  23. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  24. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  25. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 42
  26. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
  27. 2025Modifying Large Language Model Post-Training for Diverse Creative Writingcited by 39
  28. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoningcited by 39
  29. 2024Benchmarking Language Model Creativity: A Case Study on Code Generationcited by 38
  30. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
  31. 2024Benchmarking Linguistic Diversity of Large Language Modelscited by 37
  32. 2024Unveiling the Generalization Power of Fine-Tuned Large Language Modelscited by 35
  33. 2024Multi-property Steering of Large Language Models with Dynamic Activation Compositioncited by 35
  34. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  35. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignmentcited by 33
  36. 2024Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learningcited by 32
  37. 2025Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-timecited by 32
  38. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31

50 most cited of 386 All 386 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Method
  2. 2026Generalisation of RLHF under Reward Shift and Clipped KL Regularisation
  3. 2024Reinforcement Learning Enhanced LLMs: A Survey
  4. 2024Reward-Robust RLHF in LLMs
  5. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  6. 2026Towards a Theoretical Understanding to the Generalization of RLHF
  7. 2025KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexity
  8. 2023Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond