mlsanityData to 7 Oct 2026

Curiosity-Driven Reinforcement Learning from Human Feedback

arXiv 2501.11463 ↗First posted 20 Jan 2025cs.CLcited by 14

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but often at the cost of reduced output diversity. This trade-off between diversity and alignment quality remains a significant challenge. Drawing inspiration from curiosity-driven exploration in reinforcement learning, we introduce curiosity-driven RLHF (CD-RLHF), a framework that incorporates intrinsic rewards for novel states, alongside traditional sparse extrinsic rewards, to optimize both output diversity and alignment quality. We demonstrate the effectiveness of CD-RLHF through extensive experiments on a range of tasks, including text summarization and instruction following. Our approach achieves significant gains in diversity on multiple diversity-oriented metrics while maintaining alignment with human preferences comparable to standard RLHF. We make our code publicly available at https://github.com/ernie-research/CD-RLHF.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 50 references that are in the map. First posted on arXiv 20 Jan 2025.

Builds on

39 of 50 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions3 Oct
  2. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback23 Jul
  3. 2024Dense Reward for Free in Reinforcement Learning from Human Feedback1 Feb
  4. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversity10 Oct
  5. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales2 Aug
  6. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  8. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  10. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  11. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  8. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  10. 2022Red Teaming Language Models with Language Modelscited by 983
  11. 2018Exploration by Random Network Distillationcited by 966
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
16 more
  1. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  2. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  3. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  4. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  5. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  6. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
  7. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  8. 2017Count-Based Exploration with Neural Density Modelscited by 255
  9. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  10. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  11. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 127
  12. 2018Learning to Play with Intrinsically-Motivated Self-Aware Agentscited by 34
  13. 2022ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languagescited by 29
  14. 2022Rethinking and Refining the Distinct Metriccited by 27
  15. 2024I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigmcited by 27
  16. 2022Robust Preference Learning for Storytelling via Contrastive Reinforcement Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 9.

14 papers.

Sort:

By area:

  1. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 19
  2. 2025Meta-Thinking in LLMs via Multi-Agent Reinforcement Learning: A Surveycited by 10
  3. 2025Avoidance Decoding for Diverse Multi-Branch Story Generationcited by 6
  4. 2026PathWise: Planning through World Model for Automated Heuristic Design via Self-Evolving LLMscited by 5
  5. 2025Intrinsic Goals for Autonomous Agents: Model-Based Exploration in Virtual Zebrafish Predicts Ethological Behavior and Whole-Brain Dynamicscited by 3
  6. 2026Language of Thought Shapes Output Diversity in Large Language Modelscited by 3
  7. 2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimismcited by 2
  8. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  9. 2025Thinking Outside the (Gray) Box: A Context-Based Score for Assessing Value and Originality in Neural Text Generationcited by 1
  10. 2026Quality-constrained Entropy Maximization Policy Optimization for LLM Diversitycited by 1
  11. 2025Yahtzee: Reinforcement Learning Techniques for Stochastic Combinatorial Gamescited by 0
  12. 2026EthicMind: A Risk-Aware Framework for Ethical-Emotional Alignment in Multi-Turn Dialoguecited by 0
2 more
  1. 2026Know Thy Reasoner: Not All Language Models Explore Alikecited by 0
  2. 2026Position: The Alignment Community is Unintentionally Building a Censor's Toolkitcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2024MaxMin-RLHF: Alignment with Diverse Human Preferences
  3. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  4. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  6. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  7. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  8. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Exploration