mlsanityData to 7 Oct 2026

Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework

arXiv 2411.11761 ↗First posted 18 Nov 2024cs.LG cs.HCcited by 5

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Reinforcement Learning from Human feedback (RLHF) has become a powerful tool to fine-tune or train agentic machine learning models. Similar to how humans interact in social contexts, we can use many types of feedback to communicate our preferences, intentions, and knowledge to an RL agent. However, applications of human feedback in RL are often limited in scope and disregard human factors. In this work, we bridge the gap between machine learning and human-computer interaction efforts by developing a shared understanding of human feedback in interactive learning scenarios. We first introduce a taxonomy of feedback types for reward-based learning from human feedback based on nine key dimensions. Our taxonomy allows for unifying human-centered, interface-centered, and model-centered aspects. In addition, we identify seven quality metrics of human feedback influencing both the human ability to express feedback and the agent's ability to learn from the feedback. Based on the feedback taxonomy and quality criteria, we derive requirements and design choices for systems learning from human feedback. We relate these requirements and design choices to existing work in interactive machine learning. In the process, we identify gaps in existing work and future research opportunities. We call for interdisciplinary collaboration to harness the full potential of reinforcement learning with data-driven co-adaptive modeling and varied interaction mechanics.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 55 of its 224 references that are in the map. First posted on arXiv 18 Nov 2024.

Builds on

55 of 224 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May

Other references · 54

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  8. 2018Behavioral Cloning from Observationcited by 482
  9. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  10. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  11. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  12. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
38 more
  1. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  2. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  3. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  4. 2018Unity: A General Platform for Intelligent Agentscited by 232
  5. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  6. 2018Programmatically Interpretable Reinforcement Learningcited by 192
  7. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  8. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  9. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  10. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  11. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  12. 2023Verbosity Bias in Preference Labeling by Large Language Modelscited by 89
  13. 2022Correcting Robot Plans with Natural Language Feedbackcited by 68
  14. 2022Inferring Rewards from Language in Contextcited by 50
  15. 2019Leveraging Human Guidance for Deep Reinforcement Learning Taskscited by 42
  16. 2021A Survey on Interactive Reinforcement Learning: Design Principles and Open Challengescited by 41
  17. 2018DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedbackcited by 39
  18. 2024Reinforcement Learning from Human Feedback with Active Queriescited by 39
  19. 2023Interactive Robot Learning from Verbal Correctioncited by 28
  20. 2022The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Typescited by 26
  21. 2021Learning Reward Functions from Scale Feedbackcited by 24
  22. 2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
  23. 2022Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learningcited by 22
  24. 2022Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interactioncited by 18
  25. 2018An Atari Model Zoo for Analyzing, Visualizing, and Comparing Deep Reinforcement Learning Agentscited by 15
  26. 2023RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedbackcited by 15
  27. 2021Choice Set Misspecification in Reward Inferencecited by 13
  28. 2023Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-Loop Feedbackcited by 13
  29. 2018Active Inverse Reward Designcited by 12
  30. 2017Guiding Reinforcement Learning Exploration Using Natural Languagecited by 10
  31. 2018APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learningcited by 9
  32. 2017Explore, Exploit or Listen: Combining Human Feedback and Policy Model to Speed up Deep Reinforcement Learning in 3D Worldscited by 8
  33. 2022Time-Efficient Reward Learning via Visually Assisted Cluster Rankingcited by 8
  34. 2016Face valuing: Training user interfaces with facial expressions and reinforcement learningcited by 6
  35. 2022Linguistic communication as (inverse) reward designcited by 6
  36. 2022Deep Meta-learning in Recommendation Systems: A Surveycited by 5
  37. 2024ExTraCT -- Explainable Trajectory Corrections from language inputs using Textual description of featurescited by 5
  38. 2021Learning What To Do by Simulating the Pastcited by 4

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 5.

5 papers.

  1. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Surveycited by 28
  2. 2025Optimal Interactive Learning on the Job via Facility Location Planningcited by 2
  3. 2025LLM-Guided Reinforcement Learning: Addressing Training Bottlenecks through Policy Modulationcited by 2
  4. 2025Multimedia and Visual Analytics in the Agentic Eracited by 1
  5. 2025A Flexible Multi-Agent LLM-Human Framework for Fast Human Validated Tool Buildingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2025Reinforcement Learning from Multi-level and Episodic Human Feedback
  3. 2025Reward Learning from Multiple Feedback Types
  4. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  5. 2023The History and Risks of Reinforcement Learning and Human Feedback
  6. 2025Reinforcement Learning from Human Feedback
  7. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  8. 2025Multi-Task Reward Learning from Human Ratings