mlsanityData to 14 Oct 2026

Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learning

arXiv 2206.13316 ↗First posted 27 Jun 2022cs.LG cs.HC stat.MLcited by 22

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) commonly assumes access to well-specified reward functions, which many practical applications do not provide. Instead, recently, more work has explored learning what to do from interacting with humans. So far, most of these approaches model humans as being (nosily) rational and, in particular, giving unbiased feedback. We argue that these models are too simplistic and that RL researchers need to develop more realistic human models to design and evaluate their algorithms. In particular, we argue that human models have to be personal, contextual, and dynamic. This paper calls for research from different disciplines to address key questions about how humans provide feedback to AIs and how we can build more robust human-in-the-loop RL systems.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 14 of its 29 references that are in the map. First posted on arXiv 27 Jun 2022.

Builds on

14 of 29 references.

References · 14

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 349
  3. 2016Cooperative Inverse Reinforcement Learningcited by 300
  4. 2018Learning to Walk via Deep Reinforcement Learningcited by 158
  5. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 120
  6. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  7. 2021Reward (Mis)design for Autonomous Drivingcited by 104
  8. 2016The Off-Switch Gamecited by 97
  9. 2020Human-centered Explainable AI: Towards a Reflective Sociotechnical Approachcited by 90
  10. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencecited by 49
  11. 2020The EMPATHIC Framework for Task Learning from Implicit Human Feedbackcited by 38
  12. 2019Preferences Implicit in the State of the Worldcited by 34
2 more
  1. 2016Learning Preferences for Manipulation Tasks from Online Coactive Feedbackcited by 31
  2. 2021Choice Set Misspecification in Reward Inferencecited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 9; 2025: 6; 2026: 2.

22 papers.

Sort:

By area:

  1. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  3. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 114
  4. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Modelscited by 41
  5. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  6. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  7. 2024Towards Trustworthy AI: A Review of Ethical and Robust Large Language Modelscited by 22
  8. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 21
  9. 2024Mutual Theory of Mind in Human-AI Collaboration: An Empirical Study with LLM-driven AI Agents in a Real-time Shared Workspace Taskcited by 17
  10. 2023RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedbackcited by 15
  11. 2025RLHS: Mitigating Misalignment in RLHF with Hindsight Simulationcited by 15
  12. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
10 more
  1. 2025Doubly Robust Alignment for Large Language Modelscited by 12
  2. 2023Active teacher selection for reward learningcited by 7
  3. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Frameworkcited by 5
  4. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Reviewcited by 3
  5. 2025Zero-Shot LLMs in Human-in-the-Loop RL: Replacing Human Feedback for Reward Shapingcited by 3
  6. 2026GRAIL: Goal Recognition Alignment through Imitation Learningcited by 3
  7. 2026Adaptive Pluralistic Alignment: A pipeline for dynamic artificial democracycited by 2
  8. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 1
  9. 2025Towards Reinforcement Learning from Neural Feedback: Mapping fNIRS Signals to Agent Performancecited by 1
  10. 2024On the Effect of Robot Errors on Human Teaching Dynamicscited by 0