mlsanityData to 14 Oct 2026

Reward-rational (implicit) choice: A unifying formalism for reward learning

arXiv 2002.04833 ↗First posted 12 Feb 2020cs.LG cs.AI cs.HCcited by 120

Not on the map. Why?

Cited by 44 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

It is often difficult to hand-specify what the correct reward function is for a task, so researchers have instead aimed to learn reward functions from human behavior or feedback. The types of behavior interpreted as evidence of the reward function have expanded greatly in recent years. We've gone from demonstrations, to comparisons, to reading into the information leaked when the human is pushing the robot away or turning it off. And surely, there is more to come. How will a robot make sense of all these diverse types of behavior? Our key insight is that different types of behavior can be interpreted in a single unifying formalism - as a reward-rational choice that the human is making, often implicitly. The formalism offers both a unifying lens with which to view past work, as well as a recipe for interpreting new sources of information that are yet to be uncovered. We provide two examples to showcase this: interpreting a new feedback type, and reading into how the choice of feedback itself leaks information about the reward.

Key facts. Cited by 120 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 3; 2021: 13; 2022: 23; 2023: 27; 2024: 27; 2025: 13; 2026: 14.

120 papers.

Sort:

By area:

  1. 2020Learning to summarize from human feedbackcited by 2,588
  2. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  4. 2022Self-critiquing models for assisting human evaluatorscited by 380
  5. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 133
  6. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 87
  7. 2023Data Quality in Imitation Learningcited by 75
  8. 2024Building Machines that Learn and Think with Peoplecited by 73
  9. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 71
  10. 2021On the Expressivity of Markov Rewardcited by 71
  11. 2021Imitation Learning: Progress, Taxonomies and Challengescited by 58
  12. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 56
38 more
  1. 2022Inferring Rewards from Language in Contextcited by 50
  2. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  3. 2024Beyond Preferences in AI Alignmentcited by 49
  4. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  5. 2020Learning Rewards from Linguistic Feedbackcited by 41
  6. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Modelscited by 41
  7. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  8. 2023Decision-Oriented Dialogue for Human-AI Collaborationcited by 29
  9. 2024Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planningcited by 28
  10. 2022Misspecification in Inverse Reinforcement Learningcited by 27
  11. 2022The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Typescited by 26
  12. 2021Information Directed Reward Learning for Reinforcement Learningcited by 24
  13. 2021The MineRL BASALT Competition on Learning from Human Feedbackcited by 24
  14. 2021Learning Reward Functions from Scale Feedbackcited by 24
  15. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  16. 2021Cognitive science as a source of forward and inverse models of human decisions for robotics and controlcited by 22
  17. 2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
  18. 2022Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learningcited by 22
  19. 2023On The Fragility of Learned Reward Functionscited by 21
  20. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 21
  21. 2022The Expertise Problem: Learning from Specialized Feedbackcited by 20
  22. 2022Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interactioncited by 19
  23. 2022A Ranking Game for Imitation Learningcited by 18
  24. 2021Offline Preference-Based Apprenticeship Learningcited by 17
  25. 2023Inferring the Goals of Communicating Agents from Actions and Instructionscited by 17
  26. 2024Optimal Design for Human Preference Elicitationcited by 17
  27. 2022An Empirical Investigation of Representation Learning for Imitationcited by 15
  28. 2023Perspectives on the Social Impacts of Reinforcement Learning with Human Feedbackcited by 15
  29. 2023RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedbackcited by 15
  30. 2023SIRL: Similarity-based Implicit Representation Learningcited by 14
  31. 2024Large Language Models for Human-like Autonomous Driving: A Surveycited by 14
  32. 2021Choice Set Misspecification in Reward Inferencecited by 13
  33. 2022A Primer on Maximum Causal Entropy Inverse Reinforcement Learningcited by 13
  34. 2022Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learningcited by 13
  35. 2023Active Reward Learning from Online Preferencescited by 13
  36. 2023Active Reward Learning from Multiple Teacherscited by 13
  37. 2022Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approachescited by 12
  38. 2024Preference-Conditioned Language-Guided Abstractioncited by 12

50 most cited of 120 All 120 (JSON)