mlsanityData to 14 Oct 2026

Cooperative Inverse Reinforcement Learning

arXiv 1606.03137 ↗First posted 9 Jun 2016cs.AIcited by 300

Not on the map. Why?

Cited by 43 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

For an autonomous system to be helpful to humans and to pose no unwarranted risks, it needs to align its values with those of the humans in its environment in such a way that its actions contribute to the maximization of value for the humans. We propose a formal definition of the value alignment problem as cooperative inverse reinforcement learning (CIRL). A CIRL problem is a cooperative, partial-information game with two agents, human and robot; both are rewarded according to the human's reward function, but the robot does not initially know what this is. In contrast to classical IRL, where the human is assumed to act optimally in isolation, optimal CIRL solutions produce behaviors such as active teaching, active learning, and communicative actions that are more effective in achieving value alignment. We show that computing optimal joint policies in CIRL games can be reduced to solving a POMDP, prove that optimality in isolation is suboptimal in CIRL, and derive an approximate CIRL algorithm.

Key facts. Cited by 300 papers in the map (data through 14 Oct 2026). First posted on arXiv 9 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2017: 1; 2018: 9; 2019: 5; 2020: 7; 2021: 45; 2022: 35; 2023: 49; 2024: 47; 2025: 49; 2026: 53.

300 papers.

Sort:

By area:

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  4. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  5. 2018An Algorithmic Perspective on Imitation Learningcited by 459
  6. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  7. 2021Unsolved Problems in ML Safetycited by 292
  8. 2022Defining and Characterizing Reward Hackingcited by 203
  9. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
  10. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  11. 2023Understanding Social Reasoning in Language Models with Language Modelscited by 172
  12. 2021Alignment of Language Agentscited by 168
38 more
  1. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 148
  2. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  3. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperationcited by 112
  4. 2018Multi-Agent Generative Adversarial Imitation Learningcited by 100
  5. 2022Evaluating Human-Language Model Interactioncited by 92
  6. 2023Eliciting Human Preferences with Language Modelscited by 92
  7. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 85
  8. 2021Consequences of Misaligned AIcited by 80
  9. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 80
  10. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  11. 2024Building Machines that Learn and Think with Peoplecited by 73
  12. 2021On the Expressivity of Markov Rewardcited by 71
  13. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matchingcited by 67
  14. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 67
  15. 2022Forecasting Future World Events with Neural Networkscited by 63
  16. 2023Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planningcited by 60
  17. 2021What Would Jiminy Cricket Do? Towards Agents That Behave Morallycited by 57
  18. 2025Why human-AI relationships need socioaffective alignmentcited by 47
  19. 2021AGENT: A Benchmark for Core Psychological Reasoningcited by 42
  20. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
  21. 2020Learning Rewards from Linguistic Feedbackcited by 41
  22. 2025Infrastructure for AI Agentscited by 41
  23. 2022Melting Pot 2.0cited by 40
  24. 2021The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Failcited by 38
  25. 2023Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biasedcited by 37
  26. 2022Aligning to Social Norms and Values in Interactive Narrativescited by 32
  27. 2023Decision-Oriented Dialogue for Human-AI Collaborationcited by 29
  28. 2025Mitigating Deceptive Alignment via Self-Monitoringcited by 29
  29. 2018Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applicationscited by 28
  30. 2021Medical Dead-ends and Learning to Identify High-risk States and Treatmentscited by 28
  31. 2022Discovering Agentscited by 28
  32. 2024Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planningcited by 28
  33. 2022Misspecification in Inverse Reinforcement Learningcited by 27
  34. 2021Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environmentscited by 26
  35. 2021Hard Choices in Artificial Intelligencecited by 26
  36. 2023TASRA: a Taxonomy and Analysis of Societal-Scale Risks from AIcited by 26
  37. 2021Human irrationality: both bad and good for reward inferencecited by 25
  38. 2021MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learningcited by 25

50 most cited of 300 All 300 (JSON)