mlsanityData to 14 Oct 2026

(Ir)rationality in AI: State of the Art, Research Challenges and Open Questions

arXiv 2311.17165 ↗First posted 28 Nov 2023cs.AI cs.CY cs.HCcited by 5

Not on the map. Why?

Abstract (arXiv)

The concept of rationality is central to the field of artificial intelligence (AI). Whether we are seeking to simulate human reasoning, or trying to achieve bounded optimality, our goal is generally to make artificial agents as rational as possible. Despite the centrality of the concept within AI, there is no unified definition of what constitutes a rational agent. This article provides a survey of rationality and irrationality in AI, and sets out the open questions in this area. We consider how the understanding of rationality in other fields has influenced its conception within AI, in particular work in economics, philosophy and psychology. Focusing on the behaviour of artificial agents, we examine irrational behaviours that can prove to be optimal in certain scenarios. Some methods have been developed to deal with irrational agents, both in terms of identification and interaction, however work in this area remains limited. Methods that have up to now been developed for other purposes, namely adversarial scenarios, may be adapted to suit interactions with artificial agents. We further discuss the interplay between human and artificial agents, and the role that rationality plays within this interaction; many questions remain in this area, relating to potentially irrational behaviour of both humans and artificial agents.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 59 of its 281 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

59 of 281 references.

References · 59

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2016Concrete Problems in AI Safetycited by 1,642
  5. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  6. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  7. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  8. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  9. 2022Using cognitive psychology to understand GPT-3cited by 398
  10. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  11. 2021Unsolved Problems in ML Safetycited by 292
  12. 2017Learning with Opponent-Learning Awarenesscited by 222
38 more
  1. 2023GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Modelscited by 176
  2. 2022Talking About Large Language Modelscited by 175
  3. 2020Open Problems in Cooperative AIcited by 168
  4. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  5. 2022Thinking Fast and Slow in Large Language Modelscited by 137
  6. 2017Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problemscited by 131
  7. 2017The Consciousness Priorcited by 107
  8. 2016Opponent Modeling in Deep Reinforcement Learningcited by 97
  9. 2017Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Gamescited by 96
  10. 2018'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisionscited by 91
  11. 2021A Survey on Gender Bias in Natural Language Processingcited by 89
  12. 2019Predictive Inequity in Object Detectioncited by 77
  13. 2020Reinforcement Learning in Economics and Financecited by 58
  14. 2018Stable Opponent Shaping in Differentiable Gamescited by 49
  15. 2015Learning the Preferences of Ignorant, Inconsistent Agentscited by 45
  16. 2017Occam's razor is insufficient to infer the preferences of irrational agentscited by 45
  17. 2022Model-Free Opponent Shapingcited by 44
  18. 2020A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learningcited by 41
  19. 2022Can language models handle recursively nested grammatical structures? A case study on comparing models and humanscited by 41
  20. 2020When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humanscited by 34
  21. 2022The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Modelscited by 31
  22. 2021Multi-Agent Reinforcement Learning with Temporal Logic Specificationscited by 27
  23. 2022Misspecification in Inverse Reinforcement Learningcited by 27
  24. 2022The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Typescited by 26
  25. 2021Human irrationality: both bad and good for reward inferencecited by 25
  26. 2019Modelling Bounded Rationality in Multi-Agent Interactions by Generalized Recursive Reasoningcited by 24
  27. 2019Reasoning about Hypothetical Agent Behaviours and their Parameterscited by 24
  28. 2017A Game-Theoretic Taxonomy and Survey of Defensive Deception for Cybersecurity and Privacycited by 22
  29. 2013Reasoning About Beliefs and Actions Under Computational Resource Constraintscited by 18
  30. 2020Improving Generalization of Reinforcement Learning with Minimax Distributional Soft Actor-Criticcited by 12
  31. 2017Heuristic Online Goal Recognition in Continuous Domainscited by 11
  32. 2019Manipulating a Learning Defender and Ways to Counteractcited by 8
  33. 2020Indecision Modelingcited by 8
  34. 2021Marked Attribute Bias in Natural Language Inferencecited by 6
  35. 2021Beyond Algorithmic Bias: A Socio-Computational Interrogation of the Google Search by Image Algorithmcited by 5
  36. 2022BIASeD: Bringing Irrationality into Automated System Designcited by 5
  37. 2017Autonomous Quadrotor Landing using Deep Reinforcement Learningcited by 4
  38. 2022The dangers in algorithms learning humans' values and irrationalitiescited by 3

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 4.

5 papers.

  1. 2024(Ir)rationality and Cognitive Biases in Large Language Modelscited by 23
  2. 2026Motivation in Large Language Modelscited by 1
  3. 2026Rationality Measurement and Theory for Reinforcement Learning Agentscited by 0
  4. 2026Mitigating Cognitive Bias in RLHF by Altering Rationalitycited by 0
  5. 2026ContextAdapt: Evaluating Contextual Adaptation and Value Alignment in LLMscited by 0