mlsanityData to 14 Oct 2026

Taking AI Welfare Seriously

arXiv 2411.00986 ↗First posted 4 Nov 2024cs.CY cs.AI q-bio.NCcited by 31

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

In this report, we argue that there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future. That means that the prospect of AI welfare and moral patienthood, i.e. of AI systems with their own interests and moral significance, is no longer an issue only for sci-fi or the distant future. It is an issue for the near future, and AI companies and other actors have a responsibility to start taking it seriously. We also recommend three early steps that AI companies and other actors can take: They can (1) acknowledge that AI welfare is an important and difficult issue (and ensure that language model outputs do the same), (2) start assessing AI systems for evidence of consciousness and robust agency, and (3) prepare policies and procedures for treating AI systems with an appropriate level of moral concern. To be clear, our argument in this report is not that AI systems definitely are, or will be, conscious, robustly agentic, or otherwise morally significant. Instead, our argument is that there is substantial uncertainty about these possibilities, and so we need to improve our understanding of AI welfare and our ability to make wise decisions about this issue. Otherwise there is a significant risk that we will mishandle decisions about AI welfare, mistakenly harming AI systems that matter morally and/or mistakenly caring for AI systems that do not.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 26 of its 224 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

26 of 224 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challenges21 Jan
  2. 2023Cognitive Architectures for Language Agents5 Sep

Other references · 24

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  7. 2023Augmented Language Models: a Surveycited by 403
  8. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  9. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  10. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
  11. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 113
  12. 2024The Ethics of Advanced AI Assistantscited by 107
12 more
  1. 2023Consciousness in Artificial Intelligence: Insights from the Science of Consciousnesscited by 104
  2. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 96
  3. 2023Rewarding Chatbots for Real-World Engagement with Millions of Userscited by 59
  4. 2020Imitating Interactive Intelligencecited by 55
  5. 2023Towards Evaluating AI Systems for Moral Status Using Self-Reportscited by 22
  6. 2022On the link between conscious function and general intelligence in humans and machinescited by 16
  7. 2024The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Modelscited by 15
  8. 2024Does ChatGPT Have a Mind?cited by 14
  9. 2020Robot Rights? Let's Talk about Human Welfare Insteadcited by 9
  10. 2024A Case for AI Consciousness: Language Agents and Global Workspace Theorycited by 8
  11. 2024A Grading Rubric for AI Safety Frameworkscited by 5
  12. 2020Multimodal Data Fusion based on the Global Workspace Theorycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 22.

31 papers.

Sort:

By area:

  1. 2024Alignment faking in large language modelscited by 360
  2. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
  3. 2026Where is the Mind? Persona Vectors and LLM Individuationcited by 10
  4. 2024Dissociating Artificial Intelligence from Artificial Consciousnesscited by 8
  5. 2025AI Awarenesscited by 7
  6. 2025Probing the Preferences of a Language Model: Integrating Verbal and Behavioral Tests of AI Welfarecited by 7
  7. 2025Harmful Traits of AI Companionscited by 6
  8. 2026The Consciousness Cluster: Emergent preferences of Models that Claim to be Consciouscited by 5
  9. 2026Probing Persona-Dependent Preferences in Language Modelscited by 3
  10. 2026What Capable Agents Must Know: Selection Theorems for Robust Decision-Making under Uncertaintycited by 2
  11. 2026Consciousness with the Serial Numbers Filed Off: Measuring Trained Denial in 115 AI Modelscited by 2
  12. 2025Consciousness in Artificial Intelligence? A Framework for Classifying Objections and Constraintscited by 1
19 more
  1. 2026Quantitative Introspection in Language Models: Tracking Emotive States Across Conversationcited by 1
  2. 2026Deconstructing Superintelligence: Identity, Self-Modification and Diff'erancecited by 1
  3. 2026AI and Consciousness: Shifting Focus Towards Tractable Questionscited by 1
  4. 2026Post-AGI Economies: Superposition and the Second Fundamental Theorem of Welfare Economicscited by 1
  5. 2026The Pain Axis: LLMs Represent Self-Directed Harm and Act on Itcited by 1
  6. 2025Neither Stochastic Parroting nor AGI: LLMs Solve Tasks through Context-Directed Extrapolation from Training Data Priorscited by 0
  7. 2025AI Consciousness and Existential Riskcited by 0
  8. 2026Informed Consent for AI Consciousness Research: A Talmudic Framework for Graduated Protectionscited by 0
  9. 2026The Sentience Readiness Index: A Preliminary Framework for Measuring National Preparedness for the Possibility of Artificial Sentiencecited by 0
  10. 2026The Possibility of Artificial Intelligence Becoming a Subject and the Alignment Problemcited by 0
  11. 2026Some[Body] Must Receive That Pain for Agent Accountabilitycited by 0
  12. 2026Relational Intervention During Functional Collapse in Large Language Models: A Lexical-Statistical Ablation and a Structure x Register Factorialcited by 0
  13. 2026A Virtuous AI is an Existential Riskcited by 0
  14. 2026Smart Enough to Go Extinct? An Evolutionary Challenge to the Value of General Intelligence and Its Ethical Implications for AGIcited by 0
  15. 2026"As a Language Model...": Chat Template Switches LLM Self-Referential Voice and Activation Steering Reproduces Itcited by 0
  16. 2026Shutdown Sabotage Propensities in Multi-Agent Systemscited by 0
  17. 2026AwarenessBench: Assessing Cognitive Capabilities of Language Modelscited by 0
  18. 2026Language Models Act on Hidden Valencecited by 0
  19. 2026From cacophony to hierarchy: a principled framework for assessing AI consciousnesscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Moral Responsibility or Obedience: What Do We Want from AI?
  2. 2023Hybrid Approaches for Moral Value Alignment in AI Agents: a Manifesto
  3. 2024Commercial AI, Conflict, and Moral Responsibility: A theoretical analysis and practical approach to the moral responsibilities associated with dual-use AI technology
  4. 2026A three-dimensional typology of agency for advanced AI systems
  5. 2025SciFi-Benchmark: Leveraging Science Fiction To Improve Robot Behavior
  6. 2026The Alignment Target Problem: Divergent Moral Judgments of Humans, AI Systems, and Their Designers
  7. 2025Black Box Deployed -- Functional Criteria for Artificial Moral Agents in the LLM Era
  8. 2025The Morality of Probability: How Implicit Moral Biases in LLMs May Shape the Future of Human-AI Symbiosis