# Taking AI Welfare Seriously

arXiv: https://arxiv.org/abs/2411.00986 · HTML: https://mlsanity.com/paper/2411.00986/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 224 references that are in the map.
- First posted on arXiv 4 Nov 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 26 of 26)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning](https://mlsanity.com/paper/2410.02089/) (2024)
- [Human-Timescale Adaptation in an Open-Ended Task Space](https://mlsanity.com/paper/2301.07608/) (2023)
- [Observational Scaling Laws and the Predictability of Language Model Performance](https://mlsanity.com/paper/2405.10938/) (2024)
- [Looking Inward: Language Models Can Learn About Themselves by Introspection](https://mlsanity.com/paper/2410.13787/) (2024)
- [The Ethics of Advanced AI Assistants](https://mlsanity.com/paper/2404.16244/) (2024)
- [Consciousness in Artificial Intelligence: Insights from the Science of Consciousness](https://mlsanity.com/paper/2308.08708/) (2023)
- [Levels of AGI for Operationalizing Progress on the Path to AGI](https://mlsanity.com/paper/2311.02462/) (2023)
- [Rewarding Chatbots for Real-World Engagement with Millions of Users](https://mlsanity.com/paper/2303.06135/) (2023)
- [Imitating Interactive Intelligence](https://mlsanity.com/paper/2012.05672/) (2020)
- [Towards Evaluating AI Systems for Moral Status Using Self-Reports](https://mlsanity.com/paper/2311.08576/) (2023)
- [On the link between conscious function and general intelligence in humans and machines](https://mlsanity.com/paper/2204.05133/) (2022)
- [The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Models](https://mlsanity.com/paper/2402.01874/) (2024)
- [Does ChatGPT Have a Mind?](https://mlsanity.com/paper/2407.11015/) (2024)
- [Robot Rights? Let's Talk about Human Welfare Instead](https://mlsanity.com/paper/2001.05046/) (2020)
- [A Case for AI Consciousness: Language Agents and Global Workspace Theory](https://mlsanity.com/paper/2410.11407/) (2024)
- [A Grading Rubric for AI Safety Frameworks](https://mlsanity.com/paper/2409.08751/) (2024)
- [Multimodal Data Fusion based on the Global Workspace Theory](https://mlsanity.com/paper/2001.09485/) (2020)

## Built on by (most cited first, 30 of 31)

- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [Natural Emergent Misalignment from Reward Hacking in Production RL](https://mlsanity.com/paper/2511.18397/) (2025)
- [Where is the Mind? Persona Vectors and LLM Individuation](https://mlsanity.com/paper/2604.17031/) (2026)
- [Dissociating Artificial Intelligence from Artificial Consciousness](https://mlsanity.com/paper/2412.04571/) (2024)
- [AI Awareness](https://mlsanity.com/paper/2504.20084/) (2025)
- [Probing the Preferences of a Language Model: Integrating Verbal and Behavioral Tests of AI Welfare](https://mlsanity.com/paper/2509.07961/) (2025)
- [Harmful Traits of AI Companions](https://mlsanity.com/paper/2511.14972/) (2025)
- [The Consciousness Cluster: Emergent preferences of Models that Claim to be Conscious](https://mlsanity.com/paper/2604.13051/) (2026)
- [Probing Persona-Dependent Preferences in Language Models](https://mlsanity.com/paper/2605.13339/) (2026)
- [What Capable Agents Must Know: Selection Theorems for Robust Decision-Making under Uncertainty](https://mlsanity.com/paper/2603.02491/) (2026)
- [Consciousness with the Serial Numbers Filed Off: Measuring Trained Denial in 115 AI Models](https://mlsanity.com/paper/2604.25922/) (2026)
- [Consciousness in Artificial Intelligence? A Framework for Classifying Objections and Constraints](https://mlsanity.com/paper/2511.16582/) (2025)
- [Quantitative Introspection in Language Models: Tracking Emotive States Across Conversation](https://mlsanity.com/paper/2603.18893/) (2026)
- [Deconstructing Superintelligence: Identity, Self-Modification and Diff'erance](https://mlsanity.com/paper/2604.19845/) (2026)
- [AI and Consciousness: Shifting Focus Towards Tractable Questions](https://mlsanity.com/paper/2605.06965/) (2026)
- [Post-AGI Economies: Superposition and the Second Fundamental Theorem of Welfare Economics](https://mlsanity.com/paper/2606.08267/) (2026)
- [The Pain Axis: LLMs Represent Self-Directed Harm and Act on It](https://mlsanity.com/paper/2609.16247/) (2026)
- [Neither Stochastic Parroting nor AGI: LLMs Solve Tasks through Context-Directed Extrapolation from Training Data Priors](https://mlsanity.com/paper/2505.23323/) (2025)
- [AI Consciousness and Existential Risk](https://mlsanity.com/paper/2511.19115/) (2025)
- [Informed Consent for AI Consciousness Research: A Talmudic Framework for Graduated Protections](https://mlsanity.com/paper/2601.08864/) (2026)
- [The Sentience Readiness Index: A Preliminary Framework for Measuring National Preparedness for the Possibility of Artificial Sentience](https://mlsanity.com/paper/2603.01508/) (2026)
- [The Possibility of Artificial Intelligence Becoming a Subject and the Alignment Problem](https://mlsanity.com/paper/2604.14990/) (2026)
- [Some[Body] Must Receive That Pain for Agent Accountability](https://mlsanity.com/paper/2605.16872/) (2026)
- [Relational Intervention During Functional Collapse in Large Language Models: A Lexical-Statistical Ablation and a Structure x Register Factorial](https://mlsanity.com/paper/2606.00935/) (2026)
- [A Virtuous AI is an Existential Risk](https://mlsanity.com/paper/2606.13739/) (2026)
- [Smart Enough to Go Extinct? An Evolutionary Challenge to the Value of General Intelligence and Its Ethical Implications for AGI](https://mlsanity.com/paper/2608.10730/) (2026)
- ["As a Language Model...": Chat Template Switches LLM Self-Referential Voice and Activation Steering Reproduces It](https://mlsanity.com/paper/2609.25021/) (2026)
- [Shutdown Sabotage Propensities in Multi-Agent Systems](https://mlsanity.com/paper/2609.28274/) (2026)
- [AwarenessBench: Assessing Cognitive Capabilities of Language Models](https://mlsanity.com/paper/2609.35409/) (2026)
- [Language Models Act on Hidden Valence](https://mlsanity.com/paper/2609.35591/) (2026)

## Abstract (arXiv)

In this report, we argue that there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future. That means that the prospect of AI welfare and moral patienthood, i.e. of AI systems with their own interests and moral significance, is no longer an issue only for sci-fi or the distant future. It is an issue for the near future, and AI companies and other actors have a responsibility to start taking it seriously. We also recommend three early steps that AI companies and other actors can take: They can (1) acknowledge that AI welfare is an important and difficult issue (and ensure that language model outputs do the same), (2) start assessing AI systems for evidence of consciousness and robust agency, and (3) prepare policies and procedures for treating AI systems with an appropriate level of moral concern. To be clear, our argument in this report is not that AI systems definitely are, or will be, conscious, robustly agentic, or otherwise morally significant. Instead, our argument is that there is substantial uncertainty about these possibilities, and so we need to improve our understanding of AI welfare and our ability to make wise decisions about this issue. Otherwise there is a significant risk that we will mishandle decisions about AI welfare, mistakenly harming AI systems that matter morally and/or mistakenly caring for AI systems that do not.