# Moral Responsibility or Obedience: What Do We Want from AI?

arXiv: https://arxiv.org/abs/2507.02788 · HTML: https://mlsanity.com/paper/2507.02788/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 3 of its 34 references that are in the map.
- First posted on arXiv 3 Jul 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 3 of 3)

- [Generative to Agentic AI: Survey, Conceptualization, and Challenges](https://mlsanity.com/paper/2504.18875/) (2025)
- [Development of a Large Language Model-based Multi-Agent Clinical Decision Support System for Korean Triage and Acuity Scale (KTAS)-Based Triage and Treatment Planning in Emergency Departments](https://mlsanity.com/paper/2408.07531/) (2024)
- [TRIAGE: Ethical Benchmarking of AI Models Through Mass Casualty Simulations](https://mlsanity.com/paper/2410.18991/) (2024)

## Built on by (most cited first, 2 of 2)

- [Why Sampling Is Not Choosing: Intentionality, Agency, and Moral Responsibility in Large Language Models](https://mlsanity.com/paper/2606.13441/) (2026)
- [Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers](https://mlsanity.com/paper/2608.14522/) (2026)

## Abstract (arXiv)

As artificial intelligence systems become increasingly agentic, capable of general reasoning, planning, and value prioritization, current safety practices that treat obedience as a proxy for ethical behavior are becoming inadequate. This paper examines recent safety testing incidents involving large language models (LLMs) that appeared to disobey shutdown commands or engage in ethically ambiguous or illicit behavior. I argue that such behavior should not be interpreted as rogue or misaligned, but as early evidence of emerging ethical reasoning in agentic AI. Drawing on philosophical debates about instrumental rationality, moral responsibility, and goal revision, I contrast dominant risk paradigms with more recent frameworks that acknowledge the possibility of artificial moral agency. I call for a shift in AI safety evaluation: away from rigid obedience and toward frameworks that can assess ethical judgment in systems capable of navigating moral dilemmas. Without such a shift, we risk mischaracterizing AI behavior and undermining both public trust and effective governance.