mlsanityData to 7 Oct 2026

Emotional Cost Functions for AI Safety: Teaching Agents to Feel the Weight of Irreversible Consequences

arXiv 2603.14531 ↗First posted 15 Mar 2026cs.AIcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Humans learn from catastrophic mistakes not through numerical penalties, but through qualitative suffering that reshapes who they are. Current AI safety approaches replicate none of this. Reward shaping captures magnitude, not meaning. Rule-based alignment constrains behaviour, but does not change it. We propose Emotional Cost Functions, a framework in which agents develop Qualitative Suffering States, rich narrative representations of irreversible consequences that persist forward and actively reshape character. Unlike numerical penalties, qualitative suffering states capture the meaning of what was lost, the specific void it creates, and how it changes the agent's relationship to similar future situations. Our four-component architecture - Consequence Processor, Character State, Anticipatory Scan, and Story Update is grounded in one principle. Actions cannot be undone and agents must live with what they have caused. Anticipatory dread operates through two pathways. Experiential dread arises from the agent's own lived consequences. Pre-experiential dread is acquired without direct experience, through training or inter-agent transmission. Together they mirror how human wisdom accumulates across experience and culture. Ten experiments across financial trading, crisis support, and content moderation show that qualitative suffering produces specific wisdom rather than generalised paralysis. Agents correctly engage with moderate opportunities at 90-100% while numerical baselines over-refuse at 90%. Architecture ablation confirms the mechanism is necessary. The full system generates ten personal grounding phrases per probe vs. zero for a vanilla LLM. Statistical validation (N=10) confirms reproducibility at 80-100% consistency.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 14 of its 40 references that are in the map. First posted on arXiv 15 Mar 2026.

Builds on

14 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Cognitive Architectures for Language Agents5 Sep
  2. 2023A Survey on Large Language Model based Autonomous Agents22 Aug

Other references · 12

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  10. 2020Artificial Intelligence, Values and Alignmentcited by 443
  11. 2023An Overview of Catastrophic AI Riskscited by 214
  12. 2016Model-Free Episodic Controlcited by 100

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Caring Without Feeling: Affective Dynamics as the Control Layer of Human-AI Agent Collaboration
  2. 2026When Should an AI Act? A Human-Centered Model of Scene, Context, and Behavior for Agentic AI Design
  3. 2023Of Models and Tin Men: A Behavioural Economics Study of Principal-Agent Problems in AI Alignment using Large-Language Models
  4. 2026Narrative over Numbers: The Identifiable Victim Effect and its Amplification Under Alignment and Reasoning in Large Language Models
  5. 2025Inducing State Anxiety in LLM Agents Reproduces Human-Like Biases in Consumer Decision-Making
  6. 2025The Emotional Baby Is Truly Deadly: Does your Multimodal Large Reasoning Model Have Emotional Flattery towards Humans?
  7. 2026On Safety Risks in Experience-Driven Self-Evolving Agents
  8. 2025Emergent Risk Awareness in Rational Agents under Resource Constraints