# Towards Safe and Honest AI Agents with Neural Self-Other Overlap

arXiv: https://arxiv.org/abs/2412.16325 · HTML: https://mlsanity.com/paper/2412.16325/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 20 Dec 2024.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 24 of 24)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019)
- [Let's Think Dot by Dot: Hidden Computation in Transformer Language Models](https://mlsanity.com/paper/2404.15758/) (2024)
- [Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims](https://mlsanity.com/paper/2004.07213/) (2020)
- [Collective Constitutional AI: Aligning a Language Model with Public Input](https://mlsanity.com/paper/2406.07814/) (2024)
- [Truthful AI: Developing and governing AI that does not lie](https://mlsanity.com/paper/2110.06674/) (2021)
- [Deception Abilities Emerged in Large Language Models](https://mlsanity.com/paper/2307.16513/) (2023)
- [Modeling Others using Oneself in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.09640/) (2018)
- [The History and Risks of Reinforcement Learning and Human Feedback](https://mlsanity.com/paper/2310.13595/) (2023)
- [Honesty Is the Best Policy: Defining and Mitigating AI Deception](https://mlsanity.com/paper/2312.01350/) (2023)
- [Language Models Represent Beliefs of Self and Others](https://mlsanity.com/paper/2402.18496/) (2024)
- [Path-Specific Objectives for Safer Agent Incentives](https://mlsanity.com/paper/2204.10018/) (2022)
- [Predicting vs. Acting: A Trade-off Between World Modeling & Agent Modeling](https://mlsanity.com/paper/2407.02446/) (2024)
- [Towards Empathic Deep Q-Learning](https://mlsanity.com/paper/1906.10918/) (2019)
- [Unexpected Benefits of Self-Modeling in Neural Systems](https://mlsanity.com/paper/2407.10188/) (2024)
- [Deception Analysis with Artificial Intelligence: An Interdisciplinary Perspective](https://mlsanity.com/paper/2406.05724/) (2024)
- [Unmasking the Shadows of AI: Investigating Deceptive Capabilities in Large Language Models](https://mlsanity.com/paper/2403.09676/) (2024)

## Built on by (most cited first, 6 of 6)

- [Evaluating & Reducing Deceptive Dialogue From Language Models with Multi-turn RL](https://mlsanity.com/paper/2510.14318/) (2025)
- [Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Value](https://mlsanity.com/paper/2512.03399/) (2025)
- [Consistency Training Can Entrench Misalignment](https://mlsanity.com/paper/2606.03810/) (2026)
- [Contemplative Artificial Intelligence](https://mlsanity.com/paper/2504.15125/) (2025)
- [Will Power Return to the Clouds? From Divine Authority to GenAI Authority](https://mlsanity.com/paper/2512.03076/) (2025)
- [Artificial Empathy: Towards a Framework for Unsupervised Agency Detection and Policy Reconstruction](https://mlsanity.com/paper/2608.23030/) (2026)

## Abstract (arXiv)

As AI systems increasingly make critical decisions, deceptive AI poses a significant challenge to trust and safety. We present Self-Other Overlap (SOO) fine-tuning, a promising approach in AI Safety that could substantially improve our ability to build honest artificial intelligence. Inspired by cognitive neuroscience research on empathy, SOO aims to align how AI models represent themselves and others. Our experiments on LLMs with 7B, 27B, and 78B parameters demonstrate SOO's efficacy: deceptive responses of Mistral-7B-Instruct-v0.2 dropped from 73.6% to 17.2% with no observed reduction in general task performance, while in Gemma-2-27b-it and CalmeRys-78B-Orpo-v0.1 deceptive responses were reduced from 100% to 9.3% and 2.7%, respectively, with a small impact on capabilities. In reinforcement learning scenarios, SOO-trained agents showed significantly reduced deceptive behavior. SOO's focus on contrastive self and other-referencing observations offers strong potential for generalization across AI architectures. While current applications focus on language models and simple RL environments, SOO could pave the way for more trustworthy AI in broader domains. Ethical implications and long-term effects warrant further investigation, but SOO represents a significant step forward in AI safety research.