# Honesty Is the Best Policy: Defining and Mitigating AI Deception

arXiv: https://arxiv.org/abs/2312.01350 · HTML: https://mlsanity.com/paper/2312.01350/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 128 references that are in the map.
- First posted on arXiv 3 Dec 2023.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [Certified Defenses for Data Poisoning Attacks](https://mlsanity.com/paper/1706.03691/) (2017)
- [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019)
- [An Overview of Catastrophic AI Risks](https://mlsanity.com/paper/2306.12001/) (2023)
- [Deal or No Deal? End-to-End Learning for Negotiation Dialogues](https://mlsanity.com/paper/1706.05125/) (2017)
- [Do GANs leave artificial fingerprints?](https://mlsanity.com/paper/1812.11842/) (2018)
- [Talking About Large Language Models](https://mlsanity.com/paper/2212.03551/) (2022)
- [Open Problems in Cooperative AI](https://mlsanity.com/paper/2012.08630/) (2020)
- [Alignment of Language Agents](https://mlsanity.com/paper/2103.14659/) (2021)
- [Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations](https://mlsanity.com/paper/2301.04246/) (2023)
- [Taken out of context: On measuring situational awareness in LLMs](https://mlsanity.com/paper/2309.00667/) (2023)
- [Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learning](https://mlsanity.com/paper/2206.15378/) (2022)
- [Truthful AI: Developing and governing AI that does not lie](https://mlsanity.com/paper/2110.06674/) (2021)
- [Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective](https://mlsanity.com/paper/1908.04734/) (2019)
- [Is Power-Seeking AI an Existential Risk?](https://mlsanity.com/paper/2206.13353/) (2022)
- [How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions](https://mlsanity.com/paper/2309.15840/) (2023)
- [Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblocks](https://mlsanity.com/paper/2307.00175/) (2023)
- [Agent Incentives: A Causal Perspective](https://mlsanity.com/paper/2102.01685/) (2021)
- [Characterizing Manipulation from AI Systems](https://mlsanity.com/paper/2303.09387/) (2023)

## Built on by (most cited first, 30 of 40)

- [AI Sandbagging: Language Models can Strategically Underperform on Evaluations](https://mlsanity.com/paper/2406.07358/) (2024)
- [Secret Collusion among AI Agents: Multi-Agent Deception via Steganography](https://mlsanity.com/paper/2402.07510/) (2024)
- [Robust agents learn causal world models](https://mlsanity.com/paper/2402.10877/) (2024)
- [AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agents](https://mlsanity.com/paper/2409.09013/) (2024)
- [Standards for Belief Representations in LLMs](https://mlsanity.com/paper/2405.21030/) (2024)
- [The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Models](https://mlsanity.com/paper/2404.03189/) (2024)
- [The Reasons that Agents Act: Intention and Instrumental Goals](https://mlsanity.com/paper/2402.07221/) (2024)
- [A Survey on the Honesty of Large Language Models](https://mlsanity.com/paper/2409.18786/) (2024)
- [General agents contain world models](https://mlsanity.com/paper/2506.01622/) (2025)
- [From Imitation to Introspection: Probing Self-Consciousness in Language Models](https://mlsanity.com/paper/2410.18819/) (2024)
- [Difficulties with Evaluating a Deception Detector for AIs](https://mlsanity.com/paper/2511.22662/) (2025)
- [Seven Security Challenges in Cross-domain Multi-agent LLM Systems](https://mlsanity.com/paper/2505.23847/) (2025)
- [Language Models can Subtly Deceive Without Lying: A Case Study on Strategic Phrasing in Legislation](https://mlsanity.com/paper/2405.04325/) (2024)
- [The Secret Agenda: LLMs Strategically Lie and Our Current Safety Tools Are Blind](https://mlsanity.com/paper/2509.20393/) (2025)
- [Measuring Goal-Directedness](https://mlsanity.com/paper/2412.04758/) (2024)
- [Collaborate, Deliberate, Evaluate: How LLM Alignment Affects Coordinated Multi-Agent Outcomes](https://mlsanity.com/paper/2509.05882/) (2025)
- [LH-Deception: Simulating and Understanding LLM Deceptive Behaviors in Long-Horizon Interactions](https://mlsanity.com/paper/2510.03999/) (2025)
- [Towards Safe and Honest AI Agents with Neural Self-Other Overlap](https://mlsanity.com/paper/2412.16325/) (2024)
- [An Assessment of Model-On-Model Deception](https://mlsanity.com/paper/2405.12999/) (2024)
- [OpenDeception: Learning Deception and Trust in Human-AI Interaction via Multi-Agent Simulation](https://mlsanity.com/paper/2504.13707/) (2025)
- [Exploring Consciousness in LLMs: A Systematic Survey of Theories, Implementations, and Frontier Risks](https://mlsanity.com/paper/2505.19806/) (2025)
- [Enhancing LLM Agent Safety via Causal Influence Prompting](https://mlsanity.com/paper/2507.00979/) (2025)
- [Beyond Prompt-Induced Lies: Investigating LLM Deception on Benign Prompts](https://mlsanity.com/paper/2508.06361/) (2025)
- [Observation Interference in Partially Observable Assistance Games](https://mlsanity.com/paper/2412.17797/) (2024)
- [Possible Principles for Aligned Structure Learning Agents](https://mlsanity.com/paper/2410.00258/) (2024)
- [A Principle of Targeted Intervention for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2510.17697/) (2025)
- [PRISON: Unmasking the Criminal Potential of Large Language Models](https://mlsanity.com/paper/2506.16150/) (2025)
- [Learning "Partner-Aware" Collaborators in Multi-Party Collaboration](https://mlsanity.com/paper/2510.22462/) (2025)
- [Persuasion Attacks Can Decrease Effectiveness of CoT Monitoring](https://mlsanity.com/paper/2607.08066/) (2026)
- [Knowledge-Verified Emergent Deception in LLM Agents Under Conflicting Incentives](https://mlsanity.com/paper/2608.26372/) (2026)

## Abstract (arXiv)

Deceptive agents are a challenge for the safety, trustworthiness, and cooperation of AI systems. We focus on the problem that agents might deceive in order to achieve their goals (for instance, in our experiments with language models, the goal of being evaluated as truthful). There are a number of existing definitions of deception in the literature on game theory and symbolic AI, but there is no overarching theory of deception for learning agents in games. We introduce a formal definition of deception in structural causal games, grounded in the philosophy literature, and applicable to real-world machine learning systems. Several examples and results illustrate that our formal definition aligns with the philosophical and commonsense meaning of deception. Our main technical result is to provide graphical criteria for deception. We show, experimentally, that these results can be used to mitigate deception in reinforcement learning agents and language models.