# Evaluating & Reducing Deceptive Dialogue From Language Models with Multi-turn RL

arXiv: https://arxiv.org/abs/2510.14318 · HTML: https://mlsanity.com/paper/2510.14318/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 65 references that are in the map.
- First posted on arXiv 16 Oct 2025.

## Builds on (most cited first, 30 of 30)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf](https://mlsanity.com/paper/2309.04658/) (2023)
- [Deal or No Deal? End-to-End Learning for Negotiation Dialogues](https://mlsanity.com/paper/1706.05125/) (2017)
- [Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good](https://mlsanity.com/paper/1906.06725/) (2019)
- [AI Deception: A Survey of Examples, Risks, and Potential Solutions](https://mlsanity.com/paper/2308.14752/) (2023)
- [Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark](https://mlsanity.com/paper/2304.03279/) (2023)
- [Alignment of Language Agents](https://mlsanity.com/paper/2103.14659/) (2021)
- [Decoupling Strategy and Generation in Negotiation Dialogues](https://mlsanity.com/paper/1808.09637/) (2018)
- [Large Language Models can Strategically Deceive their Users when Put Under Pressure](https://mlsanity.com/paper/2311.07590/) (2023)
- [User Behavior Simulation with Large Language Model based Agents](https://mlsanity.com/paper/2306.02552/) (2023)
- [Language Models Learn to Mislead Humans via RLHF](https://mlsanity.com/paper/2409.12822/) (2024)
- [Deception Abilities Emerged in Large Language Models](https://mlsanity.com/paper/2307.16513/) (2023)
- [How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions](https://mlsanity.com/paper/2309.15840/) (2023)
- [Measuring Bargaining Abilities of LLMs: A Benchmark and A Buyer-Enhancement Method](https://mlsanity.com/paper/2402.15813/) (2024)
- [Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Models](https://mlsanity.com/paper/2308.01404/) (2023)
- [AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agents](https://mlsanity.com/paper/2409.09013/) (2024)
- [Language Models can Subtly Deceive Without Lying: A Case Study on Strategic Phrasing in Legislation](https://mlsanity.com/paper/2405.04325/) (2024)
- [Towards Safe and Honest AI Agents with Neural Self-Other Overlap](https://mlsanity.com/paper/2412.16325/) (2024)

## Built on by (most cited first, 10 of 10)

- [The Truthfulness Spectrum Hypothesis](https://mlsanity.com/paper/2602.20273/) (2026)
- [SPADE-Bench: Evaluating Spontaneous Strategic Deception in Agents via Plan-Action Divergence](https://mlsanity.com/paper/2606.02380/) (2026)
- [Adversarial Robustness of Activation Steering in Large Language Models](https://mlsanity.com/paper/2606.07696/) (2026)
- [Scaling Trends for Lie Detector Oversight in Preference Learning](https://mlsanity.com/paper/2607.01567/) (2026)
- [To Lie or Not to Lie? Investigating The Biased Spread of Global Lies by LLMs](https://mlsanity.com/paper/2604.06552/) (2026)
- [When Agents Lie: Premeditation, Persistence, and Exploitation in Repeated Games](https://mlsanity.com/paper/2607.05132/) (2026)
- [Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models](https://mlsanity.com/paper/2609.36254/) (2026)
- [Deception by Omission: Language Models Knowingly Hide Their Mistakes](https://mlsanity.com/paper/2610.11351/) (2026)
- [Training User Simulators with Latent Intrinsic Feedback](https://mlsanity.com/paper/2610.13617/) (2026)
- [Understanding and Reducing Misalignment in Persuasion Optimization](https://mlsanity.com/paper/2610.16070/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) interact with millions of people worldwide in applications such as customer support, education and healthcare. However, their ability to produce deceptive outputs, whether intentionally or inadvertently, poses significant safety concerns. The unpredictable nature of LLM behavior, combined with insufficient safeguards against hallucination, misinformation, and user manipulation, makes their misuse a serious, real-world risk. In this paper, we investigate the extent to which LLMs engage in deception within dialogue, and propose the belief misalignment metric to quantify deception. We evaluate deception across four distinct dialogue scenarios, using five established deception detection metrics and our proposed metric. Our findings reveal this novel deception measure correlates more closely with human judgments than any existing metrics we test. Additionally, our benchmarking of eight state-of-the-art models indicates that LLMs naturally exhibit deceptive behavior in approximately 26% of dialogue turns, even when prompted with seemingly benign objectives. When prompted to deceive, LLMs are capable of increasing deceptiveness by as much as 31% relative to baselines. Unexpectedly, models trained with RLHF, the predominant approach for ensuring the safety of widely-deployed LLMs, still exhibit deception at a rate of 43% on average. Given that deception in dialogue is a behavior that develops over an interaction history, its effective evaluation and mitigation necessitates moving beyond single-utterance analyses. We introduce a multi-turn reinforcement learning methodology to fine-tune LLMs to reduce deceptive behaviors, leading to a 77.6% reduction compared to other instruction-tuned models.