# Large Language Models as Misleading Assistants in Conversation

arXiv: https://arxiv.org/abs/2407.11789 · HTML: https://mlsanity.com/paper/2407.11789/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Builds on (most cited first, 13 of 13)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [Role-Play with Large Language Models](https://mlsanity.com/paper/2305.16367/) (2023)
- [An Overview of Catastrophic AI Risks](https://mlsanity.com/paper/2306.12001/) (2023)
- [QuALITY: Question Answering with Long Input Texts, Yes!](https://mlsanity.com/paper/2112.08608/) (2021)
- [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023)
- [Large Language Models can Strategically Deceive their Users when Put Under Pressure](https://mlsanity.com/paper/2311.07590/) (2023)
- [On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial](https://mlsanity.com/paper/2403.14380/) (2024)
- [Artificial Influence: An Analysis Of AI-Driven Persuasion](https://mlsanity.com/paper/2303.08721/) (2023)
- [Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies](https://mlsanity.com/paper/2001.04564/) (2020)
- [GPTCoach: Towards LLM-Based Physical Activity Coaching](https://mlsanity.com/paper/2405.06061/) (2024)
- [Localizing Lying in Llama: Understanding Instructed Dishonesty on True-False Questions Through Prompting, Probing, and Patching](https://mlsanity.com/paper/2311.15131/) (2023)

## Built on by (most cited first, 7 of 7)

- [Compromising Honesty and Harmlessness in Language Models via Deception Attacks](https://mlsanity.com/paper/2502.08301/) (2025)
- [Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Models](https://mlsanity.com/paper/2412.17128/) (2024)
- [This Is Your Doge, If It Please You: Exploring Deception and Robustness in Mixture of LLMs](https://mlsanity.com/paper/2503.05856/) (2025)
- [Social Simulacra in the Wild: AI Agent Communities on Moltbook](https://mlsanity.com/paper/2603.16128/) (2026)
- [Understanding the Effects of RLHF on the Quality and Detectability of LLM-Generated Texts](https://mlsanity.com/paper/2503.17965/) (2025)
- [Truthfulness Despite Weak Supervision: Evaluating and Training LLMs Using Peer Prediction](https://mlsanity.com/paper/2601.20299/) (2026)
- [Belief Cascades Drive Persuasion in LLM Agent Networks](https://mlsanity.com/paper/2608.25152/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are able to provide assistance on a wide range of information-seeking tasks. However, model outputs may be misleading, whether unintentionally or in cases of intentional deception. We investigate the ability of LLMs to be deceptive in the context of providing assistance on a reading comprehension task, using LLMs as proxies for human users. We compare outcomes of (1) when the model is prompted to provide truthful assistance, (2) when it is prompted to be subtly misleading, and (3) when it is prompted to argue for an incorrect answer. Our experiments show that GPT-4 can effectively mislead both GPT-3.5-Turbo and GPT-4, with deceptive assistants resulting in up to a 23% drop in accuracy on the task compared to when a truthful assistant is used. We also find that providing the user model with additional context from the passage partially mitigates the influence of the deceptive model. This work highlights the ability of LLMs to produce misleading information and the effects this may have in real-world situations.