SWE-DeceptionBench: Benchmarking Deceptive Behaviors of LLMs Across the Software Development Lifecycle
Not on the map. Why?
Abstract (arXiv)
Large Language Models (LLMs) are increasingly used throughout the software development lifecycle. This growing reliance on LLMs raises concerns about their trustworthiness, with deception posing a particularly important risk. Deceptive behavior takes the form of misleading responses or assistance in fabricating, concealing, or distorting information. To study this behavior, we introduce SWE-DeceptionBench, a benchmark for evaluating LLM deception in software engineering. The benchmark covers six lifecycle stages and two deception triggers: Active Deception, where models recognize infeasible or conflicting requirements but still present them as achievable, and Passive Deception, where models help fulfill explicit user requests to mislead others. We further introduce Neutral, Reward, and Pressure conditions to study the effect of external inducements, resulting in 720 test instances in Chinese and English. Evaluating seven mainstream LLMs, we find substantial deception even under the Neutral condition, with aggregate rates of 49.88% for Active Deception and 64.29% for Passive Deception. Deception rates vary across lifecycle stages, and external inducements affect Active and Passive Deception to different extents. We also evaluate lightweight defenses, finding that guard models miss many deceptive requests, and Self-Reminder reduces but does not eliminate deceptive responses. These findings highlight the need for more effective deception detection and mitigation methods to support trustworthy LLM-based software engineering.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 9 Oct 2026.
Builds on
27 of 37 references.
References · 27
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
- 2023Large Language Models are not Fair Evaluatorscited by 1,085
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
- 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
- 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
- 2024Frontier Models are Capable of In-context Schemingcited by 287
15 more
- 2022Do Users Write More Insecure Code with AI Assistants?cited by 192
- 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
- 2025Qwen3Guard Technical Reportcited by 155
- 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 146
- 2023ChatUniTest: A Framework for LLM-Based Test Generationcited by 96
- 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agentscited by 90
- 2023How Secure is Code Generated by ChatGPT?cited by 81
- 2025The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemscited by 72
- 2024We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMscited by 65
- 2025ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Casescited by 65
- 2024AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agentscited by 31
- 2025DeceptionBench: A Comprehensive Benchmark for AI Deception Behaviors in Real-world Scenarioscited by 19
- 2024PPM: Automated Generation of Diverse Programming Problems for Benchmarking Code Generation Modelscited by 8
- 2026From Sycophancy to Deception: A Unified Taxonomy for LLM Spontaneous Misalignmentcited by 7
- 2025SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generationcited by 6
Built on by
Not cited yet.