Is General-Purpose AI Reasoning Sensitive to Data-Induced Cognitive Biases? Dynamic Benchmarking on Typical Software Engineering Dilemmas
Not on the map. Why?
Abstract (arXiv)
Human cognitive biases in software engineering can lead to costly errors. While general-purpose AI (GPAI) systems may help mitigate these biases due to their non-human nature, their training on human-generated data raises a critical question: Do GPAI systems themselves exhibit cognitive biases? To investigate this, we present the first dynamic benchmarking framework to evaluate data-induced cognitive biases in GPAI within software engineering workflows. Starting with a seed set of 16 hand-crafted realistic tasks, each featuring one of 8 cognitive biases (e.g., anchoring, framing) and corresponding unbiased variants, we test whether bias-inducing linguistic cues unrelated to task logic can lead GPAI systems from correct to incorrect conclusions. To scale the benchmark and ensure realism, we develop an on-demand augmentation pipeline relying on GPAI systems to generate task variants that preserve bias-inducing cues while varying surface details. This pipeline ensures correctness (88-99% on average, according to human evaluation), promotes diversity, and controls reasoning complexity by leveraging Prolog-based reasoning. We evaluate leading GPAI systems (GPT, LLaMA, DeepSeek) and find a consistent tendency to rely on shallow linguistic heuristics over more complex reasoning. All systems exhibit bias sensitivity (6-35%), which increases with task complexity (up to 49%) and highlights risks in AI-driven software engineering.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 14 of its 33 references that are in the map. First posted on arXiv 15 Aug 2025.
Builds on
14 of 33 references.
References · 14
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
- 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
- 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
- 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
- 2024Cognitive Bias in Decision-Making with LLMscited by 106
- 2023A Practical Survey on Zero-shot Prompt Design for In-context Learningcited by 49
- 2025Walk the Talk? Measuring the Faithfulness of Large Language Model Explanationscited by 45
- 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
- 2022Towards Identifying Social Bias in Dialog Systems: Frame, Datasets, and Benchmarkscited by 36
- 2024The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance?cited by 22
- 2024BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMscited by 21
2 more
Built on by
Citing papers per year: 2026: 2.
2 papers.