mlsanityData to 14 Oct 2026

Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias

arXiv 2308.00225 ↗First posted 1 Aug 2023cs.AI cs.CY cs.LGcited by 53

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Recent studies show that instruction tuning (IT) and reinforcement learning from human feedback (RLHF) improve the abilities of large language models (LMs) dramatically. While these tuning methods can help align models with human objectives and generate high-quality text, not much is known about their potential adverse effects. In this work, we investigate the effect of IT and RLHF on decision making and reasoning in LMs, focusing on three cognitive biases - the decoy effect, the certainty effect, and the belief bias - all of which are known to influence human decision-making and reasoning. Our findings highlight the presence of these biases in various models from the GPT-3, Mistral, and T5 families. Notably, we find a stronger presence of biases in models that have undergone instruction tuning, such as Flan-T5, Mistral-Instruct, GPT3.5, and GPT4. Our work constitutes a step toward comprehending cognitive biases in instruction-tuned LMs, which is crucial for the development of more reliable and unbiased language models.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 32 references that are in the map. First posted on arXiv 1 Aug 2023.

Builds on

15 of 32 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  8. 2022Using cognitive psychology to understand GPT-3cited by 398
  9. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
  10. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  11. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
  12. 2022Language models show human-like content effects on reasoning taskscited by 173
2 more
  1. 2022Thinking Fast and Slow in Large Language Modelscited by 137
  2. 2022Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Biascited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 14; 2025: 23; 2026: 14.

53 papers.

Sort:

By area:

  1. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  2. 2024Cognitive Bias in Decision-Making with LLMscited by 106
  3. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 83
  4. 2024Do LLMs write like humans? Variation in grammatical and rhetorical stylescited by 62
  5. 2025Base Models Beat Aligned Models at Randomness and Creativitycited by 51
  6. 2024A Comprehensive Evaluation of Cognitive Biases in LLMscited by 34
  7. 2024(Ir)rationality and Cognitive Biases in Large Language Modelscited by 23
  8. 2024Cognitive Biases in Large Language Models: A Survey and Mitigation Experimentscited by 20
  9. 2023Instruction Tuning with Human Curriculumcited by 15
  10. 2024CBEval: A framework for evaluating and interpreting cognitive biases in LLMscited by 12
  11. 2025BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language Modelscited by 11
  12. 2025The Bias is in the Details: An Assessment of Cognitive Bias in LLMscited by 10
38 more
  1. 2025Detecting and Filtering Unsafe Training Data via Data Attribution with Denoised Representationcited by 9
  2. 2024Cognitive Biases in Large Language Models for News Recommendationcited by 8
  3. 2026Gemma Needs Help: Investigating and Mitigating Emotional Instability in LLMscited by 7
  4. 2025Do Emotions Really Affect Argument Convincingness? A Dynamic Approach with LLM-based Manipulation Checkscited by 6
  5. 2025VoiceBBQ: Investigating Effect of Content and Acoustics in Social Bias of Spoken Language Modelcited by 6
  6. 2024MindScope: Exploring cognitive biases in large language models through Multi-Agent Systemscited by 5
  7. 2025Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMscited by 5
  8. 2025Exploiting Primacy Effect To Improve Large Language Modelscited by 5
  9. 2025The Other Mind: How Language Models Exhibit Human Temporal Cognitioncited by 5
  10. 2025Self-Adaptive Cognitive Debiasing for Large Language Models in Decision-Makingcited by 4
  11. 2025Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instructioncited by 4
  12. 2026Alignment Makes Language Models Normative, Not Descriptivecited by 4
  13. 2025Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Frameworkcited by 3
  14. 2025Fine-grained Analysis of Brain-LLM Alignment through Input Attributioncited by 3
  15. 2024View From Above: A Framework for Evaluating Distribution Shifts in Model Behaviorcited by 2
  16. 2024Rolling in the deep of cognitive and AI biasescited by 2
  17. 2024Irrelevant Alternatives Bias Large Language Model Hiring Decisionscited by 2
  18. 2024Humanlike Cognitive Patterns as Emergent Phenomena in Large Language Modelscited by 2
  19. 2025Analysing Moral Bias in Finetuned LLMs through Mechanistic Interpretabilitycited by 2
  20. 2025Social Simulations with Large Language Model Risk Utopian Illusioncited by 2
  21. 2025Mitigating the Threshold Priming Effect in Large Language Model-Based Relevance Judgments via Personality Infusingcited by 2
  22. 2026A Layer-wise Analysis of Supervised Fine-Tuningcited by 2
  23. 2026AnchorBench: A Multi-Pathway Benchmark for the Anchoring Effect in LLMscited by 2
  24. 2025GG-BBQ: German Gender Bias Benchmark for Question Answeringcited by 1
  25. 2025Pathological Truth Bias in Vision-Language Modelscited by 1
  26. 2025Beyond Single Bugs: Benchmarking Large Language Models for Multi-Vulnerability Detectioncited by 1
  27. 2026Layer-wise Positional Bias in Short-Context Language Modelingcited by 1
  28. 2026Motivation in Large Language Modelscited by 1
  29. 2026Rule vs.Consequence: Dissociable Internal Representations of Moral Reasoning in LLMscited by 1
  30. 2024People will agree what I think: Investigating LLM's False Consensus Effectcited by 0
  31. 2025Through the LLM Looking Glass: A Socratic Probing of Donkeys, Elephants, and Marketscited by 0
  32. 2025Towards Contamination Resistant Benchmarkscited by 0
  33. 2025Cross-Modal Content Optimization for Steering Web Agent Preferencescited by 0
  34. 2025A Multifaceted Analysis of Social Biases in Large Language Modelscited by 0
  35. 2026Mitigating Cognitive Bias in RLHF by Altering Rationalitycited by 0
  36. 2026Evaluating Chinese Ambiguity Understanding in Large Language Modelscited by 0
  37. 2026Whose Norms? Disentangling Cultural and Personal Alignment in Large Language Modelscited by 0
  38. 2026Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detectioncited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026CogBias: Measuring and Mitigating Cognitive Bias in Large Language Models
  2. 2026Conditional Cognitive Biases in LLMs: How Biased User Turns Modulate In-Context Reasoning
  3. 2023Psychometric Predictive Power of Large Language Models
  4. 2023Can Instruction Fine-Tuned Language Models Identify Social Bias through Prompting?
  5. 2024Does Instruction Tuning Make LLMs More Consistent?
  6. 2024Large Language Models are Biased Reinforcement Learners
  7. 2024Anchoring Bias in Large Language Models: An Experimental Study
  8. 2026Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language Models