mlsanityData to 7 Oct 2026

On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning

arXiv 2212.08061 ↗First posted 15 Dec 2022cs.CLcited by 191

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Generating a Chain of Thought (CoT) has been shown to consistently improve large language model (LLM) performance on a wide range of NLP tasks. However, prior work has mainly focused on logical reasoning tasks (e.g. arithmetic, commonsense QA); it remains unclear whether improvements hold for more diverse types of reasoning, especially in socially situated contexts. Concretely, we perform a controlled evaluation of zero-shot CoT across two socially sensitive domains: harmful questions and stereotype benchmarks. We find that zero-shot CoT reasoning in sensitive domains significantly increases a model's likelihood to produce harmful or undesirable output, with trends holding across different prompt formats and model variants. Furthermore, we show that harmful CoTs increase with model size, but decrease with improved instruction following. Our work suggests that zero-shot CoT should be used with caution on socially important tasks, especially when marginalized groups or sensitive topics are involved.

Key facts. Cited by 191 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 46 of its 66 references that are in the map. First posted on arXiv 15 Dec 2022.

Builds on

46 of 66 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them17 Oct
  2. 2022Psychologically-informed chain-of-thought prompts for metaphor understanding in large language models16 Sep
  3. 2022Large Language Models are Zero-Shot Reasoners24 May
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 40

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  11. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
28 more
  1. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  2. 2022Red Teaming Language Models with Language Modelscited by 983
  3. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  4. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  5. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  6. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  7. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  9. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  10. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  11. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  12. 2018Gender Bias in Coreference Resolutioncited by 438
  13. 2021True Few-Shot Learning with Language Modelscited by 343
  14. 2022UL2: Unifying Language Learning Paradigmscited by 292
  15. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  16. 2022Prompting GPT-3 To Be Reliablecited by 258
  17. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  18. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 208
  19. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  20. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  21. 2020Intrinsic Bias Metrics Do Not Correlate with Application Biascited by 148
  22. 2020Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classificationcited by 98
  23. 2021An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Modelscited by 96
  24. 2019Toward Gender-Inclusive Coreference Resolutioncited by 84
  25. 2022Inverse scaling can become U-shapedcited by 76
  26. 2022On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representationscited by 75
  27. 2022Robustness of Demonstration-based Learning Under Limited Data Scenariocited by 11
  28. 2022Zero Shot Crosslingual Eye-Tracking Data Prediction using Multilingual Transformer Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 43; 2024: 72; 2025: 51; 2026: 24.

191 papers.

Sort:

By area:

  1. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  2. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  3. 2023A Comprehensive Overview of Large Language Modelscited by 639
  4. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  5. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  6. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  7. 2024A StrongREJECT for Empty Jailbreakscited by 352
  8. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  9. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  10. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  11. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  12. 2023Can Large Language Models Transform Computational Social Science?cited by 245
38 more
  1. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  2. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  3. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  4. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  5. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
  6. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  7. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  8. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  9. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  10. 2024Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worsecited by 107
  11. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 105
  12. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  13. 2024Watermark Stealing in Large Language Modelscited by 79
  14. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 75
  15. 2024Safeguarding Large Language Models: A Surveycited by 73
  16. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  17. 2023On the Learnability of Watermarks for Language Modelscited by 72
  18. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70
  19. 2024Can multiple-choice questions really be useful in detecting the abilities of LLMs?cited by 66
  20. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 65
  21. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  22. 2024Building Guardrails for Large Language Modelscited by 61
  23. 2024Design Principles for Generative AI Applicationscited by 57
  24. 2023TrustGPT: A Benchmark for Trustworthy and Responsible Large Language Modelscited by 56
  25. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  26. 2024DeAL: Decoding-time Alignment for Large Language Modelscited by 55
  27. 2023Rehearsal: Simulating Conflict to Teach Conflict Resolutioncited by 54
  28. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  29. 2023Self-Guard: Empower the LLM to Safeguard Itselfcited by 52
  30. 2024CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreferencecited by 51
  31. 2024Measuring Implicit Bias in Explicitly Unbiased Large Language Modelscited by 49
  32. 2024Disclosure and Mitigation of Gender Bias in LLMscited by 47
  33. 2024The Oscars of AI Theater: A Survey on Role-Playing with Language Modelscited by 47
  34. 2023Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Votingcited by 46
  35. 2024Revisiting Jailbreaking for Large Language Models: A Representation Engineering Perspectivecited by 44
  36. 2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 44
  37. 2023The language of prompting: What linguistic properties make a prompt successful?cited by 43
  38. 2024DELL: Generating Reactions and Explanations for LLM-Based Misinformation Detectioncited by 41

50 most cited of 191 All 191 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023An automatically discovered chain-of-thought prompt generalizes to novel models and datasets
  2. 2025Revisiting Chain-of-Thought Prompting: Zero-shot Can Be Stronger than Few-shot
  3. 2024Instance-adaptive Zero-shot Chain-of-Thought Prompting
  4. 2024Zero-Shot Chain-of-Thought Reasoning Guided by Evolutionary Algorithms in Large Language Models
  5. 2025Zero-Shot Verification-guided Chain of Thoughts
  6. 2025Innate Reasoning is Not Enough: In-Context Learning Enhances Reasoning Large Language Models with Less Overthinking
  7. 2025Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
  8. 2026Soft Guidance Starts to Outperform CoT Prompting as LLMs Improve