# Prompting Techniques for Reducing Social Bias in LLMs through System 1 and System 2 Cognitive Processes

arXiv: https://arxiv.org/abs/2404.17218 · HTML: https://mlsanity.com/paper/2404.17218/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 26 Apr 2024.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Toxicity in ChatGPT: Analyzing Persona-assigned Language Models](https://mlsanity.com/paper/2304.05335/) (2023)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://mlsanity.com/paper/2103.00453/) (2021)
- [Towards Understanding and Mitigating Social Biases in Language Models](https://mlsanity.com/paper/2106.13219/) (2021)
- [Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters](https://mlsanity.com/paper/2212.10001/) (2022)
- [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://mlsanity.com/paper/1906.04571/) (2019)
- [Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs](https://mlsanity.com/paper/2311.04892/) (2023)
- [SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks](https://mlsanity.com/paper/2305.17390/) (2023)
- [Thinking Fast and Slow in Large Language Models](https://mlsanity.com/paper/2212.05206/) (2022)
- [Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting](https://mlsanity.com/paper/2309.07034/) (2023)
- [What Evidence Do Language Models Find Convincing?](https://mlsanity.com/paper/2402.11782/) (2024)
- [Towards Better Chain-of-Thought Prompting Strategies: A Survey](https://mlsanity.com/paper/2310.04959/) (2023)
- [Self-Debiasing Large Language Models: Zero-Shot Recognition and Reduction of Stereotypes](https://mlsanity.com/paper/2402.01981/) (2024)
- [Evaluating Gender Bias in Large Language Models via Chain-of-Thought Prompting](https://mlsanity.com/paper/2401.15585/) (2024)
- [Do Large Language Models Show Decision Heuristics Similar to Humans? A Case Study Using GPT-3.5](https://mlsanity.com/paper/2305.04400/) (2023)
- [Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Models](https://mlsanity.com/paper/2309.08902/) (2023)
- [Evaluating Large Language Models with NeuBAROCO: Syllogistic Reasoning Ability and Human-like Biases](https://mlsanity.com/paper/2306.12567/) (2023)
- [Will the Real Linda Please Stand up...to Large Language Models? Examining the Representativeness Heuristic in LLMs](https://mlsanity.com/paper/2404.01461/) (2024)
- [Can A Cognitive Architecture Fundamentally Enhance LLMs? Or Vice Versa?](https://mlsanity.com/paper/2401.10444/) (2024)
- [AXOLOTL: Fairness through Assisted Self-Debiasing of Large Language Model Outputs](https://mlsanity.com/paper/2403.00198/) (2024)

## Built on by (most cited first, 30 of 39)

- [LLMs for Explainable AI: A Comprehensive Survey](https://mlsanity.com/paper/2504.00125/) (2025)
- [Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Context](https://mlsanity.com/paper/2406.05972/) (2024)
- [The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs](https://mlsanity.com/paper/2509.03730/) (2025)
- [A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models](https://mlsanity.com/paper/2509.03871/) (2025)
- [Dual Reasoning: A GNN-LLM Collaborative Framework for Knowledge Graph Question Answering](https://mlsanity.com/paper/2406.01145/) (2024)
- [Reasoning on a Spectrum: Aligning LLMs to System 1 and System 2 Thinking](https://mlsanity.com/paper/2502.12470/) (2025)
- [Exploring Changes in Nation Perception with Nationality-Assigned Personas in LLMs](https://mlsanity.com/paper/2406.13993/) (2024)
- [System 2 thinking in OpenAI's o1-preview model: Near-perfect performance on a mathematics exam](https://mlsanity.com/paper/2410.07114/) (2024)
- [The Invisible Hand: Unveiling Provider Bias in Large Language Models for Code Generation](https://mlsanity.com/paper/2501.07849/) (2025)
- [Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?](https://mlsanity.com/paper/2506.00751/) (2025)
- [Guiding LLM Decision-Making with Fairness Reward Models](https://mlsanity.com/paper/2507.11344/) (2025)
- [An Investigation of Prompt Variations for Zero-shot LLM-based Rankers](https://mlsanity.com/paper/2406.14117/) (2024)
- ['Since Lawyers are Males..': Examining Implicit Gender Bias in Hindi Language Generation by LLMs](https://mlsanity.com/paper/2409.13484/) (2024)
- [Exposing Product Bias in LLM Investment Recommendation](https://mlsanity.com/paper/2503.08750/) (2025)
- [Poor Alignment and Steerability of Large Language Models: Evidence from College Admission Essays](https://mlsanity.com/paper/2503.20062/) (2025)
- ["A Woman is More Culturally Knowledgeable than A Man?": The Effect of Personas on Cultural Norm Interpretation in LLMs](https://mlsanity.com/paper/2409.11636/) (2024)
- [HM3: Heterogeneous Multi-Class Model Merging](https://mlsanity.com/paper/2409.19173/) (2024)
- [Giving AI Personalities Leads to More Human-Like Reasoning](https://mlsanity.com/paper/2502.14155/) (2025)
- [PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimization](https://mlsanity.com/paper/2507.16679/) (2025)
- [Cross-Lingual Prompt Steerability: Towards Accurate and Robust LLM Behavior across Languages](https://mlsanity.com/paper/2512.02841/) (2025)
- [Talent or Luck? Evaluating Attribution Bias in Large Language Models](https://mlsanity.com/paper/2505.22910/) (2025)
- [From Anger to Joy: How Nationality Personas Shape Emotion Attribution in Large Language Models](https://mlsanity.com/paper/2506.02431/) (2025)
- [BiasFreeBench: a Benchmark for Mitigating Bias in Large Language Model Responses](https://mlsanity.com/paper/2510.00232/) (2025)
- [Bias Mitigation Agent: Optimizing Source Selection for Fair and Balanced Knowledge Retrieval](https://mlsanity.com/paper/2508.18724/) (2025)
- [Risk Profiling and Modulation for LLMs](https://mlsanity.com/paper/2509.23058/) (2025)
- [Aligned but Stereotypical? How System Prompts Shape Demographic Bias in LLM-Based Text-to-Image Models](https://mlsanity.com/paper/2512.04981/) (2025)
- [Routing Sensitivity Without Controllability: A Diagnostic Study of Fairness in MoE Language Models](https://mlsanity.com/paper/2603.27141/) (2026)
- [The role of System 1 and System 2 semantic memory structure in human and LLM biases](https://mlsanity.com/paper/2604.12816/) (2026)
- [Strategic Reflectivism In Intelligent Systems](https://mlsanity.com/paper/2505.22987/) (2025)
- [Debiasing LLMs by Masking Unfairness-Driving Attention Heads](https://mlsanity.com/paper/2510.10142/) (2025)

## Abstract (arXiv)

Dual process theory posits that human cognition arises via two systems. System 1, which is a quick, emotional, and intuitive process, which is subject to cognitive biases, and System 2, is a slow, onerous, and deliberate process. Prior research in LLMs found that using chain-of-thought (CoT) prompting in LLMs, which has been often compared to System 2 reasoning, can lead to reduced gender bias. Along these lines, we investigate the relationship between bias, CoT prompting, a direct debiasing, and dual process theory modeling in LLMs. We compare zero-shot CoT, debiasing, and dual process theory-based prompting strategies on two bias datasets spanning nine different social bias categories. We incorporate human and machine personas to determine whether LLM modeling of the effects of dual process theory exist independent of explicit persona models or are tied to the LLM's modeling of human-like generation. We find that a human persona, debiasing, System 2, and CoT prompting all tend to reduce social biases in LLMs, though the best combination of features depends on the exact model and bias category -- resulting in up to a 33 percent drop in stereotypical judgments by an LLM.