mlsanityData to 14 Oct 2026

Thinking Fair and Slow: On the Efficacy of Structured Prompts for Debiasing Language Models

arXiv 2405.10431 ↗First posted 16 May 2024cs.CLcited by 45

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Existing debiasing techniques are typically training-based or require access to the model's internals and output distributions, so they are inaccessible to end-users looking to adapt LLM outputs for their particular needs. In this study, we examine whether structured prompting techniques can offer opportunities for fair text generation. We evaluate a comprehensive end-user-focused iterative framework of debiasing that applies System 2 thinking processes for prompts to induce logical, reflective, and critical text generation, with single, multi-step, instruction, and role-based variants. By systematically evaluating many LLMs across many datasets and different prompting strategies, we show that the more complex System 2-based Implicative Prompts significantly improve over other techniques demonstrating lower mean bias in the outputs with competitive performance on the downstream tasks. Our work offers research directions for the design and the potential of end-user-focused evaluative frameworks for LLM use.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 37 references that are in the map. First posted on arXiv 16 May 2024.

Builds on

25 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  5. 2023Mistral 7Bcited by 3,164
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2024Mixtral of Expertscited by 1,823
  9. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  10. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  11. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  12. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
12 more
  1. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 285
  2. 2022Prompting GPT-3 To Be Reliablecited by 260
  3. 2020Measuring and Reducing Gendered Correlations in Pre-trained Modelscited by 224
  4. 2023Better Zero-Shot Reasoning with Role-Play Promptingcited by 223
  5. 2019Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphologycited by 200
  6. 2020Towards Debiasing Sentence Representationscited by 171
  7. 2023Guiding Large Language Models via Directional Stimulus Promptingcited by 95
  8. 2023A Survey on Fairness in Large Language Modelscited by 95
  9. 2023Fairness-guided Few-shot Prompting for Large Language Modelscited by 53
  10. 2023Unveiling Gender Bias in Terms of Profession Across LLMs: Analyzing and Addressing Sociological Implicationscited by 29
  11. 2023An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Modelscited by 16
  12. 2023All Should Be Equal in the Eyes of Language Models: Counterfactually Aware Fair Text Generationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 25; 2026: 17.

45 papers.

Sort:

By area:

  1. 2024LLM-Driven Robots Risk Enacting Discrimination, Violence, and Unlawful Actionscited by 22
  2. 2024IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answeringcited by 19
  3. 2025Reasoning on a Spectrum: Aligning LLMs to System 1 and System 2 Thinkingcited by 16
  4. 2025Strategic Tradeoffs Between Humans and AI in Multi-Agent Bargainingcited by 11
  5. 2024Persona Setting Pitfall: Persistent Outgroup Biases in Large Language Models Arising from Social Identity Adoptioncited by 9
  6. 2026CatRAG: Functor-Guided Structural Debiasing with Retrieval Augmentation for Fair LLMscited by 9
  7. 2025FairEval: Evaluating Fairness in LLM-Based Recommendations with Personality Awarenesscited by 8
  8. 2025Poor Alignment and Steerability of Large Language Models: Evidence from College Admission Essayscited by 6
  9. 2025Mitigating Biases in Language Models via Bias Unlearningcited by 6
  10. 2025Reasoning Towards Fairness: Mitigating Bias in Language Models through Reasoning-Guided Fine-Tuningcited by 5
  11. 2026Bi-directional Bias Attribution: Debiasing Large Language Models without Modifying Promptscited by 5
  12. 2025On Fairness of Task Arithmetic: The Role of Task Vectorscited by 3
33 more
  1. 2025KLAAD: Refining Attention Mechanisms to Reduce Societal Bias in Generative Language Modelscited by 3
  2. 2025Bias after Prompting: Persistent Discrimination in Large Language Modelscited by 3
  3. 2025DSO: Direct Steering Optimization for Bias Mitigationcited by 3
  4. 2025Investigating Gender Bias in LLM-Generated Stories via Psychological Stereotypescited by 2
  5. 2025REFER: Mitigating Bias in Opinion Summarisation via Frequency Framed Promptingcited by 2
  6. 2025Bias Mitigation or Cultural Commonsense? Evaluating LLMs with a Japanese Datasetcited by 2
  7. 2025BiasFreeBench: a Benchmark for Mitigating Bias in Large Language Model Responsescited by 2
  8. 2025Can SAEs reveal and mitigate racial biases of LLMs in healthcare?cited by 2
  9. 2026When LLMs Imagine People: A Human-Centered Persona Brainstorm Audit for Bias and Fairness in Creative Applicationscited by 2
  10. 2026When Bigger Isn't Better: A Comprehensive Fairness Evaluation of Political Bias in Multi-News Summarisationcited by 2
  11. 2025'Rich Dad, Poor Lad': How do Large Language Models Contextualize Socioeconomic Factors in College Admission ?cited by 1
  12. 2025Evaluating Implicit Biases in LLM Reasoning through Logic Grid Puzzlescited by 1
  13. 2025Aligned but Stereotypical? How System Prompts Shape Demographic Bias in LLM-Based Text-to-Image Modelscited by 1
  14. 2026Identity-Robust Language Model Generation via Content Integrity Preservationcited by 1
  15. 2026Uncertainty and Fairness Awareness in LLM-Based Recommendation Systemscited by 1
  16. 2026DeFrame: Debiasing Large Language Models Against Framing Effectscited by 1
  17. 2026Mitigating Prompt-Induced Cognitive Biases in General-Purpose AI for Software Engineeringcited by 1
  18. 2026Bias in the Tails: How Name-conditioned Evaluative Framing in Resume Summaries Destabilizes LLM-based Hiringcited by 1
  19. 2025DeCAP: Context-Adaptive Prompt Generation for Debiasing Zero-shot Question Answering in Large Language Modelscited by 0
  20. 2025Fairness Is Not Enough: Auditing Competence and Intersectional Bias in AI-powered Resume Screeningcited by 0
  21. 2025Labels or Input? Rethinking Augmentation in Multimodal Hate Detectioncited by 0
  22. 2025SMARTER: A Data-efficient Framework to Improve Toxicity Detection with Explanation via Self-augmenting Large Language Modelscited by 0
  23. 2025Bridging Fairness and Explainability: Can Input-Based Explanations Promote Fairness in Hate Speech Detection?cited by 0
  24. 2025Debiasing LLMs by Masking Unfairness-Driving Attention Headscited by 0
  25. 2025Improving Fairness of Large Language Model-Based ICU Mortality Prediction via Case-Based Promptingcited by 0
  26. 2026LLM Nepotism in Organizational Governancecited by 0
  27. 2026S2H-DPO: Hardness-Aware Preference Optimization for Vision-Language Modelscited by 0
  28. 2026Beyond Single Plots: A Benchmark for Question Answering on Multi-Chartscited by 0
  29. 2026TriAlign: Towards Universal Truth Consistency in Personalized LLM Alignmentcited by 0
  30. 2026Fine-Tuned Multi-Agent Framework for Detecting OCEAN in Life Narrativescited by 0
  31. 2026AutoJourn: Multi-Perspective Summarisation, Bias Detection and Bias Neutralisation for LLM-Generated News in Automated Journalismcited by 0
  32. 2026Beyond Outcome Gaps: Process-Aware Fairness Diagnosis for LLM-based Multi-Agent Decision Systemscited by 0
  33. 2026Extracting Persona Subspaces Through Iterative Nullspace Projection For Modulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LIDAO: Towards Limited Interventions for Debiasing (Large) Language Models
  2. 2025Rethinking Prompt-based Debiasing in Large Language Models
  3. 2024Expert-Guided Extinction of Toxic Tokens for Debiased Generation
  4. 2024Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustment
  5. 2024A Multi-LLM Debiasing Framework
  6. 2026DebiasRAG: A Tuning-Free Path to Fair Generation in Large Language Models through Retrieval-Augmented Generation
  7. 2023Establishing Performance Baselines in Fine-Tuning, Retrieval-Augmented Generation and Soft-Prompting for Non-Specialist LLM Users
  8. 2025BiasFilter: An Inference-Time Debiasing Framework for Large Language Models