mlsanityData to 7 Oct 2026

Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought

arXiv 2403.05518 ↗First posted 8 Mar 2024cs.CL cs.AIcited by 37

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Chain-of-thought prompting (CoT) has the potential to improve the explainability of language model reasoning. But CoT can also systematically misrepresent the factors influencing models' behavior -- for example, rationalizing answers in line with a user's opinion. We first create a new dataset of 9 different biases that affect GPT-3.5-Turbo and Llama-8b models. These consist of spurious-few-shot patterns, post hoc rationalization, and sycophantic settings. Models switch to the answer implied by the bias, without mentioning the effect of the bias in the CoT. To mitigate this biased reasoning problem, we introduce bias-augmented consistency training (BCT), an unsupervised fine-tuning scheme that trains models to give consistent reasoning across prompts with and without biasing features. We construct a suite testing nine forms of biased reasoning on seven question-answering tasks, and find that applying BCT to GPT-3.5-Turbo with one bias reduces the rate of biased reasoning by 86% on held-out tasks. Moreover, this model generalizes to other forms of bias, reducing biased reasoning on held-out biases by an average of 37%. As BCT generalizes to held-out biases and does not require gold labels, this method may hold promise for reducing biased reasoning from as-of-yet unknown biases and on tasks where ground truth reasoning is unavailable.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 59 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

22 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustment5 Mar
  2. 2023Measuring Faithfulness in Chain-of-Thought Reasoning17 Jul

Other references · 20

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  3. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  4. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  5. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  6. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  7. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  8. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  9. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  10. 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
  11. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  12. 2022Faithful Reasoning Using Large Language Modelscited by 130
8 more
  1. 2019On the Measure of Intelligencecited by 129
  2. 2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoningcited by 107
  3. 2023Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanationscited by 85
  4. 2023System 2 Attention (is something you might need too)cited by 85
  5. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  6. 2024Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuningcited by 26
  7. 2024Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatabilitycited by 22
  8. 2023ALMANACS: A Simulatability Benchmark for Language Model Explainabilitycited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 13; 2026: 19.

37 papers.

Sort:

By area:

  1. 2025Reasoning Models Don't Always Say What They Thinkcited by 388
  2. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  3. 2025Chain-of-Thought Reasoning In The Wild Is Not Always Faithfulcited by 166
  4. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 105
  5. 2025Are DeepSeek R1 And Other Reasoning Models More Faithful?cited by 69
  6. 2024Are Large Language Models Consistent over Value-laden Questions?cited by 58
  7. 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMscited by 58
  8. 2024Steering Without Side Effects: Improving Post-Deployment Control of Language Modelscited by 44
  9. 2025Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuningcited by 19
  10. 2025Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoningcited by 19
  11. 2025Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Modelscited by 16
  12. 2025Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanationscited by 9
25 more
  1. 2024Chain of Thought Still Thinks Fast: APriCoT Helps with Thinking Slowcited by 8
  2. 2026Good Arguments Against the People Pleasers: How Reasoning Mitigates (Yet Masks) LLM Sycophancycited by 7
  3. 2025A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languagescited by 6
  4. 2026Post-Hoc Reasoning in Chain of Thought: Decoding and Steering Pre-Committed Answerscited by 6
  5. 2025Reasoning Towards Fairness: Mitigating Bias in Language Models through Reasoning-Guided Fine-Tuningcited by 5
  6. 2026ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessionscited by 5
  7. 2026Consistency Training while Mitigating Obfuscation via Rate Matchingcited by 4
  8. 2025Can LLMs Help Uncover Insights about LLMs? A Large-Scale, Evolving Literature Analysis of Frontier LLMscited by 3
  9. 2026Training Agents to Self-Report Misbehaviorcited by 3
  10. 2026Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experimentscited by 2
  11. 2025CLARity: Reasoning Consistency Alone Can Teach Reinforced Expertscited by 1
  12. 2025Introducing Visual Scenes and Reasoning: A More Realistic Benchmark for Spoken Language Understandingcited by 1
  13. 2026Measuring and Mitigating Post-hoc Rationalization in Reverse Chain-of-Thought Generationcited by 1
  14. 2026On-Policy Consistency Training Improves LLM Safety with Minimal Capability Degradationcited by 1
  15. 2026Consistency Training Can Entrench Misalignmentcited by 1
  16. 2026Consistency Training Along the Transformer Stackcited by 1
  17. 2026How Does Alignment Tuning Shape Representations of Sycophancy and Related Cue-Induced Biases in LLMs?cited by 1
  18. 2026Position: It's Time to Optimize LLMs for Self-Consistencycited by 1
  19. 2026Self-Mined Hardness for Safety Fine-Tuningcited by 0
  20. 2026Mitigating Adaptive Attacks against Reasoning Models with Activation Consistency Trainingcited by 0
  21. 2026Training Large Language Models for Self-Explanation Faithfulnesscited by 0
  22. 2026Adaptive Triggering for Bias Correction in LLM Reasoningcited by 0
  23. 2026Chain-of-Thought Faithfulness of Reasoning Models Varies with Where and How Preference Cues Are Deliveredcited by 0
  24. 2026A Removal Based Approach to Improve LLM Faithfulness at Test-Timecited by 0
  25. 2026Unlearning Deceptive Behaviors in LLMs with Contrastive Forget Setscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Unveiling Confirmation Bias in Chain-of-Thought Reasoning
  2. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models
  4. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methods
  5. 2024Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns
  6. 2023It's Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning
  7. 2025Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
  8. 2024Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?