# Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought

arXiv: https://arxiv.org/abs/2403.05518 · HTML: https://mlsanity.com/paper/2403.05518/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 22 of its 59 references that are in the map.
- First posted on arXiv 8 Mar 2024.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 22 of 22)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [Measuring Progress on Scalable Oversight for Large Language Models](https://mlsanity.com/paper/2211.03540/) (2022)
- [Simple synthetic data reduces sycophancy in large language models](https://mlsanity.com/paper/2308.03958/) (2023)
- [The Capacity for Moral Self-Correction in Large Language Models](https://mlsanity.com/paper/2302.07459/) (2023)
- [On the Measure of Intelligence](https://mlsanity.com/paper/1911.01547/) (2019)
- [Faithful Reasoning Using Large Language Models](https://mlsanity.com/paper/2208.14271/) (2022)
- [Question Decomposition Improves the Faithfulness of Model-Generated Reasoning](https://mlsanity.com/paper/2307.11768/) (2023)
- [System 2 Attention (is something you might need too)](https://mlsanity.com/paper/2311.11829/) (2023)
- [Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations](https://mlsanity.com/paper/2307.08678/) (2023)
- [Evaluating Superhuman Models with Consistency Checks](https://mlsanity.com/paper/2306.09983/) (2023)
- [Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustment](https://mlsanity.com/paper/2403.02738/) (2024)
- [Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuning](https://mlsanity.com/paper/2401.13986/) (2024)
- [Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability](https://mlsanity.com/paper/2401.08574/) (2024)
- [ALMANACS: A Simulatability Benchmark for Language Model Explainability](https://mlsanity.com/paper/2312.12747/) (2023)

## Built on by (most cited first, 30 of 39)

- [Reasoning Models Don't Always Say What They Think](https://mlsanity.com/paper/2505.05410/) (2025)
- [Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2406.07791/) (2024)
- [Chain-of-Thought Reasoning In The Wild Is Not Always Faithful](https://mlsanity.com/paper/2503.08679/) (2025)
- [Looking Inward: Language Models Can Learn About Themselves by Introspection](https://mlsanity.com/paper/2410.13787/) (2024)
- [Are DeepSeek R1 And Other Reasoning Models More Faithful?](https://mlsanity.com/paper/2501.08156/) (2025)
- [School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs](https://mlsanity.com/paper/2508.17511/) (2025)
- [Are Large Language Models Consistent over Value-laden Questions?](https://mlsanity.com/paper/2407.02996/) (2024)
- [Steering Without Side Effects: Improving Post-Deployment Control of Language Models](https://mlsanity.com/paper/2406.15518/) (2024)
- [Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2506.22777/) (2025)
- [Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuning](https://mlsanity.com/paper/2502.09673/) (2025)
- [Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Models](https://mlsanity.com/paper/2507.12428/) (2025)
- [Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanations](https://mlsanity.com/paper/2504.05294/) (2025)
- [Chain of Thought Still Thinks Fast: APriCoT Helps with Thinking Slow](https://mlsanity.com/paper/2408.08651/) (2024)
- [A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languages](https://mlsanity.com/paper/2510.09555/) (2025)
- [Good Arguments Against the People Pleasers: How Reasoning Mitigates (Yet Masks) LLM Sycophancy](https://mlsanity.com/paper/2603.16643/) (2026)
- [Training Agents to Self-Report Misbehavior](https://mlsanity.com/paper/2602.22303/) (2026)
- [Post-Hoc Reasoning in Chain of Thought: Decoding and Steering Pre-Committed Answers](https://mlsanity.com/paper/2603.01437/) (2026)
- [Reasoning Towards Fairness: Mitigating Bias in Language Models through Reasoning-Guided Fine-Tuning](https://mlsanity.com/paper/2504.05632/) (2025)
- [ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions](https://mlsanity.com/paper/2605.24279/) (2026)
- [Consistency Training while Mitigating Obfuscation via Rate Matching](https://mlsanity.com/paper/2606.02211/) (2026)
- [Can LLMs Help Uncover Insights about LLMs? A Large-Scale, Evolving Literature Analysis of Frontier LLMs](https://mlsanity.com/paper/2502.18791/) (2025)
- [Measuring and Mitigating Post-hoc Rationalization in Reverse Chain-of-Thought Generation](https://mlsanity.com/paper/2602.14469/) (2026)
- [On-Policy Consistency Training Improves LLM Safety with Minimal Capability Degradation](https://mlsanity.com/paper/2605.21834/) (2026)
- [Consistency Training Can Entrench Misalignment](https://mlsanity.com/paper/2606.03810/) (2026)
- [Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments](https://mlsanity.com/paper/2608.16747/) (2026)
- [CLARity: Reasoning Consistency Alone Can Teach Reinforced Experts](https://mlsanity.com/paper/2510.09278/) (2025)
- [Introducing Visual Scenes and Reasoning: A More Realistic Benchmark for Spoken Language Understanding](https://mlsanity.com/paper/2511.19005/) (2025)
- [Consistency Training Along the Transformer Stack](https://mlsanity.com/paper/2606.05817/) (2026)
- [How Does Alignment Tuning Shape Representations of Sycophancy and Related Cue-Induced Biases in LLMs?](https://mlsanity.com/paper/2607.18114/) (2026)
- [Position: It's Time to Optimize LLMs for Self-Consistency](https://mlsanity.com/paper/2608.05188/) (2026)

## Abstract (arXiv)

Chain-of-thought prompting (CoT) has the potential to improve the explainability of language model reasoning. But CoT can also systematically misrepresent the factors influencing models' behavior -- for example, rationalizing answers in line with a user's opinion. We first create a new dataset of 9 different biases that affect GPT-3.5-Turbo and Llama-8b models. These consist of spurious-few-shot patterns, post hoc rationalization, and sycophantic settings. Models switch to the answer implied by the bias, without mentioning the effect of the bias in the CoT. To mitigate this biased reasoning problem, we introduce bias-augmented consistency training (BCT), an unsupervised fine-tuning scheme that trains models to give consistent reasoning across prompts with and without biasing features. We construct a suite testing nine forms of biased reasoning on seven question-answering tasks, and find that applying BCT to GPT-3.5-Turbo with one bias reduces the rate of biased reasoning by 86% on held-out tasks. Moreover, this model generalizes to other forms of bias, reducing biased reasoning on held-out biases by an average of 37%. As BCT generalizes to held-out biases and does not require gold labels, this method may hold promise for reducing biased reasoning from as-of-yet unknown biases and on tasks where ground truth reasoning is unavailable.