# Mitigating Spurious Correlations Between Question and Answer via Chain-of-Thought Correctness Perception Distillation

arXiv: https://arxiv.org/abs/2509.05602 · HTML: https://mlsanity.com/paper/2509.05602/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 34 of its 38 references that are in the map.
- First posted on arXiv 6 Sep 2025.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 30 of 34)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Large Language Models Are Reasoning Teachers](https://mlsanity.com/paper/2212.10071/) (2022)
- [Teaching Small Language Models to Reason](https://mlsanity.com/paper/2212.08410/) (2022)
- [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://mlsanity.com/paper/2110.07178/) (2021)
- [Specializing Smaller Language Models towards Multi-Step Reasoning](https://mlsanity.com/paper/2301.12726/) (2023)
- [Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step](https://mlsanity.com/paper/2306.14050/) (2023)
- [Explanations from Large Language Models Make Small Reasoners Better](https://mlsanity.com/paper/2210.06726/) (2022)
- [Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sources](https://mlsanity.com/paper/2305.13269/) (2023)
- [SCOTT: Self-Consistent Chain-of-Thought Distillation](https://mlsanity.com/paper/2305.01879/) (2023)
- [Learning From Mistakes Makes LLM Better Reasoner](https://mlsanity.com/paper/2310.20689/) (2023)
- [Can LLMs Learn from Previous Mistakes? Investigating LLMs' Errors to Boost for Reasoning](https://mlsanity.com/paper/2403.20046/) (2024)
- [AI Flow at the Network Edge](https://mlsanity.com/paper/2411.12469/) (2024)
- [Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational Agents](https://mlsanity.com/paper/2310.09343/) (2023)
- [Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment](https://mlsanity.com/paper/2311.04072/) (2023)
- [PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning](https://mlsanity.com/paper/2305.13888/) (2023)
- [MCC-KD: Multi-CoT Consistent Knowledge Distillation](https://mlsanity.com/paper/2310.14747/) (2023)
- [Effective Distillation of Table-based Reasoning Ability from LLMs](https://mlsanity.com/paper/2309.13182/) (2023)
- [Investigating Mysteries of CoT-Augmented Distillation](https://mlsanity.com/paper/2406.14511/) (2024)

## Built on by (most cited first, 6 of 6)

- [Real-Time Aligned Reward Model beyond Semantics](https://mlsanity.com/paper/2601.22664/) (2026)
- [MITS: Enhanced Tree Search Reasoning for LLMs via Pointwise Mutual Information](https://mlsanity.com/paper/2510.03632/) (2025)
- [Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models](https://mlsanity.com/paper/2607.14552/) (2026)
- [Multi-Intent Spoken Language Understanding: Methods, Trends, and Challenges](https://mlsanity.com/paper/2512.11258/) (2025)
- [A benchmark for joint dialogue satisfaction, emotion recognition, and emotion state transition prediction](https://mlsanity.com/paper/2603.03327/) (2026)
- [Dynamic Knowledge Fusion for Multi-Domain Dialogue State Tracking](https://mlsanity.com/paper/2603.10367/) (2026)

## Abstract (arXiv)

Large language models (LLMs) excel at reasoning tasks but are expensive to deploy. Thus small language models (SLMs) are fine-tuned on CoT data generated by LLMs to copy LLMs' abilities. However, these CoT data may include noisy rationales that either fail to substantiate the answers or contribute no additional information to support answer prediction, which leads SLMs to capture spurious correlations between questions and answers and compromise the quality of reasoning. In this work, we propose Chain-of-Thought Correctness Perception Distillation (CoPeD), which aims to improve the reasoning quality of the student model from the perspectives of task setting and data utilization. Firstly, we introduce a correctness-aware task setting that encourages the student model to predict answers based on correct rationales and revise them when they are incorrect. This setting improves the faithfulness of reasoning and allows the model to learn from its mistakes. Then, we propose a Correctness-Aware Weighted loss, which dynamically adjusts the contribution of each training instance based on the combined loss of the rationale and the answer. This strategy encourages the model to focus more on samples where the rationale offers stronger support for the correct answer. Experiments have shown that CoPeD is effective on both in-distribution (IND) and out-of-distribution (OOD) benchmark reasoning datasets.