# Improve Student's Reasoning Generalizability through Cascading Decomposed CoTs Distillation

arXiv: https://arxiv.org/abs/2405.19842 · HTML: https://mlsanity.com/paper/2405.19842/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 3 of its 44 references that are in the map.
- First posted on arXiv 30 May 2024.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)
- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 3 of 3)

- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [Thinking Fast and Slow in Large Language Models](https://mlsanity.com/paper/2212.05206/) (2022)

## Built on by (most cited first, 9 of 9)

- [Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retention](https://mlsanity.com/paper/2505.15774/) (2025)
- [Mitigating Spurious Correlations Between Question and Answer via Chain-of-Thought Correctness Perception Distillation](https://mlsanity.com/paper/2509.05602/) (2025)
- [Neural-Symbolic Collaborative Distillation: Advancing Small Language Models for Complex Reasoning Tasks](https://mlsanity.com/paper/2409.13203/) (2024)
- [Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillation](https://mlsanity.com/paper/2602.12172/) (2026)
- [Honey, I Shrunk the Language Model: Impact of Knowledge Distillation Methods on Performance and Explainability](https://mlsanity.com/paper/2504.16056/) (2025)
- [MIND: From Passive Mimicry to Active Reasoning through Capability-Aware Multi-Perspective CoT Distillation](https://mlsanity.com/paper/2601.03717/) (2026)
- [ReaLM: Reflection-Enhanced Autonomous Reasoning with Small Language Models](https://mlsanity.com/paper/2508.12387/) (2025)
- [LiteStage: Latency-aware Layer Skipping for Multi-stage Reasoning](https://mlsanity.com/paper/2510.14211/) (2025)
- [GateKD: Confidence-Gated Closed-Loop Distillation for Robust Reasoning](https://mlsanity.com/paper/2605.13136/) (2026)

## Abstract (arXiv)

Large language models (LLMs) exhibit enhanced reasoning at larger scales, driving efforts to distill these capabilities into smaller models via teacher-student learning. Previous works simply fine-tune student models on teachers' generated Chain-of-Thoughts (CoTs) data. Although these methods enhance in-domain (IND) reasoning performance, they struggle to generalize to out-of-domain (OOD) tasks. We believe that the widespread spurious correlations between questions and answers may lead the model to preset a specific answer which restricts the diversity and generalizability of its reasoning process. In this paper, we propose Cascading Decomposed CoTs Distillation (CasCoD) to address these issues by decomposing the traditional single-step learning process into two cascaded learning steps. Specifically, by restructuring the training objectives -- removing the answer from outputs and concatenating the question with the rationale as input -- CasCoD's two-step learning process ensures that students focus on learning rationales without interference from the preset answers, thus improving reasoning generalizability. Extensive experiments demonstrate the effectiveness of CasCoD on both IND and OOD benchmark reasoning datasets. Code can be found at https://github.com/C-W-D/CasCoD.