# Towards Efficient CoT Distillation: Self-Guided Rationale Selector for Better Performance with Fewer Rationales

arXiv: https://arxiv.org/abs/2509.23574 · HTML: https://mlsanity.com/paper/2509.23574/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 28 Sep 2025.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 28 of 28)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [Solving General Arithmetic Word Problems](https://mlsanity.com/paper/1608.01413/) (2016)
- [Large Language Models Are Reasoning Teachers](https://mlsanity.com/paper/2212.10071/) (2022)
- [ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://mlsanity.com/paper/2107.02137/) (2021)
- [Specializing Smaller Language Models towards Multi-Step Reasoning](https://mlsanity.com/paper/2301.12726/) (2023)
- [Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step](https://mlsanity.com/paper/2306.14050/) (2023)
- [SCOTT: Self-Consistent Chain-of-Thought Distillation](https://mlsanity.com/paper/2305.01879/) (2023)
- [Distillation Scaling Laws](https://mlsanity.com/paper/2502.08606/) (2025)
- [PINTO: Faithful Language Reasoning Using Prompt-Generated Rationales](https://mlsanity.com/paper/2211.01562/) (2022)
- [Interactive AI with Retrieval-Augmented Generation for Next Generation Networking](https://mlsanity.com/paper/2401.11391/) (2024)
- [Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Models](https://mlsanity.com/paper/2402.10430/) (2024)
- [MCC-KD: Multi-CoT Consistent Knowledge Distillation](https://mlsanity.com/paper/2310.14747/) (2023)
- [Beyond Imitation: Learning Key Reasoning Steps from Dual Chain-of-Thoughts in Reasoning Distillation](https://mlsanity.com/paper/2405.19737/) (2024)
- [Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning](https://mlsanity.com/paper/2405.13448/) (2024)
- [Mentor-KD: Making Small Language Models Better Multi-step Reasoners](https://mlsanity.com/paper/2410.09037/) (2024)
- [ELAD: Explanation-Guided Large Language Models Active Distillation](https://mlsanity.com/paper/2402.13098/) (2024)
- [LLMs-as-Instructors: Learning from Errors Toward Automating Model Improvement](https://mlsanity.com/paper/2407.00497/) (2024)

## Built on by (most cited first, 8 of 8)

- [Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning](https://mlsanity.com/paper/2601.09088/) (2026)
- [Student-in-the-Loop Chain-of-Thought Distillation via Generation-Time Selection](https://mlsanity.com/paper/2604.02819/) (2026)
- [From Fact Overwriting to Knowledge Evolution: Causal Editing via On-Policy Self-Distillation](https://mlsanity.com/paper/2605.28303/) (2026)
- [Structural Rationale Distillation via Reasoning Space Compression](https://mlsanity.com/paper/2605.07139/) (2026)
- [Characterize Then Distill: Mechanistic Reasoning in Large Output Spaces](https://mlsanity.com/paper/2606.06840/) (2026)
- [Decision Shifts, Lost Label Functionality, and an Inconclusive Grounding Audit in Correctness-Gated Multi-Teacher Distillation](https://mlsanity.com/paper/2609.09702/) (2026)
- [Overtaught, Underlearned: Learner-Centered Distillation from Multi-Agent Discussion](https://mlsanity.com/paper/2610.16543/) (2026)
- [Audit Before You Distill: A Controlled Workflow for Reasoning Transfer into Compact Medical Language Models](https://mlsanity.com/paper/2610.16713/) (2026)

## Abstract (arXiv)

Chain-of-thought (CoT) distillation aims to enhance small language models' (SLMs) reasoning by transferring multi-step reasoning capability from the larger teacher models. However, existing work underestimates rationale quality, focusing primarily on data quantity, which may transfer noisy or incorrect information to the student model. To address the above issues, we proposed Model-Oriented Rationale Selection Distillation (MoRSD), which can discern and select high quality rationales for distillation to improve performance further. We further propose a Rationale Difficulty (RD) metric to measure the ability of the student model to generate the correct answer under a given rationale. Compared to the baseline, we achieved 4.6% average improvement on seven datasets over three tasks, using fewer rationales by controlling their accuracy, diversity, and difficulty. Our results reveal that a small portion of the high quality rationales can enhance the reasoning ability of student models than the entire dataset. Our method promises to be a possible solution for efficient CoT distillation. Our code will be released in https://github.com/Leon221220/MoRSD.