# Keypoint-based Progressive Chain-of-Thought Distillation for LLMs

arXiv: https://arxiv.org/abs/2405.16064 · HTML: https://mlsanity.com/paper/2405.16064/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 17 of its 48 references that are in the map.
- First posted on arXiv 25 May 2024.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 17 of 17)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers](https://mlsanity.com/paper/2106.15772/) (2021)
- [A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models](https://mlsanity.com/paper/2303.10420/) (2023)
- [Active Prompting with Chain-of-Thought for Large Language Models](https://mlsanity.com/paper/2302.12246/) (2023)
- [On the Road to Portability: Compressing End-to-End Motion Planner for Autonomous Driving](https://mlsanity.com/paper/2403.01238/) (2024)

## Built on by (most cited first, 23 of 23)

- [Video-R1: Reinforcing Video Reasoning in MLLMs](https://mlsanity.com/paper/2503.21776/) (2025)
- [Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?](https://mlsanity.com/paper/2410.01623/) (2024)
- [AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes](https://mlsanity.com/paper/2506.14728/) (2025)
- [DrugPilot: LLM-based Parameterized Reasoning Agent for Drug Discovery](https://mlsanity.com/paper/2505.13940/) (2025)
- [Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation](https://mlsanity.com/paper/2603.13260/) (2026)
- [Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges](https://mlsanity.com/paper/2507.19672/) (2025)
- [Rethinking Selective Knowledge Distillation](https://mlsanity.com/paper/2602.01395/) (2026)
- [Incentivizing Strong Reasoning from Weak Supervision](https://mlsanity.com/paper/2505.20072/) (2025)
- [Finding the Evidence: Discovering Decision-Supporting Tokens for On-Policy Reasoning Distillation](https://mlsanity.com/paper/2606.22830/) (2026)
- [Merge-of-Thought Distillation](https://mlsanity.com/paper/2509.08814/) (2025)
- [Protecting Language Models Against Unauthorized Distillation through Trace Rewriting](https://mlsanity.com/paper/2602.15143/) (2026)
- [Self-Filtered Distillation with LLMs-generated Trust Indicators for Reliable Patent Classification](https://mlsanity.com/paper/2510.05431/) (2025)
- [Reasoning-Trace Collapse: Evaluating the Loss of Explicit Reasoning During Fine-Tuning](https://mlsanity.com/paper/2605.21127/) (2026)
- [1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation](https://mlsanity.com/paper/2609.24432/) (2026)
- [HybridServe: Efficient Serving of Large AI Models with Confidence-Based Cascade Routing](https://mlsanity.com/paper/2505.12566/) (2025)
- [RF-Agent: A Practical Framework for Building Language Agents for RFIC Design](https://mlsanity.com/paper/2607.18772/) (2026)
- [Audited Reasoning Refinement: Fine-Tuning Language Models via LLM-Guided Step-Wise Evaluation and Correction](https://mlsanity.com/paper/2509.12476/) (2025)
- [Characterize Then Distill: Mechanistic Reasoning in Large Output Spaces](https://mlsanity.com/paper/2606.06840/) (2026)
- [TELLER: Dual-Path Iterative Preference Optimization for Table Entity Linking](https://mlsanity.com/paper/2607.28680/) (2026)
- [Trajectory Learnability for Offline On-Policy Distillation with Imperfect Teachers](https://mlsanity.com/paper/2609.18321/) (2026)
- [A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions](https://mlsanity.com/paper/2609.33727/) (2026)
- [Teacher-Student Gaps Are Not Enough: Outcome-Guided On-Policy Distillation for Multi-Turn Autonomous Agents](https://mlsanity.com/paper/2609.35319/) (2026)
- [Progress-Aware Distillation for Mitigating Stagnation in Small Language Model Agents](https://mlsanity.com/paper/2610.19439/) (2026)

## Abstract (arXiv)

Chain-of-thought distillation is a powerful technique for transferring reasoning abilities from large language models (LLMs) to smaller student models. Previous methods typically require the student to mimic the step-by-step rationale produced by LLMs, often facing the following challenges: (i) Tokens within a rationale vary in significance, and treating them equally may fail to accurately mimic keypoint tokens, leading to reasoning errors. (ii) They usually distill knowledge by consistently predicting all the steps in a rationale, which falls short in distinguishing the learning order of step generation. This diverges from the human cognitive progression of starting with easy tasks and advancing to harder ones, resulting in sub-optimal outcomes. To this end, we propose a unified framework, called KPOD, to address these issues. Specifically, we propose a token weighting module utilizing mask learning to encourage accurate mimicry of keypoint tokens by the student during distillation. Besides, we develop an in-rationale progressive distillation strategy, starting with training the student to generate the final reasoning steps and gradually extending to cover the entire rationale. To accomplish this, a weighted token generation loss is proposed to assess step reasoning difficulty, and a value function is devised to schedule the progressive distillation by considering both step difficulty and question diversity. Extensive experiments on four reasoning benchmarks illustrate our KPOD outperforms previous methods by a large margin.