mlsanityData to 14 Oct 2026

Trace-of-Thought Prompting: Investigating Prompt-Based Knowledge Distillation Through Question Decomposition

arXiv 2504.20946 ↗First posted 29 Apr 2025cs.CL cs.AIcited by 3

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

also in 1 more
Abstract (arXiv)

Knowledge distillation allows smaller neural networks to emulate the performance of larger, teacher models with reduced computational demands. Traditional methods for Large Language Models (LLMs) often necessitate extensive fine-tuning, which limits their accessibility. To address this, we introduce Trace-of-Thought Prompting, a novel framework designed to distill critical reasoning capabilities from high-resource teacher models (over 8 billion parameters) to low-resource student models (up to 8 billion parameters). This approach leverages problem decomposition to enhance interpretability and facilitate human-in-the-loop interventions. Empirical evaluations on the GSM8K and MATH datasets show that student models achieve accuracy gains of up to 113% on GSM8K and 21% on MATH, with significant improvements particularly notable in smaller models like Llama 2 and Zephyr. Our results suggest a promising pathway for open-source, low-resource models to eventually serve both as both students and teachers, potentially reducing our reliance on high-resource, proprietary models.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 42 of its 43 references that are in the map. First posted on arXiv 29 Apr 2025.

Builds on

42 of 43 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Mitigating Misleading Chain-of-Thought Reasoning with Selective Filtering28 Mar
  2. 2023The Art of SOCRATIC QUESTIONING: Recursive Thinking with Large Language Models24 May
  3. 2023PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents23 May
  4. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models6 May
  5. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Models19 Apr
  6. 2022Distilling Reasoning Capabilities into Smaller Language Models1 Dec
  7. 2022Large Language Models are Zero-Shot Reasoners24 May
  8. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  10. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  6. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  7. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  8. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  9. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  10. 2023Textbooks Are All You Needcited by 569
  11. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  12. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
20 more
  1. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  2. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  3. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 431
  4. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  6. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 139
  7. 2022Successive Prompting for Decomposing Complex Questionscited by 94
  8. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoningcited by 86
  9. 2024Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solvingcited by 84
  10. 2024Embodied LLM Agents Learn to Cooperate in Organized Teamscited by 82
  11. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  12. 2022PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptationcited by 31
  13. 2024CoMM: Collaborative Multi-Agent, Multi-Reasoning-Path Prompting for Complex Problem Solvingcited by 31
  14. 2024Adapting LLMs for Efficient Context Processing through Soft Prompt Compressioncited by 22
  15. 2023Social Learning: Towards Collaborative Learning with Large Language Modelscited by 15
  16. 2024SocraSynth: Multi-LLM Reasoning with Conditional Statisticscited by 13
  17. 2024Evolving Knowledge Distillation with Large Language Models and Active Learningcited by 12
  18. 2024Can LLMs perform structured graph reasoning?cited by 9
  19. 2024Socratic Reasoning Improves Positive Text Rewritingcited by 9
  20. 2024textttLM^texttt2: A Simple Society of Language Models Solves Complex Reasoningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Interpretable Traces, Unexpected Outcomes: Investigating the Disconnect in Trace-Based Knowledge Distillationcited by 8
  2. 2026LLM-Guided Semantic Bootstrapping for Interpretable Text Classification with Tsetlin Machinescited by 0
  3. 2026MedThink: Enhancing Diagnostic Accuracy in Small Models via Teacher-Guided Reasoning Correctioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Teaching Small Language Models to Reason
  2. 2024Enhancing Knowledge Distillation for LLMs with Response-Priming Prompting
  3. 2022Large Language Models Are Reasoning Teachers
  4. 2025Concise Reasoning, Big Gains: Pruning Long Reasoning Trace with Difficulty-Aware Prompting
  5. 2025Honey, I Shrunk the Language Model: Impact of Knowledge Distillation Methods on Performance and Explainability
  6. 2026Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning
  7. 2026Task-Specific Knowledge Distillation via Intermediate Probes
  8. 2026Hidden Thoughts Are Not Secret: Reasoning Trace Exposure in LLMs