K2P: Label-Free Knowledge to Prompt Distillation
On the map: Knowledge distillation → Knowledge distillation for LLMs
Abstract (arXiv)
Knowledge distillation can transfer reasoning from stronger teachers to frozen students through reusable prompts, but avoiding weight updates does not eliminate supervision. Without ground-truth answers, teacher solutions are unverified, and agreement with the teacher can reward shared mistakes. We introduce Knowledge-to-Prompt (K2P) for label-free knowledge distillation to prompts. K2P synthesizes reusable instructions from teacher solutions, refines them using paired teacher and student responses, and guides search and selection with answer agreement. It retains candidates that adaptive search may undervalue and selects on reserved questions. Deployment uses only the frozen student and selected prompt. Our theory separates generation and selection gaps and gives conditions under which agreement-guided construction yields accuracy guarantees despite imperfect teacher references. Across reasoning tasks and students, K2P outperforms label-free alternatives overall and remains competitive with supervised prompt optimization. Ablations and archive diagnostics assess the contributions of teacher solutions and refinement, while revealing the limits of agreement-guided selection.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 10 of its 51 references that are in the map. First posted on arXiv 30 Sep 2026.
Builds on
10 of 51 references.
References · 10
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
- 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
- 2023Large Language Models as Optimizerscited by 864
- 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
- 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 542
- 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
- 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 284
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation
- 2025Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation
- 2025Concise Reasoning, Big Gains: Pruning Long Reasoning Trace with Difficulty-Aware Prompting
- 2024Enhancing Knowledge Distillation for LLMs with Response-Priming Prompting
- 2025Trace-of-Thought Prompting: Investigating Prompt-Based Knowledge Distillation Through Question Decomposition
- 2026Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning
- 2026Task-Specific Knowledge Distillation via Intermediate Probes
- 2025Dual-Forward Path Teacher Knowledge Distillation: Bridging the Capacity Gap Between Teacher and Student