mlsanityData to 14 Oct 2026

Concept Distillation from Strong to Weak Models via Hypotheses-to-Theories Prompting

arXiv 2408.09365 ↗First posted 18 Aug 2024cs.AI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Hand-crafting high quality prompts to optimize the performance of language models is a complicated and labor-intensive process. Furthermore, when migrating to newer, smaller, or weaker models (possibly due to latency or cost gains), prompts need to be updated to re-optimize the task performance. We propose Concept Distillation (CD), an automatic prompt optimization technique for enhancing weaker models on complex tasks. CD involves: (1) collecting mistakes made by weak models with a base prompt (initialization), (2) using a strong model to generate reasons for these mistakes and create rules/concepts for weak models (induction), and (3) filtering these rules based on validation set performance and integrating them into the base prompt (deduction/verification). We evaluated CD on NL2Code and mathematical reasoning tasks, observing significant performance boosts for small and weaker language models. Notably, Mistral-7B's accuracy on Multi-Arith increased by 20%, and Phi-3-mini-3.8B's accuracy on HumanEval rose by 34%. Compared to other automated methods, CD offers an effective, cost-efficient strategy for improving weak models' performance on complex tasks and enables seamless workload migration across different language models without compromising performance.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 21 of its 40 references that are in the map. First posted on arXiv 18 Aug 2024.

Builds on

21 of 40 references.

References · 21

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  4. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  5. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  6. 2022A Survey on In-context Learningcited by 721
  7. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  8. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  9. 2016Solving General Arithmetic Word Problemscited by 360
  10. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 336
  11. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 233
  12. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 193
9 more
  1. 2021Towards Understanding Knowledge Distillationcited by 158
  2. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  3. 2023Large Language Models can Learn Rulescited by 69
  4. 2024In-Context Principle Learning from Mistakescited by 44
  5. 2024Language Models for Text Classification: Is In-Context Learning Enough?cited by 40
  6. 2023How are Prompts Different in Terms of Sensitivity?cited by 35
  7. 2022PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptationcited by 31
  8. 2023Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulationcited by 30
  9. 2023Evoke: Evoking Critical Thinking Abilities in LLMs via Reviewer-Author Prompt Editingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Agentic Multi-Source Grounding for Enhanced Query Intent Understanding: A DoorDash Case Studycited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2023Mixed Distillation Helps Smaller Language Model Better Reasoning
  2. 2025UNDO: Understanding Distillation as Optimization
  3. 2024Enhancing Knowledge Distillation for LLMs with Response-Priming Prompting
  4. 2024DistiLLM: Towards Streamlined Distillation for Large Language Models
  5. 2025PromptCoT 2.0: Scaling Prompt Synthesis for Large Language Model Reasoning
  6. 2024Leveraging Zero-Shot Prompting for Efficient Language Model Distillation
  7. 2024GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizers
  8. 2023PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning