mlsanityData to 14 Oct 2026

Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichment

arXiv 2607.08268 ↗First posted 9 Jul 2026cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

High-volume structured extraction pays a large model's latency on every item, so distilling the task into a small on-device model is attractive: comparable output at a fraction of the time and cost. We measure what that distillation actually delivers, per sub-task. Each news article is mapped to one JSON object with a short summary and five categorical labels. We distill an 8B reasoning teacher (deepseek-r1:8b) into a 0.6B student (Qwen3-0.6B; QLoRA, three seeds), and add two teacher controls: a same-size non-reasoning teacher and a larger managed pipeline. A blinded, reference-free, three-judge panel scores every arm against the full article, alongside two non-distillation baselines, few-shot prompting and constrained decoding. The student runs at about 0.8 s per article against the teacher's 39 s, and recovers 58% of the base-to-teacher gap on summary quality, beating its primary baseline (constrained decoding) by +16.8 points and few-shot prompting by a secondary +4.9. A same-size non-reasoning teacher trains a student no better than the untuned base, so the summary gain follows from the teacher's reasoning nature rather than its scale. Capabilities then split by teacher: the reasoning teacher transfers writing quality and the managed pipeline transfers label diversity, while a same-size instruction teacher's students stay more grounded on the 22 short, thin-source articles in the 93-item test set (74 versus 55 faithful), where the reasoning-lineage student fabricates. That grounding difference is a consistent ordering rather than a significant aggregate effect, and the subgroup is small, so we report it as a direction. Because no single engine wins every field, the deliverable is a per-field routing map for on-device enrichment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 40 of its 44 references that are in the map. First posted on arXiv 9 Jul 2026.

Builds on

40 of 44 references.

References · 40

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  5. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  7. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  8. 2016Sequence-Level Knowledge Distillationcited by 801
  9. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  10. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  11. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  12. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
28 more
  1. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  2. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  3. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  4. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  5. 2023A Survey on Model Compression for Large Language Modelscited by 269
  6. 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
  7. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 257
  8. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  9. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 135
  10. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
  11. 2023UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognitioncited by 72
  12. 2025JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Modelscited by 71
  13. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 70
  14. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 68
  15. 2025Distillation Scaling Lawscited by 67
  16. 2025Are Reasoning Models More Prone to Hallucination?cited by 64
  17. 2025Unveiling the Key Factors for Distilling Chain-of-Thought Reasoningcited by 40
  18. 2025OptimalThinkingBench: Evaluating Over and Underthinking in LLMscited by 35
  19. 2025NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Taskscited by 32
  20. 2025The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learningcited by 30
  21. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 30
  22. 2025Overconfidence in LLM-as-a-Judge: Diagnosis and Confidence-Driven Solutioncited by 18
  23. 2023Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarizationcited by 16
  24. 2024Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Datacited by 12
  25. 2023Distilling Large Language Models for Biomedical Knowledge Extraction: A Case Study on Adverse Drug Eventscited by 11
  26. 2024Evaluating the Factuality of Zero-shot Summarizers Across Varied Domainscited by 10
  27. 2025On the Generalization vs Fidelity Paradox in Knowledge Distillationcited by 7
  28. 2025Revealing the Power of Post-Training for Small Language Models via Knowledge Distillationcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reliability Gated Multi-Teacher Distillation for Low Resource Abstractive Summarization
  2. 2025When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA
  3. 2025Who Taught You That? Tracing Teachers in Model Distillation
  4. 2026Rethinking Selective Knowledge Distillation
  5. 2026Curriculum Learning-Guided Progressive Distillation in Large Language Models
  6. 2024BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Data
  7. 2024DistilDoc: Knowledge Distillation for Visually-Rich Document Applications
  8. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling