mlsanityData to 14 Oct 2026

Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation

arXiv 2609.13770 ↗First posted 12 Sep 2026cs.LG cs.CLcited by 0

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

also in 1 more
Abstract (arXiv)

Specialist distillation effectively transfers domain expertise to student models via teacher-generated reasoning trajectories. However, when these specialists are trained solely on question--answer pairs without explicit reasoning supervision, what governs the trajectories they generate? In this work, we show that specialist optimization implicitly selects from this latent trajectory space. To isolate and observe this latent distribution, we leverage student distillation not as a downstream goal, but as an agnostic probe---since students inherit no parameterization or optimization constraints from the specialist, inheriting only the sampled trajectories themselves. Through this probe, our empirical analysis unveils a tight governing relationship: across 27 specialist--student pairings, their specialization--generalization profiles correlate exceptionally strongly. Crucially, explicitly controlling the specialist's distributional drift systematically shifts both the teacher and its distilled student along a controllable trade-off between domain precision and general-capability retention. Across chemistry, physics, and multilingual settings, distilled students systematically reflect these specialist-induced profiles, even across divergent model families. Our findings establish a new view of specialist training: when gold reasoning is absent, tuning choices directly control the latent supervision passed to downstream models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 46 of its 59 references that are in the map. First posted on arXiv 12 Sep 2026.

Builds on

46 of 59 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting7 May
  2. 2022Large Language Models Are Reasoning Teachers20 Dec
  3. 2022Teaching Small Language Models to Reason16 Dec
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them17 Oct

Other references · 42

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2023Let's Verify Step by Stepcited by 3,789
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  9. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
30 more
  1. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  2. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  3. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  4. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  5. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  6. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  7. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  8. 2020A Brief Review of Domain Adaptationcited by 288
  9. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 251
  10. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  11. 2026Qwen3.5-Omni Technical Reportcited by 174
  12. 2022Faithful Reasoning Using Large Language Modelscited by 130
  13. 2024LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Datasetcited by 128
  14. 2025BIG-Bench Extra Hardcited by 125
  15. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 90
  16. 2025MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoningcited by 59
  17. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 40
  18. 2023UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankerscited by 37
  19. 2021Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domainscited by 36
  20. 2024Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languagescited by 36
  21. 2021UNIREX: A Unified Learning Framework for Language Model Rationale Extractioncited by 32
  22. 2025Physics: Benchmarking Foundation Models on University-Level Physics Problem Solvingcited by 31
  23. 2022Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translationcited by 21
  24. 2025Please Translate Again: Two Simple Experiments on Whether Human-Like Reasoning Helps Translationcited by 7
  25. 2025From Reasoning LLMs to BERT: A Two-Stage Distillation Framework for Search Relevancecited by 7
  26. 2025Knowledge Hierarchy Guided Biological-Medical Dataset Distillation for Domain LLM Trainingcited by 5
  27. 2025Test-Time Scaling of Reasoning Models for Machine Translationcited by 5
  28. 2025Blessing of Multilinguality: A Systematic Analysis of Multilingual In-Context Learningcited by 3
  29. 2024METEOR: Evolutionary Journey of Large Language Models from Guidance to Self-Growthcited by 1
  30. 2025LLaMAX2: Your Translation-Enhanced Model also Performs Well in Reasoningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Self-Specialized Teachers for Domain Post-Training
  2. 2026Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models
  3. 2024Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Experts
  4. 2025UNDO: Understanding Distillation as Optimization
  5. 2025On the Generalization vs Fidelity Paradox in Knowledge Distillation
  6. 2026ChemOPD: Multi-Teacher On-Policy Distillation for Multi-Task Chemical Reasoning
  7. 2025Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMs
  8. 2026Task-Specific Knowledge Distillation via Intermediate Probes