mlsanityData to 14 Oct 2026

Audit Before You Distill: A Controlled Workflow for Reasoning Transfer into Compact Medical Language Models

arXiv 2610.16713 ↗First posted 12 Oct 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Teacher rationales can help a smaller model answer health professions questions, but they can also reinforce errors. We examined when they help and how this evidence can guide training using Brazilian Portuguese questions from medical and multiprofessional residency examinations. First, we measured how a teacher rationale changed the student's probability of selecting the correct answer and compared this effect with rationales drawn from other questions. The complete rationale increased this probability by 21~p.p. when the teacher was correct and decreased it by 14~p.p. when the teacher was wrong. By intervening on the reasoning content, we showed that shuffled rationales did not produce the same pattern. Guided by this result, we trained only with rationales attached to correct teacher answers and compared answer-only training, step-by-step distillation, and our proposed Distill-SFT method. Distill-SFT showed a modest advantage over step-by-step distillation on previously missed questions under reasoning, while using fewer training instances by combining the answer and rationale into a single target. We also investigated catastrophic forgetting using the out-of-domain ENEM dataset. Answer-only training reduced reasoning accuracy by 12~p.p. relative to baseline, compared with approximately 4~p.p. for either rationale-based training method. Together, these findings support a workflow that audits rationale content before distillation, verifies teacher answers before supervision, and trains compact models efficiently for health professions question answering.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 28 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

20 of 28 references.

References · 20

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  3. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  4. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  5. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  6. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  7. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  8. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 225
  9. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 158
  10. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 148
  11. 2023Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Taskscited by 64
  12. 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillationcited by 47
8 more
  1. 2024LLMs May Perform MCQA by Selecting the Least Incorrect Optioncited by 33
  2. 2025NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Taskscited by 32
  3. 2024xFinder: Large Language Models as Automated Evaluators for Reliable Evaluationcited by 19
  4. 2026Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignmentcited by 13
  5. 2024Mitigating Misleading Chain-of-Thought Reasoning with Selective Filteringcited by 10
  6. 2025Towards Efficient CoT Distillation: Self-Guided Rationale Selector for Better Performance with Fewer Rationalescited by 8
  7. 2024Boosting LLM Translation Skills without General Ability Loss via Rationale Distillationcited by 6
  8. 2025Does Rationale Quality Matter? Enhancing Mental Disorder Detection via Selective Reasoning Distillationcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Better Accuracies, Worse Reasoning: A Step-Level Audit of Medical Chain-of-Thought Distillation
  2. 2026MedThink: Enhancing Diagnostic Accuracy in Small Models via Teacher-Guided Reasoning Correction
  3. 2024Investigating Mysteries of CoT-Augmented Distillation
  4. 2026Can Large Language Models Self-Correct in Medical Question Answering? An Exploratory Study
  5. 2026Medical Reasoning with Large Language Models: A Survey and MR-Bench
  6. 2025UNDO: Understanding Distillation as Optimization
  7. 2025Honey, I Shrunk the Language Model: Impact of Knowledge Distillation Methods on Performance and Explainability
  8. 2025Not All Correct Answers Are Equal: Why Your Distillation Source Matters