mlsanityData to 14 Oct 2026

Decision Shifts, Lost Label Functionality, and an Inconclusive Grounding Audit in Correctness-Gated Multi-Teacher Distillation

arXiv 2609.09702 ↗First posted 9 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Candidate decision correctness and rationale grounding are different objectives. We examine correctness-gated multi-teacher distillation in a fixed experiment. Eight arms share 4,330 sources, a 63.9M-parameter student, 12,990 optimization rows, 406 updates, evidence inputs, and a decoder; seven teacher-based arms use one fixed three-response pool. Three seeds are evaluated on 267 held-out examples. Relative to unfiltered distillation, the correctness-weighted arm differed in accuracy by +0.1660 (95% observed-matrix interval [0.0670, 0.2455]), five-label macro-F1 by +0.1323 ([0.0916, 0.1731]), and task-defined conditional unsafe-action rate by -0.4979 ([-0.5926, -0.3686]). These shifts do not imply uniformly better behavior. Source-label SFT had the highest mean macro-F1 (0.586). The weighted arm had zero Refuted recall in every seed, and two seeds assigned NotEnoughInfo to all 167 claim examples. In an availability-amended audit at one reference seed, weighted and unfiltered outputs had 0/20 versus 1/20 evidence-supported positives and 20/20 versus 19/20 positives containing unsupported material. Samples were non-paired, source overlap was not serialized, and the amendment followed automatic summarization but preceded annotation. The audit therefore cannot estimate a common-source grounding effect and is inconclusive about system-level improvement or harm. Hard filtering already achieved 0.660 accuracy, 0.530 macro-F1, and 0.135 conditional unsafe rate. The implemented weighted arm showed no demonstrated incremental decision benefit over hard filtering. This fixed-matrix failure analysis shows decision redistribution with lost label functionality; the available human audit does not establish a grounding gain.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 9 Sep 2026.

Builds on

26 of 33 references.

References · 26

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2017Selective Classification for Deep Neural Networkscited by 852
  4. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  5. 2020Selective Question Answering under Domain Shiftcited by 238
  6. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
  7. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 158
  8. 2023SCOTT: Self-Consistent Chain-of-Thought Distillationcited by 105
  9. 2020Reinforced Multi-Teacher Selection for Knowledge Distillationcited by 63
  10. 2024Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Modelscited by 57
  11. 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillationcited by 47
  12. 2024LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedbackcited by 41
14 more
  1. 2021One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teacherscited by 40
  2. 2020Selective Classification via One-Sided Predictioncited by 38
  3. 2024RDRec: Rationale Distillation for LLM-based Recommendationcited by 22
  4. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  5. 2024Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Reviewcited by 14
  6. 2024Mentor-KD: Making Small Language Models Better Multi-step Reasonerscited by 13
  7. 2024ELAD: Explanation-Guided Large Language Models Active Distillationcited by 12
  8. 2022Fact Checking with Insufficient Evidencecited by 11
  9. 2024QCRD: Quality-guided Contrastive Rationale Distillation for Large Language Modelscited by 8
  10. 2024Rationale-Aware Answer Verification by Pairwise Self-Evaluationcited by 8
  11. 2025Towards Efficient CoT Distillation: Self-Guided Rationale Selector for Better Performance with Fewer Rationalescited by 8
  12. 2026Project Aletheia: Verifier-Guided Distillation of Backtracking for Small Language Modelscited by 8
  13. 2025Does Rationale Quality Matter? Enhancing Mental Disorder Detection via Selective Reasoning Distillationcited by 6
  14. 2023Selective Knowledge Distillation for Non-Autoregressive Neural Machine Translationcited by 2

Built on by

Not cited yet.