mlsanityData to 14 Oct 2026

Dual-Head Reasoning Distillation: Improving Classifier Accuracy with Train-Time-Only Reasoning

arXiv 2509.21487 ↗First posted 25 Sep 2025cs.CL cs.AIcited by 3

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

also in 1 more
Abstract (arXiv)

Chain-of-Thought (CoT) prompting often improves classification accuracy, but it introduces a significant throughput penalty with rationale generation (Wei et al., 2022; Cheng and Van Durme, 2024). To resolve this trade-off, we introduce Dual-Head Reasoning Distillation (DHRD), a simple training method for decoder-only language models (LMs) that adds (i) a pooled classification head used during training and inference and (ii) a reasoning head supervised by teacher rationales used only in training. We train with a loss function that is a weighted sum of label cross-entropy and token-level LM loss over input-plus-rationale sequences. On seven SuperGLUE tasks, DHRD yields relative gains of 0.65-5.47% over pooled baselines, with notably larger gains on entailment/causal tasks. Since we disable the reasoning head at test time, inference throughput matches pooled classifiers and exceeds CoT decoding on the same backbones by 96-142 times in QPS.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 49 references that are in the map. First posted on arXiv 25 Sep 2025.

Builds on

19 of 49 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Expediting and Elevating Large Language Model Reasoning via Hidden Chain-of-Thought Decoding13 Sep
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 15

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  8. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  9. 2024Compressed Chain of Thought: Efficient Reasoning Through Dense Representationscited by 201
  10. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectivenesscited by 119
  11. 2024Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoningcited by 108
  12. 2021Explain and Predict, and then Predict Againcited by 29
3 more
  1. 2025Adapting Decoder-Based Language Models for Diverse Encoder Downstream Taskscited by 10
  2. 2024Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Headscited by 5
  3. 2024Dual-Head Knowledge Distillation: Enhancing Logits Utilization with an Auxiliary Headcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncationcited by 5
  2. 2026Internalized Reasoning for Long-Context Visual Document Understandingcited by 0
  3. 2026Multilingual and Cross-Lingual Citation Needed Detection on Wikipedia for Lower-Resource Languagescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillation
  2. 2022Distilling Reasoning Capabilities into Smaller Language Models
  3. 2026Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
  4. 2026MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation
  5. 2025Small Models Struggle to Learn from Strong Reasoners
  6. 2026Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation
  7. 2023Mixed Distillation Helps Smaller Language Model Better Reasoning
  8. 2025Towards Efficient CoT Distillation: Self-Guided Rationale Selector for Better Performance with Fewer Rationales