mlsanityData to 14 Oct 2026

The Quality-Utility Paradox: Why High-Reward Data Impairs Small Model Mathematical Reasoning

arXiv 2606.16152 ↗First posted 15 Jun 2026cs.AIcited by 0

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

also in 2 more
Abstract (arXiv)

Knowledge distillation from powerful reasoning models is widely used to improve Small Language Models (SLMs) on mathematical reasoning, often assuming that traces with higher reward model scores provide more useful supervision. We identify a counterintuitive Quality-Utility Paradox in mathematical reasoning distillation. Data refined or synthesized by a stronger Oracle obtains higher perceived quality according to reward models, yet consistently underperforms traces generated by the SLM itself and selected through rejection sampling across Qwen2.5, LLaMA-3, and DeepSeek families. Our analysis shows that Oracle refinement couples logical repair with distributional drift away from the SLM's native reasoning distribution. This drift increases the learner's adaptation cost and can outweigh the benefit of improved reasoning logic. To test this mechanism, we introduce Style-Aligned Refinement, which preserves the native trajectory of the SLM while retaining logical repair from the Oracle. This intervention lowers adaptation cost and restores downstream utility. These findings suggest that effective mathematical reasoning distillation should jointly optimize perceived solution quality and learner-data compatibility, rather than relying solely on reward-model scores. The datasets and code are available at https://github.com/Dracoqhl/Quality-Utility-Paradox.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 32 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

20 of 32 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Small Models Struggle to Learn from Strong Reasoners17 Feb

Other references · 19

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2023Let's Verify Step by Stepcited by 3,789
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  11. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  12. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
7 more
  1. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
  2. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 75
  3. 2025Black-Box On-Policy Distillation of Large Language Modelscited by 59
  4. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Modelscited by 48
  5. 2025Early science acceleration experiments with GPT-5cited by 41
  6. 2026Learning from Mistakes: Negative Reasoning Samples Enhance Out-of-Domain Generalizationcited by 5
  7. 2025Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanationscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
  2. 2025UNDO: Understanding Distillation as Optimization
  3. 2025Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning
  4. 2024Improving Mathematical Reasoning Capabilities of Small Language Models via Feedback-Driven Distillation
  5. 2025Enhancing Reasoning Capabilities in SLMs with Reward Guided Dataset Distillation
  6. 2025Can Large Reasoning Models Improve Accuracy on Mathematical Tasks Using Flawed Thinking?
  7. 2025The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason
  8. 2025Catch Me If You Can: How Smaller Reasoning Models Pretend to Reason with Mathematical Fidelity