# Turning Dust into Gold: Distilling Complex Reasoning Capabilities from LLMs by Leveraging Negative Data

arXiv: https://arxiv.org/abs/2312.12832 · HTML: https://mlsanity.com/paper/2312.12832/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 0 of its 41 references that are in the map.
- First posted on arXiv 20 Dec 2023.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Built on by (most cited first, 21 of 21)

- [Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning](https://mlsanity.com/paper/2401.10480/) (2024)
- [Efficient Reasoning Models: A Survey](https://mlsanity.com/paper/2504.10903/) (2025)
- [Skip-Thinking: Chunk-wise Chain-of-Thought Distillation Enable Smaller Language Models to Reason Better and Faster](https://mlsanity.com/paper/2505.18642/) (2025)
- [Distilling Reasoning Ability from Large Language Models with Adaptive Thinking](https://mlsanity.com/paper/2404.09170/) (2024)
- [Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweighting](https://mlsanity.com/paper/2510.08696/) (2025)
- [Revisiting Self-Consistency from Dynamic Distributional Alignment Perspective on Answer Aggregation](https://mlsanity.com/paper/2502.19830/) (2025)
- [LOGIN: A Large Language Model Consulted Graph Neural Network Training Framework](https://mlsanity.com/paper/2405.13902/) (2024)
- [Focused Large Language Models are Stable Many-Shot Learners](https://mlsanity.com/paper/2408.13987/) (2024)
- [ALinFiK: Learning to Approximate Linearized Future Influence Kernel for Scalable Third-Party LLM Data Valuation](https://mlsanity.com/paper/2503.01052/) (2025)
- [Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation](https://mlsanity.com/paper/2602.05548/) (2026)
- [Retrieved In-Context Principles from Previous Mistakes](https://mlsanity.com/paper/2407.05682/) (2024)
- [Speculative Decoding for Multi-Sample Inference](https://mlsanity.com/paper/2503.05330/) (2025)
- [Self-Enhanced Reasoning Training: Activating Latent Reasoning in Small Models for Enhanced Reasoning Distillation](https://mlsanity.com/paper/2502.12744/) (2025)
- [Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling](https://mlsanity.com/paper/2607.01612/) (2026)
- [Self-Correction Distillation for Structured Data Question Answering](https://mlsanity.com/paper/2511.07998/) (2025)
- [Poor-Supervised Evaluation for SuperLLM via Mutual Consistency](https://mlsanity.com/paper/2408.13738/) (2024)
- [SleepCoT: A Lightweight Personalized Sleep Health Model via Chain-of-Thought Distillation](https://mlsanity.com/paper/2410.16924/) (2024)
- [KSOD: Knowledge Supplement for LLMs On Demand](https://mlsanity.com/paper/2503.07550/) (2025)
- [Failures Are the Stepping Stones to Success: Enhancing Few-Shot In-Context Learning by Leveraging Negative Samples](https://mlsanity.com/paper/2507.23211/) (2025)
- [Investigating Model Compression for Neural Machine Translation in the Biomedical Domain](https://mlsanity.com/paper/2610.07032/) (2026)
- [Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation](https://mlsanity.com/paper/2610.17607/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have performed well on various reasoning tasks, but their inaccessibility and numerous parameters hinder wide application in practice. One promising way is distilling the reasoning ability from LLMs to small models by the generated chain-of-thought reasoning paths. In some cases, however, LLMs may produce incorrect reasoning chains, especially when facing complex mathematical problems. Previous studies only transfer knowledge from positive samples and drop the synthesized data with wrong answers. In this work, we illustrate the merit of negative data and propose a model specialization framework to distill LLMs with negative samples besides positive ones. The framework consists of three progressive steps, covering from training to inference stages, to absorb knowledge from negative data. We conduct extensive experiments across arithmetic reasoning tasks to demonstrate the role of negative data in distillation from LLM.