mlsanityData to 7 Oct 2026

Embarrassingly Simple Self-Distillation Improves Code Generation

arXiv 2604.01193 ↗First posted 1 Apr 2026cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

Can a large language model (LLM) improve at code generation using only its own raw outputs, without a verifier, a teacher model, or reinforcement learning? We answer in the affirmative with simple self-distillation (SSD): sample solutions from the model with certain temperature and truncation configurations, then fine-tune on those samples with standard supervised fine-tuning. SSD improves Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6, with gains concentrating on harder problems, and it generalizes across Qwen and Llama models at 4B, 8B, and 30B scale, including both instruct and thinking variants. To understand why such a simple method can work, we trace these gains to a precision-exploration conflict in LLM decoding and show that SSD reshapes token distributions in a context-dependent way, suppressing distractor tails where precision matters while preserving useful diversity where exploration matters. Taken together, SSD offers a complementary post-training direction for improving LLM code generation. Our code is available at https://github.com/apple/ml-ssd

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 54 of its 57 references that are in the map. First posted on arXiv 1 Apr 2026.

Builds on

54 of 57 references.

References · 54

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
38 more
  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  2. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  3. 2018Hierarchical Neural Story Generationcited by 1,182
  4. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  5. 2021Measuring Coding Challenge Competence With APPScited by 879
  6. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  7. 2016Sequence-Level Knowledge Distillationcited by 761
  8. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  9. 2022Large Language Models Can Self-Improvecited by 656
  10. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  11. 2024Self-Rewarding Language Modelscited by 654
  12. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  13. 2018Born Again Neural Networkscited by 539
  14. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  15. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  16. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  17. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  18. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  19. 2026Reinforcement Learning via Self-Distillationcited by 285
  20. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
  21. 2026Self-Distillation Enables Continual Learningcited by 212
  22. 2025Learning to Reason without External Rewardscited by 199
  23. 2026On-Policy Context Distillation for Language Modelscited by 161
  24. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  25. 2019Revisiting Self-Training for Neural Sequence Generationcited by 154
  26. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
  27. 2022Truncation Sampling as Language Model Desmoothingcited by 121
  28. 2025Competitive Programming with Large Reasoning Modelscited by 115
  29. 2025Maximizing Confidence Alone Improves Reasoningcited by 97
  30. 2022Self-Training: A Surveycited by 85
  31. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
  32. 2026Privileged Information Distillation for Language Modelscited by 79
  33. 2025rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Datasetcited by 49
  34. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  35. 2025Emergent Hierarchical Reasoning in LLMs through Reinforcement Learningcited by 42
  36. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoningcited by 35
  37. 2025Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuningcited by 33
  38. 2024Forking Paths in Neural Text Generationcited by 30

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 36.

36 papers.

Sort:

By area:

  1. 2026Self-Distilled Agentic Reinforcement Learningcited by 49
  2. 2026Rethinking On-Policy Self-Distillation for Thinking Modelscited by 28
  3. 2026Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loopscited by 25
  4. 2026Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoningcited by 20
  5. 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularizationcited by 10
  6. 2026Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvementcited by 8
  7. 2026Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoningcited by 7
  8. 2026UniSD: Towards a Unified Self-Distillation Framework for Large Language Modelscited by 6
  9. 2026Self-Improving 4D Perception via Self-Distillationcited by 3
  10. 2026Self-Policy Distillation via Capability-Selective Subspace Projectioncited by 3
  11. 2026BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolutioncited by 3
  12. 2026Pythagoras-Prover: Advancing Efficient Formal Proving via Augmented Lean Formalisationcited by 3
24 more
  1. 2026Procedural Memory Distillation: Online Reflection for Self-Improving Language Modelscited by 3
  2. 2026The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillationcited by 3
  3. 2026Agent trajectories as programs: fingerprinting and programming coding-agent behaviorcited by 2
  4. 2026The Verifier is the Curriculum: Precision Sets the Return on Search in Code Self-Distillationcited by 2
  5. 2026LLM-as-a-Coach: Experiential Learning for Non-Verifiable Taskscited by 2
  6. 2026AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignmentcited by 1
  7. 2026Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipelinecited by 1
  8. 2026Retrieval, Reward, and Training Protocols: What Matters in Training Search Agents?cited by 1
  9. 2026Language Models Need Sleep: Learning to Self-Modify and Consolidate Memoriescited by 1
  10. 2026Failed Reasoning Traces Tell You What Is Fixable (But Not by Reading Them)cited by 1
  11. 2026Dynamo: Dynamic Skill-Tool Evolution for Vision-Language Agentscited by 1
  12. 2026Nautile-370M: Spectral Memory Meets Attention in a Small Reasoning Modelcited by 0
  13. 2026Iterative Finetuning is Mostly Idempotentcited by 0
  14. 2026Semantic Flow Regularization: Teaching LLMs to Generate Diverse Yet Coherent Responsescited by 0
  15. 2026Not All Synthetic Data Is Yours to Learn Fromcited by 0
  16. 2026ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalizationcited by 0
  17. 2026Self-Improving is Often Sudden: Enlightenment-style Finetuning for Large-Scale Modelscited by 0
  18. 2026Training Specialist Models without Reasoning Trajectories for Domain Expert Distillationcited by 0
  19. 2026Lightning Weave: Improving the Accuracy-Efficiency Frontier of Reasoning Models through Capability Compositioncited by 0
  20. 2026PSD: Pseudo Self-Distillation of Memory Representation Capabilities for LLM Agentscited by 0
  21. 2026Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Modelscited by 0
  22. 2026Continuous Context Managementcited by 0
  23. 2026Shockingly Simple Self-retrospection Improves Agentic Models Without RLcited by 0
  24. 2026SAPD: Step-Aligned Privileged Distillationcited by 0