mlsanityData to 14 Oct 2026

OpenCodeReasoning: Advancing Data Distillation for Competitive Coding

arXiv 2504.01943 ↗First posted 2 Apr 2025cs.CLcited by 113

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Since the advent of reasoning-based large language models, many have found great success from distilling reasoning capabilities into student models. Such techniques have significantly bridged the gap between reasoning and standard LLMs on coding tasks. Despite this, much of the progress on distilling reasoning models remains locked behind proprietary datasets or lacks details on data curation, filtering and subsequent training. To address this, we construct a superior supervised fine-tuning (SFT) dataset that we use to achieve state-of-the-art coding capability results in models of various sizes. Our distilled models use only SFT to achieve 61.8% on LiveCodeBench and 24.6% on CodeContests, surpassing alternatives trained with reinforcement learning. We then perform analysis on the data sources used to construct our dataset, the impact of code execution filtering, and the importance of instruction/solution diversity. We observe that execution filtering negatively affected benchmark accuracy, leading us to prioritize instruction diversity over solution correctness. Finally, we also analyze the token efficiency and reasoning patterns utilized by these models. We will open-source these datasets and distilled models to the community.

Key facts. Cited by 113 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 38 of its 54 references that are in the map. First posted on arXiv 2 Apr 2025.

Builds on

38 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Qwen2.5-Coder Technical Report18 Sep
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code12 Mar
  3. 2022Competition-Level Code Generation with AlphaCode8 Feb

Other references · 35

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  11. 2025s1: Simple test-time scalingcited by 1,341
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
23 more
  1. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  2. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  3. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  4. 2021Measuring Coding Challenge Competence With APPScited by 902
  5. 2023StarCoder: may the source be with you!cited by 799
  6. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  7. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  8. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  9. 2021Unified Pre-training for Program Understanding and Generationcited by 366
  10. 2022Teaching Small Language Models to Reasoncited by 284
  11. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  12. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
  13. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 195
  14. 2025Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMscited by 163
  15. 2023TACO: Topics in Algorithmic COde generation datasetcited by 160
  16. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
  17. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 109
  18. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 84
  19. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 62
  20. 2023WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuningcited by 42
  21. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 41
  22. 2024Code Comprehension then Auditing for Unsupervised LLM Evaluationcited by 32
  23. 2024Genetic Instruct: Scaling up Synthetic Generation of Coding Instructions for Large Language Modelscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 48; 2026: 65.

113 papers.

Sort:

By area:

  1. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 304
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025Scaling Latent Reasoning via Looped Language Modelscited by 165
  4. 2025Llama-Nemotron: Efficient Reasoning Modelscited by 158
  5. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 108
  6. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 107
  7. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 104
  8. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Modelscited by 93
  9. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learningcited by 87
  10. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergycited by 77
  11. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoringcited by 54
  12. 2025rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Datasetcited by 51
38 more
  1. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 48
  2. 2026On Data Engineering for Scaling LLM Terminal Capabilitiescited by 43
  3. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 42
  4. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
  5. 2025Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimizationcited by 39
  6. 2026OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesiscited by 37
  7. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 32
  8. 2025LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contextscited by 29
  9. 2025OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critiquecited by 25
  10. 2025CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignmentcited by 24
  11. 2025Set Block Decoding is a Language Model Inference Acceleratorcited by 23
  12. 2025CtrlDiff: Boosting Large Diffusion Language Models with Dynamic Block Prediction and Controllable Generationcited by 17
  13. 2026Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoningcited by 17
  14. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  15. 2025Don't Pass@k: A Bayesian Framework for Large Language Model Evaluationcited by 15
  16. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  17. 2026On-Policy Delta Distillationcited by 15
  18. 2025Reasoning with OmniThought: A Large CoT Dataset with Verbosity and Cognitive Difficulty Annotationscited by 14
  19. 2025Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Modelscited by 13
  20. 2025Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruningcited by 13
  21. 2025Xolver: Multi-Agent Reasoning with Holistic Experience Learning Just Like an Olympiad Teamcited by 13
  22. 2026Agentic Proposing: Enhancing Large Language Model Reasoning via Compositional Skill Synthesiscited by 12
  23. 2025TreeDiff: AST-Guided Code Generation with Diffusion LLMscited by 11
  24. 2026Recursive Multi-Agent Systemscited by 10
  25. 2025PromptCoT 2.0: Scaling Prompt Synthesis for Large Language Model Reasoningcited by 9
  26. 2025Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levelscited by 8
  27. 2026Lost in the Noise: How Reasoning Models Fail with Contextual Distractorscited by 8
  28. 2026How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Datacited by 8
  29. 2026Distilled Reinforcement Learning for LLM Post-trainingcited by 8
  30. 2025Why Do Open-Source LLMs Struggle with Data Analysis? A Systematic Empirical Studycited by 7
  31. 2025Unearthing Gems from Stones: Policy Optimization with Negative Sample Augmentation for LLM Reasoningcited by 6
  32. 2026Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibilitycited by 6
  33. 2025AgentPack: A Dataset of Code Changes, Co-Authored by Agents and Humanscited by 5
  34. 2026CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Modelscited by 5
  35. 2026Rethinking KV Cache Eviction via a Unified Information-Theoretic Objectivecited by 5
  36. 2025LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?cited by 4
  37. 2025QueST: Incentivizing LLMs to Generate Difficult Problemscited by 4
  38. 2025Closing the Data Loop: Using OpenDataArena to Engineer Superior Training Datasetscited by 4

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025CodeReasoner: Enhancing the Code Reasoning Ability with Reinforcement Learning
  2. 2025Not All Correct Answers Are Equal: Why Your Distillation Source Matters
  3. 2025Chain of Execution Supervision Promotes General Reasoning in Large Language Models
  4. 2024Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoning
  5. 2026Think Multilingual, Not Harder: A Data-Efficient Framework for Teaching Reasoning Models to Code-Switch
  6. 2025Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuning
  7. 2025Seed-Coder: Let the Code Model Curate Data for Itself
  8. 2025Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning