# OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs

arXiv: https://arxiv.org/abs/2504.04030 · HTML: https://mlsanity.com/paper/2504.04030/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 63 references that are in the map.
- First posted on arXiv 5 Apr 2025.

## Builds on (most cited first, 30 of 54)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation](https://mlsanity.com/paper/2303.12570/) (2023)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [Unsupervised Translation of Programming Languages](https://mlsanity.com/paper/2006.03511/) (2020)
- [Magicoder: Empowering Code Generation with OSS-Instruct](https://mlsanity.com/paper/2312.02120/) (2023)
- [OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement](https://mlsanity.com/paper/2402.14658/) (2024)
- [RepairAgent: An Autonomous, LLM-Based Agent for Program Repair](https://mlsanity.com/paper/2403.17134/) (2024)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)

## Built on by (most cited first, 30 of 60)

- [NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model](https://mlsanity.com/paper/2508.14444/) (2025)
- [Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2512.20848/) (2025)
- [HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages](https://mlsanity.com/paper/2505.11475/) (2025)
- [OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critique](https://mlsanity.com/paper/2507.09075/) (2025)
- [DMax: Aggressive Parallel Decoding for dLLMs](https://mlsanity.com/paper/2604.08302/) (2026)
- [Fast and Accurate Causal Parallel Decoding using Jacobi Forcing](https://mlsanity.com/paper/2512.14681/) (2025)
- [SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation](https://mlsanity.com/paper/2605.07711/) (2026)
- [Goedel-Code-Prover: Hierarchical Proof Search for Open State-of-the-Art Code Verification](https://mlsanity.com/paper/2603.19329/) (2026)
- [Stop Training for the Worst: Progressive Unmasking Accelerates Masked Diffusion Training](https://mlsanity.com/paper/2602.10314/) (2026)
- [DEER: Draft with Diffusion, Verify with Autoregressive Models](https://mlsanity.com/paper/2512.15176/) (2025)
- [Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models](https://mlsanity.com/paper/2505.16056/) (2025)
- [CD4LM: Consistency Distillation and aDaptive Decoding for Diffusion Language Models](https://mlsanity.com/paper/2601.02236/) (2026)
- [Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs](https://mlsanity.com/paper/2602.07729/) (2026)
- [When Style Breaks Safety: Defending LLMs Against Superficial Style Alignment](https://mlsanity.com/paper/2506.07452/) (2025)
- [Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models](https://mlsanity.com/paper/2601.15220/) (2026)
- [A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data](https://mlsanity.com/paper/2601.17717/) (2026)
- [P-EAGLE: Parallel-Drafting EAGLE with Scalable Training](https://mlsanity.com/paper/2602.01469/) (2026)
- [Scaling Towards the Information Boundary of Instruction Sets: The Infinity Instruct Subject Technical Report](https://mlsanity.com/paper/2507.06968/) (2025)
- [CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Models](https://mlsanity.com/paper/2602.17684/) (2026)
- [Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization](https://mlsanity.com/paper/2608.01078/) (2026)
- [PRO: Enabling Precise and Robust Text Watermark for Open-Source LLMs](https://mlsanity.com/paper/2510.23891/) (2025)
- [Data Science and Technology Towards AGI Part I: Tiered Data Management](https://mlsanity.com/paper/2602.09003/) (2026)
- [AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters](https://mlsanity.com/paper/2607.19223/) (2026)
- [AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding](https://mlsanity.com/paper/2607.25852/) (2026)
- [WebGen-V Bench: Structured Representation for Enhancing Visual Design in LLM-based Web Generation and Evaluation](https://mlsanity.com/paper/2510.15306/) (2025)
- [Robust Policy Optimization to Prevent Catastrophic Forgetting](https://mlsanity.com/paper/2602.08813/) (2026)
- [Step-Level Sparse Autoencoder for Reasoning Process Interpretation](https://mlsanity.com/paper/2603.03031/) (2026)
- [Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models](https://mlsanity.com/paper/2603.12248/) (2026)
- [Expert-Choice Routing Enables Adaptive Computation in Diffusion Language Models](https://mlsanity.com/paper/2604.01622/) (2026)
- [Demystifying Data Organization for Enhanced LLM Training](https://mlsanity.com/paper/2605.30334/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have transformed software development by enabling code generation, automated debugging, and complex reasoning. However, their continued advancement is constrained by the scarcity of high-quality, publicly available supervised fine-tuning (SFT) datasets tailored for coding tasks. To bridge this gap, we introduce OpenCodeInstruct, the largest open-access instruction tuning dataset, comprising 5 million diverse samples. Each sample includes a programming question, solution, test cases, execution feedback, and LLM-generated quality assessments. We fine-tune various base models, including LLaMA and Qwen, across multiple scales (1B+, 3B+, and 7B+) using our dataset. Comprehensive evaluations on popular benchmarks (HumanEval, MBPP, LiveCodeBench, and BigCodeBench) demonstrate substantial performance improvements achieved by SFT with OpenCodeInstruct. We also present a detailed methodology encompassing seed data curation, synthetic instruction and solution generation, and filtering.