mlsanityData to 7 Oct 2026

OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs

arXiv 2504.04030 ↗First posted 5 Apr 2025cs.SE cs.CLcited by 57

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have transformed software development by enabling code generation, automated debugging, and complex reasoning. However, their continued advancement is constrained by the scarcity of high-quality, publicly available supervised fine-tuning (SFT) datasets tailored for coding tasks. To bridge this gap, we introduce OpenCodeInstruct, the largest open-access instruction tuning dataset, comprising 5 million diverse samples. Each sample includes a programming question, solution, test cases, execution feedback, and LLM-generated quality assessments. We fine-tune various base models, including LLaMA and Qwen, across multiple scales (1B+, 3B+, and 7B+) using our dataset. Comprehensive evaluations on popular benchmarks (HumanEval, MBPP, LiveCodeBench, and BigCodeBench) demonstrate substantial performance improvements achieved by SFT with OpenCodeInstruct. We also present a detailed methodology encompassing seed data curation, synthetic instruction and solution generation, and filtering.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 54 of its 63 references that are in the map. First posted on arXiv 5 Apr 2025.

Builds on

54 of 63 references.

References · 54

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  11. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  12. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
38 more
  1. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  2. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  3. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  4. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  5. 2023Teaching Large Language Models to Self-Debugcited by 963
  6. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  7. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  8. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  9. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  10. 2023Textbooks Are All You Needcited by 561
  11. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  12. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  13. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  14. 2020Unsupervised Translation of Programming Languagescited by 262
  15. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  16. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  17. 2024RepairAgent: An Autonomous, LLM-Based Agent for Program Repaircited by 225
  18. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
  19. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  20. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  21. 2023TACO: Topics in Algorithmic COde generation datasetcited by 150
  22. 2022Practical Program Repair in the Era of Large Pre-trained Language Modelscited by 133
  23. 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agentscited by 129
  24. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  25. 2023Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPTcited by 125
  26. 2023Learning Performance-Improving Code Editscited by 103
  27. 2022Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Modelscited by 102
  28. 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 99
  29. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 90
  30. 2023Impact of Code Language Models on Automated Program Repaircited by 85
  31. 2022Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learningcited by 76
  32. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 61
  33. 2021AVATAR: A Parallel Corpus for Java-Python Program Translationcited by 59
  34. 2023Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repaircited by 55
  35. 2024Repoformer: Selective Retrieval for Repository-Level Code Completioncited by 55
  36. 2023Large Language Models for Compiler Optimizationcited by 52
  37. 2023WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuningcited by 42
  38. 2023Understanding In-Context Learning from Repetitionscited by 27

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 43.

57 papers.

Sort:

By area:

  1. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 94
  2. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
  3. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  4. 2025OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critiquecited by 24
  5. 2026DMax: Aggressive Parallel Decoding for dLLMscited by 14
  6. 2025Fast and Accurate Causal Parallel Decoding using Jacobi Forcingcited by 11
  7. 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillationcited by 11
  8. 2026Goedel-Code-Prover: Hierarchical Proof Search for Open State-of-the-Art Code Verificationcited by 10
  9. 2025DEER: Draft with Diffusion, Verify with Autoregressive Modelscited by 9
  10. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  11. 2026CD4LM: Consistency Distillation and aDaptive Decoding for Diffusion Language Modelscited by 8
  12. 2026Stop Training for the Worst: Progressive Unmasking Accelerates Masked Diffusion Trainingcited by 8
38 more
  1. 2025When Style Breaks Safety: Defending LLMs Against Superficial Style Alignmentcited by 6
  2. 2026Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Modelscited by 6
  3. 2026Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMscited by 6
  4. 2025Scaling Towards the Information Boundary of Instruction Sets: The Infinity Instruct Subject Technical Reportcited by 5
  5. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 5
  6. 2026P-EAGLE: Parallel-Drafting EAGLE with Scalable Trainingcited by 5
  7. 2026CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Modelscited by 5
  8. 2025PRO: Enabling Precise and Robust Text Watermark for Open-Source LLMscited by 4
  9. 2026AngelSpec: Towards Real-World High Performance Inference with Speculative Decodingcited by 4
  10. 2026Robust Policy Optimization to Prevent Catastrophic Forgettingcited by 3
  11. 2026Step-Level Sparse Autoencoder for Reasoning Process Interpretationcited by 3
  12. 2026Expert-Choice Routing Enables Adaptive Computation in Diffusion Language Modelscited by 3
  13. 2026Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMscited by 3
  14. 2026Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantizationcited by 3
  15. 2025WebGen-V Bench: Structured Representation for Enhancing Visual Design in LLM-based Web Generation and Evaluationcited by 2
  16. 2026Data Science and Technology Towards AGI Part I: Tiered Data Managementcited by 2
  17. 2026Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Modelscited by 2
  18. 2026Yet Even Less Is Even Better For Agentic, Reasoning, and Coding LLMscited by 2
  19. 2026Demystifying Data Organization for Enhanced LLM Trainingcited by 2
  20. 2026Masked Language Flow Modelscited by 2
  21. 2026TailSFT: Filtered Fine-Tuning Improves Post-Training Performancecited by 2
  22. 2025From Context to Intent: Reasoning-Guided Function-Level Code Completioncited by 1
  23. 2026InfoSynth: Information-Guided Benchmark Synthesis for LLMscited by 1
  24. 2026BackPlay: Head-Only Look-Back Self-Correction for Diffusion Language Modelscited by 1
  25. 2026Learning to Learn from Language Feedback with Social Meta-Learningcited by 1
  26. 2026Self-Improving Code Generation via Semantic Entropy and Behavioral Consensuscited by 1
  27. 2026CODEBLOCK: Learning to Supervise Code at the Right Granularitycited by 1
  28. 2026Self-CTRL: Self-Consistency Training with Reinforcement Learningcited by 1
  29. 2026CForce: Boosting Parallel Decoding for dLLMs via Consistency Forcingcited by 1
  30. 2026ELF-REG: Scaling Continuous Diffusion Language Models to Reasoning Taskscited by 1
  31. 2025Toward Explaining Large Language Models in Software Engineering Taskscited by 0
  32. 2025State-of-the-art Small Language Coder Model: Mify-Codercited by 0
  33. 2026Ability Transfer and Recovery via Modularized Parameters Localizationcited by 0
  34. 2026Improving LLM Code Reasoning via Semantic Equivalence Self-Play with Formal Verificationcited by 0
  35. 2026Domain-Adaptable Reinforcement Learning for Code Generation with Dense Rewardscited by 0
  36. 2026Semantic Flow Regularization: Teaching LLMs to Generate Diverse Yet Coherent Responsescited by 0
  37. 2026Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Codecited by 0
  38. 2026Muonp: Muon with Fractional Spectral Powerscited by 0

50 most cited of 57 All 57 (JSON)