mlsanityData to 14 Oct 2026

OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models

arXiv 2411.04905 ↗First posted 7 Nov 2024cs.CL cs.PLcited by 130

Not on the map. Why?

Cited by 36 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) for code have become indispensable in various domains, including code generation, reasoning tasks and agent systems. While open-access code LLMs are increasingly approaching the performance levels of proprietary models, high-quality code LLMs suitable for rigorous scientific investigation, particularly those with reproducible data processing pipelines and transparent training protocols, remain limited. The scarcity is due to various challenges, including resource constraints, ethical considerations, and the competitive advantages of keeping models advanced. To address the gap, we introduce OpenCoder, a top-tier code LLM that not only achieves performance comparable to leading models but also serves as an "open cookbook" for the research community. Unlike most prior efforts, we release not only model weights and inference code, but also the reproducible training data, complete data processing pipeline, rigorous experimental ablation results, and detailed training protocols for open scientific research. Through this comprehensive release, we identify the key ingredients for building a top-tier code LLM: (1) code optimized heuristic rules for data cleaning and methods for data deduplication, (2) recall of text corpus related to code and (3) high-quality synthetic data in both annealing and supervised fine-tuning stages. By offering this level of openness, we aim to broaden access to all aspects of a top-tier code LLM, with OpenCoder serving as both a powerful model and an open foundation to accelerate research, and enable reproducible advancements in code AI.

Key facts. Cited by 130 papers in the map (data through 14 Oct 2026). Builds on 68 of its 92 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

68 of 92 references.

References · 68

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2023Qwen Technical Reportcited by 3,344
  12. 2024Qwen2 Technical Reportcited by 2,120
38 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  3. 2020GLU Variants Improve Transformercited by 1,804
  4. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  5. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  6. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  9. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  10. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  11. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  12. 2023StarCoder: may the source be with you!cited by 799
  13. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  14. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  15. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  16. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  17. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  18. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
  19. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  20. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  21. 2023Textbooks Are All You Needcited by 569
  22. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  23. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  24. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  25. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  26. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  27. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  28. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  29. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  30. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  31. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  32. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  33. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  34. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 175
  35. 2023SantaCoder: don't reach for the stars!cited by 170
  36. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  37. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  38. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 136

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 81; 2026: 46.

130 papers.

Sort:

By area:

  1. 2025Kimi K2: Open Agentic Intelligencecited by 567
  2. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  3. 2025CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingscited by 115
  4. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  5. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 104
  6. 2025OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scalecited by 91
  7. 2025A Survey on Latent Reasoningcited by 81
  8. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergycited by 77
  9. 2025NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agentscited by 65
  10. 2025OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMscited by 60
  11. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 60
  12. 2025CodeI/O: Condensing Reasoning Patterns via Code Input-Output Predictioncited by 55
38 more
  1. 2025ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinementcited by 54
  2. 2025rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Datasetcited by 51
  3. 2025Any-Order Flexible Length Masked Diffusioncited by 51
  4. 20251.4 Million Open-Source Distilled Reasoning Dataset to Empower Large Language Model Trainingcited by 49
  5. 2025Diffusion Language Models are Super Data Learnerscited by 48
  6. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 48
  7. 2025Fine-Tuning Masked Diffusion for Provable Self-Correctioncited by 39
  8. 2025Arctic-Text2SQL-R1: Simple Rewards, Strong Reasoning in Text-to-SQLcited by 38
  9. 2025Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Modelscited by 37
  10. 2025Solving Formal Math Problems by Decomposition and Iterative Reflectioncited by 35
  11. 2025Continuously Augmented Discrete Diffusion model for Categorical Generative Modelingcited by 34
  12. 2025LaDiR: Latent Diffusion Enhances LLMs for Text Reasoningcited by 29
  13. 2024ProSec: Fortifying Code LLMs with Proactive Security Alignmentcited by 25
  14. 2025Mid-Training of Large Language Models: A Surveycited by 21
  15. 2024ExecRepoBench: Multi-level Executable Code Completion Evaluationcited by 20
  16. 2025SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Mannercited by 20
  17. 2025FineMedLM-o1: Enhancing Medical Knowledge Reasoning Ability of LLM from Supervised Fine-Tuning to Test-Time Trainingcited by 19
  18. 2025Sequential Diffusion Language Modelscited by 19
  19. 2026DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvascited by 19
  20. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 18
  21. 2025DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Trainingcited by 18
  22. 2025Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimizationcited by 18
  23. 2025PurpCode: Reasoning for Safer Code Generationcited by 18
  24. 2025ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decodingcited by 18
  25. 2025Agent-Q: Fine-Tuning Large Language Models for Quantum Circuit Generation and Optimizationcited by 17
  26. 2025Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Spacecited by 17
  27. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  28. 2026dLLM: Simple Diffusion Language Modelingcited by 14
  29. 2025Every Sample Matters: Leveraging Mixture-of-Experts and High-Quality Data for Efficient and Accurate Code LLMcited by 13
  30. 2026Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Modelcited by 13
  31. 2024Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Modelscited by 12
  32. 2025Data Mixing Optimization for Supervised Fine-Tuning of Large Language Modelscited by 12
  33. 2025Text2SQL-Flow: A Robust SQL-Aware Data Augmentation Framework for Text-to-SQLcited by 11
  34. 2025BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoningcited by 10
  35. 2025MTIR-SQL: Multi-turn Tool-Integrated Reasoning Reinforcement Learning for Text-to-SQLcited by 10
  36. 2025TuRTLe: A Unified Evaluation of LLMs for RTL Generationcited by 9
  37. 2025PromptCoT 2.0: Scaling Prompt Synthesis for Large Language Model Reasoningcited by 9
  38. 2025Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levelscited by 8

50 most cited of 130 All 130 (JSON)