mlsanityData to 7 Oct 2026

OctoPack: Instruction Tuning Code Large Language Models

arXiv 2308.07124 ↗First posted 14 Aug 2023cs.CL cs.AIcited by 178

Not on the map. Why?

Cited by 21 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Finetuning large language models (LLMs) on instructions leads to vast performance improvements on natural language tasks. We apply instruction tuning using code, leveraging the natural structure of Git commits, which pair code changes with human instructions. We compile CommitPack: 4 terabytes of Git commits across 350 programming languages. We benchmark CommitPack against other natural and synthetic code instructions (xP3x, Self-Instruct, OASST) on the 16B parameter StarCoder model, and achieve state-of-the-art performance among models not trained on OpenAI outputs, on the HumanEval Python benchmark (46.2% pass@1). We further introduce HumanEvalPack, expanding the HumanEval benchmark to a total of 3 coding tasks (Code Repair, Code Explanation, Code Synthesis) across 6 languages (Python, JavaScript, Java, Go, C++, Rust). Our models, OctoCoder and OctoGeeX, achieve the best performance across HumanEvalPack among all permissive models, demonstrating CommitPack's benefits in generalizing to a wider set of languages and natural coding tasks. Code, models and data are freely available at https://github.com/bigcode-project/octopack.

Key facts. Cited by 178 papers in the map (data through 7 Oct 2026). Builds on 134 of its 168 references that are in the map. First posted on arXiv 14 Aug 2023.

Builds on

134 of 168 references.

References · 134

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
38 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,509
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  13. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  14. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  15. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  16. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  17. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  18. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  19. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  20. 2023LIMA: Less Is More for Alignmentcited by 1,080
  21. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  22. 2023Teaching Large Language Models to Self-Debugcited by 963
  23. 2021Measuring Coding Challenge Competence With APPScited by 879
  24. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  25. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  26. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  27. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  28. 2023StarCoder: may the source be with you!cited by 790
  29. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  30. 2022Galactica: A Large Language Model for Sciencecited by 736
  31. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  32. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  33. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  34. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  35. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  36. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  37. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  38. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700

50 most cited of 134 All 134 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 81; 2025: 61; 2026: 26.

178 papers.

Sort:

By area:

  1. 2023Qwen Technical Reportcited by 3,273
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  4. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  5. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  6. 2024LoRA Learns Less and Forgets Lesscited by 313
  7. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  8. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  9. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 255
  10. 2024Generative Representational Instruction Tuningcited by 227
  11. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  12. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
38 more
  1. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  2. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  3. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 122
  4. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  5. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  6. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  7. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  8. 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 66
  9. 2024Instruction Tuning for Secure Code Generationcited by 63
  10. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
  11. 2024NExT: Teaching Large Language Models to Reason about Code Executioncited by 60
  12. 2024Agents in Software Engineering: Survey, Landscape, and Visioncited by 60
  13. 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 59
  14. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 56
  15. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 55
  16. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  17. 2023RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repaircited by 50
  18. 2024Long Code Arena: a Set of Benchmarks for Long-Context Code Modelscited by 48
  19. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  20. 2023Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructionscited by 46
  21. 2024From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debuggingcited by 46
  22. 2023Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMscited by 45
  23. 2024Improving Reward Models with Synthetic Critiquescited by 44
  24. 2024CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repaircited by 41
  25. 2024CodeV: Empowering LLMs with HDL Generation through Multi-Level Summarizationcited by 38
  26. 2025BaxBench: Can LLMs Generate Correct and Secure Backends?cited by 38
  27. 2025Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworkscited by 36
  28. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 34
  29. 2024RAR-b: Reasoning as Retrieval Benchmarkcited by 33
  30. 2025What Prompts Don't Say: Understanding and Managing Underspecification in LLM Promptscited by 32
  31. 2024CodeMind: Evaluating Large Language Models for Code Reasoningcited by 30
  32. 2024Zero-Shot Tokenizer Transfercited by 30
  33. 2026SWE-Lego: Pushing the Limits of Supervised Fine-tuning for Software Issue Resolvingcited by 30
  34. 2024Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Modelscited by 29
  35. 2024The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmerscited by 28
  36. 2024EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimizationcited by 28
  37. 2025Multi-Turn Code Generation Through Single-Step Rewardscited by 27
  38. 2024MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generationcited by 25

50 most cited of 178 All 178 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
  3. 2026OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding
  4. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling
  5. 2026SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring
  6. 2024Mercury: A Code Efficiency Benchmark for Code Large Language Models
  7. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement
  8. 2024XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts