# StarCoder 2 and The Stack v2: The Next Generation

arXiv: https://arxiv.org/abs/2402.19173 · HTML: https://mlsanity.com/paper/2402.19173/

## Key facts

- Cited by 502 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Feb 2024.

## Built on by (most cited first, 30 of 502)

- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [Training Language Models to Self-Correct via Reinforcement Learning](https://mlsanity.com/paper/2409.12917/) (2024)
- [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025)
- [DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence](https://mlsanity.com/paper/2406.11931/) (2024)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)
- [InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output](https://mlsanity.com/paper/2407.03320/) (2024)
- [DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation](https://mlsanity.com/paper/2506.20639/) (2025)
- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [Building and better understanding vision-language models: insights and future directions](https://mlsanity.com/paper/2408.12637/) (2024)
- [CodeGemma: Open Code Models Based on Gemma](https://mlsanity.com/paper/2406.11409/) (2024)
- [Observational Scaling Laws and the Predictability of Language Model Performance](https://mlsanity.com/paper/2405.10938/) (2024)
- [OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models](https://mlsanity.com/paper/2411.04905/) (2024)
- [Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL](https://mlsanity.com/paper/2406.08426/) (2024)
- [Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies](https://mlsanity.com/paper/2407.13623/) (2024)
- [Granite Code Models: A Family of Open Foundation Models for Code Intelligence](https://mlsanity.com/paper/2405.04324/) (2024)
- [Scaling Laws for Precision](https://mlsanity.com/paper/2411.04330/) (2024)
- [LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation](https://mlsanity.com/paper/2409.20550/) (2024)
- [ACECODER: Acing Coder RL via Automated Test-Case Synthesis](https://mlsanity.com/paper/2502.01718/) (2025)
- [CodeRAG-Bench: Can Retrieval Augment Code Generation?](https://mlsanity.com/paper/2406.14497/) (2024)
- [OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scale](https://mlsanity.com/paper/2503.02240/) (2025)
- [Practical Efficiency of Muon for Pretraining](https://mlsanity.com/paper/2505.02222/) (2025)
- [TreeRL: LLM Reinforcement Learning with On-Policy Tree Search](https://mlsanity.com/paper/2506.11902/) (2025)
- [T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling](https://mlsanity.com/paper/2501.11651/) (2025)
- [EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories](https://mlsanity.com/paper/2404.00599/) (2024)
- [DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories](https://mlsanity.com/paper/2405.19856/) (2024)

## Abstract (arXiv)

The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on top of the digital commons of their source code archive. Alongside the SWH repositories spanning 619 programming languages, we carefully select other high-quality data sources, such as GitHub pull requests, Kaggle notebooks, and code documentation. This results in a training set that is 4x larger than the first StarCoder dataset. We train StarCoder2 models with 3B, 7B, and 15B parameters on 3.3 to 4.3 trillion tokens and thoroughly evaluate them on a comprehensive set of Code LLM benchmarks. We find that our small model, StarCoder2-3B, outperforms other Code LLMs of similar size on most benchmarks, and also outperforms StarCoderBase-15B. Our large model, StarCoder2- 15B, significantly outperforms other models of comparable size. In addition, it matches or outperforms CodeLlama-34B, a model more than twice its size. Although DeepSeekCoder- 33B is the best-performing model at code completion for high-resource languages, we find that StarCoder2-15B outperforms it on math and code reasoning benchmarks, as well as several low-resource languages. We make the model weights available under an OpenRAIL license and ensure full transparency regarding the training data by releasing the SoftWare Heritage persistent IDentifiers (SWHIDs) of the source code data.