mlsanityData to 7 Oct 2026

The Stack: 3 TB of permissively licensed source code

arXiv 2211.15533 ↗First posted 20 Nov 2022cs.CL cs.AIcited by 354

Not on the map. Why?

Cited by 21 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) play an ever-increasing role in the field of Artificial Intelligence (AI)--not only for natural language processing but also for code understanding and generation. To stimulate open and responsible research on LLMs for code, we introduce The Stack, a 3.1 TB dataset consisting of permissively licensed source code in 30 programming languages. We describe how we collect the full dataset, construct a permissively licensed subset, present a data governance plan, discuss limitations, and show promising results on text2code benchmarks by training 350M-parameter decoders on different Python subsets. We find that (1) near-deduplicating the data significantly boosts performance across all experiments, and (2) it is possible to match previously reported HumanEval and MBPP performance using only permissively licensed data. We make the dataset available at https://hf.co/BigCode, provide a tool called "Am I in The Stack" (https://hf.co/spaces/bigcode/in-the-stack) for developers to search The Stack for copies of their code, and provide a process for code to be removed from the dataset by following the instructions at https://www.bigcode-project.org/docs/about/the-stack/.

Key facts. Cited by 354 papers in the map (data through 7 Oct 2026). Builds on 34 of its 50 references that are in the map. First posted on arXiv 20 Nov 2022.

Builds on

34 of 50 references.

References · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2020Extracting Training Data from Large Language Modelscited by 2,144
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  12. 2018Datasheets for Datasetscited by 1,371
22 more
  1. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  2. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  3. 2021Measuring Coding Challenge Competence With APPScited by 879
  4. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  5. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  6. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  7. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  8. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  9. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  10. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  11. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  12. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  13. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  14. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  15. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 147
  16. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 123
  17. 2021DOBF: A Deobfuscation Pre-Training Objective for Programming Languagescited by 68
  18. 2022CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequencescited by 49
  19. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 47
  20. 2022Data Governance in the Age of Large-Scale Data-Driven Language Technologycited by 32
  21. 2022A Hazard Analysis Framework for Code Synthesis Large Language Modelscited by 22
  22. 2019COSET: A Benchmark for Evaluating Neural Program Embeddingscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 72; 2024: 134; 2025: 92; 2026: 55.

354 papers.

Sort:

By area:

  1. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  2. 2023StarCoder: may the source be with you!cited by 790
  3. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  4. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  5. 2024A Survey on Large Language Models for Code Generationcited by 646
  6. 2023A Comprehensive Overview of Large Language Modelscited by 639
  7. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  8. 2023Textbooks Are All You Needcited by 561
  9. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  10. 2023The Falcon Series of Open Language Modelscited by 526
  11. 2023Scaling Data-Constrained Language Modelscited by 391
  12. 2023Llemma: An Open Language Model For Mathematicscited by 385
38 more
  1. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  2. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  3. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  4. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  5. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  6. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  7. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 206
  8. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  9. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  10. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 183
  11. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  12. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  13. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  14. 2023SantaCoder: don't reach for the stars!cited by 169
  15. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 168
  16. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Proverscited by 161
  17. 2024RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectorscited by 157
  18. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  19. 2023REST: Retrieval-Based Speculative Decodingcited by 148
  20. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  21. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 136
  22. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  23. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 124
  24. 2023Foundation Models and Fair Usecited by 112
  25. 2024EuroLLM: Multilingual Language Models for Europecited by 103
  26. 2023StarVector: Generating Scalable Vector Graphics Code from Images and Textcited by 96
  27. 2024Reducing hallucination in structured outputs via Retrieval-Augmented Generationcited by 92
  28. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  29. 2025Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learningcited by 82
  30. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 75
  31. 2023FP8-LM: Training FP8 Large Language Modelscited by 75
  32. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  33. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  34. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  35. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 71
  36. 2025OmniSVG: A Unified Scalable Vector Graphics Generation Modelcited by 71
  37. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 70
  38. 2023MathPile: A Billion-Token-Scale Pretraining Corpus for Mathcited by 67

50 most cited of 354 All 354 (JSON)