mlsanityData to 7 Oct 2026

SantaCoder: don't reach for the stars!

arXiv 2301.03988 ↗First posted 9 Jan 2023cs.SE cs.AI cs.LGcited by 169

Not on the map. Why?

Abstract (arXiv)

The BigCode project is an open-scientific collaboration working on the responsible development of large language models for code. This tech report describes the progress of the collaboration until December 2022, outlining the current state of the Personally Identifiable Information (PII) redaction pipeline, the experiments conducted to de-risk the model architecture, and the experiments investigating better preprocessing methods for the training data. We train 1.1B parameter models on the Java, JavaScript, and Python subsets of The Stack and evaluate them on the MultiPL-E text-to-code benchmark. We find that more aggressive filtering of near-duplicates can further boost performance and, surprisingly, that selecting files from repositories with 5+ GitHub stars deteriorates performance significantly. Our best model outperforms previous open-source multilingual code generation models (InCoder-6.7B and CodeGen-Multi-2.7B) in both left-to-right generation and infilling on the Java, JavaScript, and Python portions of MultiPL-E, despite being a substantially smaller model. All models are released under an OpenRAIL license at https://hf.co/bigcode.

Key facts. Cited by 169 papers in the map (data through 7 Oct 2026). Builds on 33 of its 47 references that are in the map. First posted on arXiv 9 Jan 2023.

Builds on

33 of 47 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  9. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  10. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  11. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  12. 2021Measuring Coding Challenge Competence With APPScited by 879
21 more
  1. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  2. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  3. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  4. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  5. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  6. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  7. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  8. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  9. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  10. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  11. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  12. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  13. 2019SPoC: Search-based Pseudocode to Codecited by 176
  14. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  15. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  16. 2018Mapping Language to Code in Programmatic Contextcited by 104
  17. 2021DOBF: A Deobfuscation Pre-Training Objective for Programming Languagescited by 68
  18. 2022CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequencescited by 49
  19. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 47
  20. 2021Multilingual training for Software Engineeringcited by 43
  21. 2019TypeWriter: Neural Type Prediction with Search-based Validationcited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 55; 2024: 76; 2025: 29; 2026: 9.

169 papers.

Sort:

By area:

  1. 2023Qwen Technical Reportcited by 3,273
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  4. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  5. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  6. 2023StarCoder: may the source be with you!cited by 790
  7. 2023Textbooks Are All You Needcited by 561
  8. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  9. 2023Scaling Data-Constrained Language Modelscited by 391
  10. 2023Llemma: An Open Language Model For Mathematicscited by 385
  11. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  12. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
38 more
  1. 2024LoRA Learns Less and Forgets Lesscited by 313
  2. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  3. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  5. 2024Generative Representational Instruction Tuningcited by 227
  6. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  7. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  8. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solvingcited by 177
  9. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  10. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 149
  11. 2023What's In My Big Data?cited by 129
  12. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  13. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 124
  14. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  15. 2023Who Wrote this Code? Watermarking for Code Generationcited by 116
  16. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 116
  17. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  18. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
  19. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  20. 2024Large Language Models Meet NLP: A Surveycited by 102
  21. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  22. 2024MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualizationcited by 98
  23. 2023StarVector: Generating Scalable Vector Graphics Code from Images and Textcited by 96
  24. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 92
  25. 2024Scaling Laws for Precisioncited by 92
  26. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 86
  27. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  28. 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedbackcited by 74
  29. 2023PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedbackcited by 69
  30. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  31. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
  32. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  33. 2023Large Language Models for Compiler Optimizationcited by 52
  34. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  35. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 50
  36. 2023FinGPT: Large Generative Models for a Small Languagecited by 49
  37. 2024A Performance Study of LLM-Generated Code on Leetcodecited by 48
  38. 2025Seed-Coder: Let the Code Model Curate Data for Itselfcited by 46

50 most cited of 169 All 169 (JSON)