mlsanityData to 14 Oct 2026

StarCoder: may the source be with you!

arXiv 2305.06161 ↗First posted 9 May 2023cs.CL cs.AI cs.PLcited by 799

Not on the map. Why?

Cited by 75 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context length, infilling capabilities and fast large-batch inference enabled by multi-query attention. StarCoderBase is trained on 1 trillion tokens sourced from The Stack, a large collection of permissively licensed GitHub repositories with inspection tools and an opt-out process. We fine-tuned StarCoderBase on 35B Python tokens, resulting in the creation of StarCoder. We perform the most comprehensive evaluation of Code LLMs to date and show that StarCoderBase outperforms every open Code LLM that supports multiple programming languages and matches or outperforms the OpenAI code-cushman-001 model. Furthermore, StarCoder outperforms every model that is fine-tuned on Python, can be prompted to achieve 40% pass@1 on HumanEval, and still retains its performance on other programming languages. We take several important steps towards a safe open-access model release, including an improved PII redaction pipeline and a novel attribution tracing tool, and make the StarCoder models publicly available under a more commercially viable version of the Open Responsible AI Model license.

Key facts. Cited by 799 papers in the map (data through 14 Oct 2026). Builds on 74 of its 113 references that are in the map. First posted on arXiv 9 May 2023.

Builds on

74 of 113 references.

References · 74

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  11. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  12. 2020Scaling Laws for Neural Language Modelscited by 6,586
38 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  12. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  13. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  14. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  15. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  16. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  17. 2018Model Cards for Model Reportingcited by 1,080
  18. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  19. 2022Red Teaming Language Models with Language Modelscited by 1,026
  20. 2022In-context Learning and Induction Headscited by 964
  21. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  22. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  23. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  24. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  25. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  26. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  27. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  28. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  29. 2018CoQA: A Conversational Question Answering Challengecited by 683
  30. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
  31. 2022PAL: Program-aided Language Modelscited by 652
  32. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
  33. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  34. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  35. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
  36. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  37. 2021Unified Pre-training for Program Understanding and Generationcited by 366
  38. 2022A Systematic Evaluation of Large Language Models of Codecited by 363

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 145; 2024: 322; 2025: 201; 2026: 131.

799 papers.

Sort:

By area:

  1. 2023Qwen Technical Reportcited by 3,344
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  3. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  6. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  7. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  8. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  9. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  10. 2023A Comprehensive Overview of Large Language Modelscited by 648
  11. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  12. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
38 more
  1. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  2. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  3. 2023Textbooks Are All You Needcited by 569
  4. 2023The Falcon Series of Open Language Modelscited by 535
  5. 2024Large Language Models: A Surveycited by 525
  6. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  7. 2023Cognitive Architectures for Language Agentscited by 429
  8. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 427
  9. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  10. 2023Scaling Data-Constrained Language Modelscited by 403
  11. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  12. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  13. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  14. 2024LoRA Learns Less and Forgets Lesscited by 328
  15. 2024Discrete Flow Matchingcited by 317
  16. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  17. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  18. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  19. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  20. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 282
  21. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 274
  22. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 274
  23. 2024Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Modelcited by 268
  24. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  25. 2023FireAct: Toward Language Agent Fine-tuningcited by 256
  26. 2024Generative Representational Instruction Tuningcited by 233
  27. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  28. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 200
  29. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197
  30. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 192
  31. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  32. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  33. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 187
  34. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 183
  35. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  36. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 178
  37. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 168
  38. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161

50 most cited of 799 All 799 (JSON)