mlsanityData to 7 Oct 2026

Competition-Level Code Generation with AlphaCode

arXiv 2203.07814 ↗First posted 8 Feb 2022cs.PL cs.AI cs.LGcited by 1,427

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Programming is a powerful and ubiquitous problem-solving tool. Developing systems that can assist programmers or even generate programs independently could make programming more productive and accessible, yet so far incorporating innovations in AI has proven challenging. Recent large-scale language models have demonstrated an impressive ability to generate code, and are now able to complete simple programming tasks. However, these models still perform poorly when evaluated on more complex, unseen problems that require problem-solving skills beyond simply translating instructions into code. For example, competitive programming problems which require an understanding of algorithms and complex natural language remain extremely challenging. To address this gap, we introduce AlphaCode, a system for code generation that can create novel solutions to these problems that require deeper reasoning. In simulated evaluations on recent programming competitions on the Codeforces platform, AlphaCode achieved on average a ranking of top 54.3% in competitions with more than 5,000 participants. We found that three key components were critical to achieve good and reliable performance: (1) an extensive and clean competitive programming dataset for training and evaluation, (2) large and efficient-to-sample transformer-based architectures, and (3) large-scale model sampling to explore the search space, followed by filtering based on program behavior to a small set of submissions.

Key facts. Cited by 1,427 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 36 of its 83 references that are in the map. First posted on arXiv 8 Feb 2022.

Builds on

36 of 83 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2020Extracting Training Data from Large Language Modelscited by 2,144
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
24 more
  1. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  2. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  4. 2018Hierarchical Neural Story Generationcited by 1,182
  5. 2021Ethical and social risks of harm from Language Modelscited by 957
  6. 2021Measuring Coding Challenge Competence With APPScited by 879
  7. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  8. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  9. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  10. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  11. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  12. 2019SPoC: Search-based Pseudocode to Codecited by 176
  13. 2016DeepCoder: Learning to Write Programscited by 145
  14. 2017RobustFill: Neural Program Learning under Noisy I/Ocited by 124
  15. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 123
  16. 2019Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagationcited by 87
  17. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  18. 2016Latent Predictor Networks for Code Generationcited by 73
  19. 2021Learning to Synthesize Programs as Interpretable and Generalizable Policiescited by 59
  20. 2020Learning Autocompletion from Real-World Datasetscited by 19
  21. 2021Solving Linear Algebra by Program Synthesiscited by 14
  22. 2020Neural Learning of One-of-Many Solutions for Combinatorial Problems in Structured Output Spacescited by 12
  23. 2021Solving Probability and Statistics Problems by Program Synthesiscited by 9
  24. 2020Softmax Tempering for Training Neural Machine Translation Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 95; 2023: 270; 2024: 316; 2025: 401; 2026: 345.

1,427 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  12. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
38 more
  1. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  2. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  3. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  4. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  5. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  8. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  9. 2023Teaching Large Language Models to Self-Debugcited by 963
  10. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  11. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  12. 2022A Generalist Agentcited by 802
  13. 2023StarCoder: may the source be with you!cited by 790
  14. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 763
  15. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  16. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  17. 2024A Survey on Large Language Models for Code Generationcited by 646
  18. 2023A Comprehensive Overview of Large Language Modelscited by 639
  19. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  20. 2023Textbooks Are All You Needcited by 561
  21. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  22. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  23. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  24. 2025LIMO: Less is More for Reasoningcited by 488
  25. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  26. 2025Process Reinforcement through Implicit Rewardscited by 478
  27. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  28. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  29. 2023Cognitive Architectures for Language Agentscited by 422
  30. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
  31. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  32. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  33. 2022Self-critiquing models for assisting human evaluatorscited by 372
  34. 2022CodeT: Code Generation with Generated Testscited by 369
  35. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  36. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  37. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  38. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344

50 most cited of 1,427 All 1,427 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating and Improving Large Language Models for Competitive Program Generation
  2. 2023Think Outside the Code: Brainstorming Boosts Large Language Models in Code Generation
  3. 2024Seed-CTS: Unleashing the Power of Tree Search for Superior Performance in Competitive Coding Tasks
  4. 2025Beyond the Surface: A Solution-Aware Retrieval Model for Competition-level Code Generation ↗
  5. 2025Enhancing LLM Code Generation: A Systematic Evaluation of Multi-Agent Collaboration and Runtime Debugging for Improved Accuracy, Reliability, and Latency
  6. 2025rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset
  7. 2025LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?
  8. 2025Increasing LLM Coding Capabilities through Diverse Synthetic Coding Tasks