mlsanityData to 14 Oct 2026

Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering

arXiv 2401.08500 ↗First posted 16 Jan 2024cs.LG cs.CL cs.SEcited by 103

Not on the map. Why?

Cited by 45 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Code generation problems differ from common natural language problems - they require matching the exact syntax of the target language, identifying happy paths and edge cases, paying attention to numerous small details in the problem spec, and addressing other code-specific issues and requirements. Hence, many of the optimizations and tricks that have been successful in natural language generation may not be effective for code tasks. In this work, we propose a new approach to code generation by LLMs, which we call AlphaCodium - a test-based, multi-stage, code-oriented iterative flow, that improves the performances of LLMs on code problems. We tested AlphaCodium on a challenging code generation dataset called CodeContests, which includes competitive programming problems from platforms such as Codeforces. The proposed flow consistently and significantly improves results. On the validation set, for example, GPT-4 accuracy (pass@5) increased from 19% with a single well-designed direct prompt to 44% with the AlphaCodium flow. Many of the principles and best practices acquired in this work, we believe, are broadly applicable to general code generation tasks. Full implementation is available at: https://github.com/Codium-ai/AlphaCodium

Key facts. Cited by 103 papers in the map (data through 14 Oct 2026). Builds on 9 of its 16 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

9 of 16 references.

References · 9

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  6. 2021Measuring Coding Challenge Competence With APPScited by 902
  7. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 329
  8. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 324
  9. 2023CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modulescited by 58

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 29; 2025: 39; 2026: 35.

103 papers.

Sort:

By area:

  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  2. 2024A Survey on Large Language Models for Code Generationcited by 658
  3. 2024AFlow: Automating Agentic Workflow Generationcited by 432
  4. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
  5. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 178
  6. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solvingcited by 177
  7. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 158
  8. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  9. 2025S*: Test Time Scaling for Code Generationcited by 98
  10. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 86
  11. 2025ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimizationcited by 56
  12. 2025Co-Evolving LLM Coder and Unit Tester via Reinforcement Learningcited by 56
38 more
  1. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
  2. 2024MASAI: Modular Architecture for Software-engineering AI Agentscited by 49
  3. 2026Code as Agent Harnesscited by 47
  4. 2025Automated Hypothesis Validation with Agentic Sequential Falsificationscited by 44
  5. 2025Self-Supervised Prompt Optimizationcited by 37
  6. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  7. 2025CODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debuggingcited by 32
  8. 2025Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositoriescited by 31
  9. 2024What Makes Large Language Models Reason in (Multi-Turn) Code Generation?cited by 29
  10. 2025Automated Benchmark Generation for Repository-Level Coding Taskscited by 29
  11. 2024CodeS: Natural Language to Code Repository via Multi-Layer Sketchcited by 28
  12. 2025Multi-Turn Code Generation Through Single-Step Rewardscited by 28
  13. 2024ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?cited by 27
  14. 2024DataNarrative: Automated Data-Driven Storytelling with Visualizations and Textscited by 25
  15. 2024Materials science in the era of large language models: a perspectivecited by 24
  16. 2026RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL Systemcited by 24
  17. 2025Revisit Self-Debugging with Self-Generated Tests for Code Generationcited by 22
  18. 2025Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executorscited by 22
  19. 2025SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugscited by 22
  20. 2024A Pair Programming Framework for Code Generation via Multi-Plan Exploration and Feedback-Driven Refinementcited by 21
  21. 2024The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?cited by 20
  22. 2024MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domainscited by 19
  23. 2025debug-gym: A Text-Based Environment for Interactive Debuggingcited by 19
  24. 2025GraphCodeAgent: Dual Graph-Guided LLM Agent for Retrieval-Augmented Repo-Level Code Generationcited by 17
  25. 2025Beyond ReAct: A Planner-Centric Framework for Complex Tool-Augmented LLM Reasoningcited by 17
  26. 2025ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programmingcited by 16
  27. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  28. 2025A2Flow: Automating Agentic Workflow Generation via Self-Adaptive Abstraction Operatorscited by 14
  29. 2024HDFlow: Enhancing LLM Complex Problem-Solving with Hybrid Thinking and Dynamic Workflowscited by 13
  30. 2025Scoring Verifiers: Evaluating Synthetic Verification for Code and Reasoningcited by 9
  31. 2025BioDSA-1K: Benchmarking Data Science Agents for Biomedical Researchcited by 8
  32. 2024Collaborative Evolving Strategy for Automatic Data-Centric Developmentcited by 6
  33. 2024Hermes: A Large Language Model Framework on the Journey to Autonomous Networkscited by 6
  34. 2024MageBench: Bridging Large Multimodal Models to Agentscited by 6
  35. 2024Is Functional Correctness Enough to Evaluate Code Language Models? Exploring Diversity of Generated Codescited by 5
  36. 2024Programming with AI: Evaluating ChatGPT, Gemini, AlphaCode, and GitHub Copilot for Programmerscited by 5
  37. 2025ASyMOB: Algebraic Symbolic Mathematical Operations Benchmarkcited by 5
  38. 2026MechMath Agent Team: LLM Driven Agents for Mathematical Researchcited by 5

50 most cited of 103 All 103 (JSON)