mlsanityData to 7 Oct 2026

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

arXiv 2403.07974 ↗First posted 12 Mar 2024cs.SE cs.CL cs.LGcited by 1,906

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Large Language Models (LLMs) applied to code-related applications have emerged as a prominent field, attracting significant interest from both academia and industry. However, as new and improved LLMs are developed, existing evaluation benchmarks (e.g., HumanEval, MBPP) are no longer sufficient for assessing their capabilities. In this work, we propose LiveCodeBench, a comprehensive and contamination-free evaluation of LLMs for code, which continuously collects new problems over time from contests across three competition platforms, namely LeetCode, AtCoder, and CodeForces. Notably, our benchmark also focuses on a broader range of code related capabilities, such as self-repair, code execution, and test output prediction, beyond just code generation. Currently, LiveCodeBench hosts four hundred high-quality coding problems that were published between May 2023 and May 2024. We have evaluated 18 base LLMs and 34 instruction-tuned LLMs on LiveCodeBench. We present empirical findings on contamination, holistic performance comparisons, potential overfitting in existing benchmarks as well as individual model comparisons. We will release all prompts and model completions for further community analysis, along with a general toolkit for adding new scenarios and model

Key facts. Cited by 1,906 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 70 of its 90 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

70 of 90 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement22 Feb
  2. 2023Competition-Level Problems are Effective LLM Evaluators4 Dec
  3. 2023Teaching Large Language Models to Self-Debug11 Apr
  4. 2022Competition-Level Code Generation with AlphaCode8 Feb

Other references · 66

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  10. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  11. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  12. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
38 more
  1. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  2. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  3. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  4. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  5. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  6. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  7. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  8. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  9. 2022CodeT: Code Generation with Generated Testscited by 369
  10. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  11. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  12. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  13. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  14. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  15. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  16. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  17. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  18. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  19. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
  20. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  21. 2018code2seq: Generating Sequences from Structured Representations of Codecited by 177
  22. 2019SPoC: Search-based Pseudocode to Codecited by 176
  23. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 175
  24. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  25. 2023TACO: Topics in Algorithmic COde generation datasetcited by 150
  26. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  27. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 110
  28. 2023Large Language Models as Analogical Reasonerscited by 107
  29. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  30. 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
  31. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  32. 2024Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineeringcited by 100
  33. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 89
  34. 2023No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generationcited by 82
  35. 2019A Neural Model for Generating Natural Language Summaries of Program Subroutinescited by 76
  36. 2024RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generationcited by 69
  37. 2022CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Contextcited by 64
  38. 2024Quantifying Contamination in Evaluating Code Generation Capabilities of Language Modelscited by 62

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 104; 2025: 731; 2026: 1071.

1,906 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2025Qwen3-VL Technical Reportcited by 2,534
  5. 2024Qwen2 Technical Reportcited by 2,076
  6. 2025s1: Simple test-time scalingcited by 1,311
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  8. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  10. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  11. 2024DeepSeek-V3 Technical Reportcited by 724
  12. 2025Qwen2.5-Omni Technical Reportcited by 704
38 more
  1. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  2. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  3. 2025Kimi K2: Open Agentic Intelligencecited by 523
  4. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  5. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  6. 2025Process Reinforcement through Implicit Rewardscited by 478
  7. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  8. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  9. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  10. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 423
  11. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  12. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
  13. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  14. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  15. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  16. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  17. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  18. 2026Reinforcement Learning via Self-Distillationcited by 285
  19. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
  20. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 276
  21. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  22. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 246
  23. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  24. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 236
  25. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  26. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
  27. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  28. 2025Dynamic Early Exit in Reasoning Modelscited by 217
  29. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  30. 2025AIDE: AI-Driven Exploration in the Space of Codecited by 206
  31. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  32. 2025Learning to Reason without External Rewardscited by 199
  33. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  34. 2025Survey on Evaluation of LLM-based Agentscited by 190
  35. 2026Gemma 4 Technical Reportcited by 187
  36. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 186
  37. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 182
  38. 2025Reasoning Models Can Be Effective Without Thinkingcited by 182

50 most cited of 1,906 All 1,906 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Holistic Evaluation of State-of-the-Art LLMs for Code Generation
  2. 2026Narrowing the Complexity Gap in the Evaluation of Large Language Models ↗
  3. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories
  4. 2025Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation
  5. 2025Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworks
  6. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories
  7. 2024Mercury: A Code Efficiency Benchmark for Code Large Language Models
  8. 2026CodeEval: A pedagogical approach for targeted evaluation of code-trained Large Language Models