mlsanityData to 7 Oct 2026

A Performance Study of LLM-Generated Code on Leetcode

arXiv 2407.21579 ↗First posted 31 Jul 2024cs.SE cs.AIcited by 48

Not on the map. Why?

Abstract (arXiv)

This study evaluates the efficiency of code generation by Large Language Models (LLMs) and measures their performance against human-crafted solutions using a dataset from Leetcode. We compare 18 LLMs, considering factors such as model temperature and success rate, and their impact on code performance. This research introduces a novel method for measuring and comparing the speed of LLM-generated code, revealing that LLMs produce code with comparable performance, irrespective of the adopted LLM. We also find that LLMs are capable of generating code that is, on average, more efficient than the code written by humans. The paper further discusses the use of Leetcode as a benchmarking dataset, the limitations imposed by potential data contamination, and the platform's measurement reliability. We believe that our findings contribute to a better understanding of LLM capabilities in code generation and set the stage for future optimizations in the field.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 26 of its 42 references that are in the map. First posted on arXiv 31 Jul 2024.

Builds on

26 of 42 references.

References · 26

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  7. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  8. 2021Measuring Coding Challenge Competence With APPScited by 879
  9. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  10. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  11. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  12. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
14 more
  1. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  2. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  3. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
  4. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  5. 2022Do Users Write More Insecure Code with AI Assistants?cited by 186
  6. 2023SantaCoder: don't reach for the stars!cited by 169
  7. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  8. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  9. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 47
  10. 2023TrojanPuzzle: Covertly Poisoning Code-Suggestion Modelscited by 47
  11. 2023Large Language Models and Simple, Stupid Bugscited by 45
  12. 2023Supersonic: Learning to Generate Source Code Optimizations in C/C++cited by 15
  13. 2023RAPGen: An Approach for Fixing Code Inefficiencies in Zero-Shotcited by 14
  14. 2022WhyGen: Explaining ML-powered Code Generation by Referring to Training Examplescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 37; 2026: 9.

48 papers.

Sort:

By area:

  1. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 39
  2. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 26
  3. 2025GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agentscited by 24
  4. 2025AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?cited by 24
  5. 2025Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruningcited by 13
  6. 2025AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learningcited by 13
  7. 2025From Prompts to Power: Measuring the Energy Footprint of LLM Inferencecited by 12
  8. 2025LLM4EFFI: Leveraging Large Language Models to Enhance Code Efficiency and Correctnesscited by 10
  9. 2025Comparative Analysis Based on DeepSeek, ChatGPT, and Google Gemini: Features, Techniques, Performance, Future Prospectscited by 9
  10. 2024LLM4DS: Evaluating Large Language Models for Data Science Code Generationcited by 7
  11. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resourcecited by 7
  12. 2025LLM-ProS: Analyzing Large Language Models' Performance in Competitive Problem Solvingcited by 6
36 more
  1. 2024The Impact of Hyperparameters on Large Language Model Inference Performance: An Evaluation of vLLM and HuggingFace Pipelinescited by 5
  2. 2025Performance Review on LLM for solving leetcode problemscited by 5
  3. 2025MARCO: Multi-Agent Code Optimization with Real-Time Knowledge Integration for High-Performance Computingcited by 5
  4. 2025Generating Energy-Efficient Code via Large-Language Models -- Where are we now?cited by 5
  5. 2025Teaching AI to Handle Exceptions: Supervised Fine-Tuning with Human-Aligned Judgmentcited by 4
  6. 2025EyeMulator: Improving Code Language Models by Mimicking Human Visual Attentioncited by 4
  7. 2025HypoSpace: A Diagnostic Benchmark for Set-Valued Hypothesis Generation under Underdetermination and Sublinear Coverage Boundscited by 4
  8. 2025Leveraging Metamemory Agent for Enhanced Data-Free Code Generation in Large Language Modelscited by 3
  9. 2025On the Evaluation of Engineering Artificial General Intelligencecited by 3
  10. 2025PerfCoder: Large Language Models for Interpretable Code Performance Optimizationcited by 3
  11. 2025CCCI: Code Completion with Contextual Information for Complex Data Transfer Tasks Using Large Language Modelscited by 2
  12. 2025Boardwalk: Towards a Framework for Creating Board Games with LLMscited by 2
  13. 2025Exploring Solution Divergence and Its Effect on Large Language Model Problem Solvingcited by 2
  14. 2025Efficient LLM Safety Evaluation through Multi-Agent Debatecited by 2
  15. 2025A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standardscited by 1
  16. 2025LLM Performance for Code Generation on Noisy Taskscited by 1
  17. 2025ReGA: Model-Based Safeguard for LLMs via Representation-Guided Abstractioncited by 1
  18. 2025Deployability-Centric Infrastructure-as-Code Generation: Fail, Learn, Refine, and Succeed through LLM-Empowered DevOps Simulationcited by 1
  19. 2025Can Large Language Models Capture Human Risk Preferences? A Cross-Cultural Studycited by 1
  20. 2025How Do Agentic AI Systems Address Performance Optimizations? A BERTopic-Based Analysis of Pull Requestscited by 1
  21. 2026RAT: RunAnyThing via Fully Automated Environment Configurationcited by 1
  22. 2026Characterizing Readability Issue Patterns and the Role of Prompt Design in LLM-Generated Codecited by 1
  23. 2026Unlocking LLM Code Correction with Iterative Feedback Loopscited by 1
  24. 2025Evaluating the Limitations of Local LLMs in Solving Complex Programming Challengescited by 0
  25. 2025MEC3O: Multi-Expert Consensus for Code Time Complexity Predictioncited by 0
  26. 2025Medical Interpretability and Knowledge Maps of Large Language Modelscited by 0
  27. 2025Do AI Models Dream of Faster Code? An Empirical Study on LLM-Proposed Performance Improvements in Real-World Softwarecited by 0
  28. 2025Usando LLMs para Programar Jogos de Tabuleiro e Variacc~oescited by 0
  29. 2025LLM4SFC: Sequential Function Chart Generation via Large Language Modelscited by 0
  30. 2025Holistic Evaluation of State-of-the-Art LLMs for Code Generationcited by 0
  31. 2026RACC: Representation-Aware Coverage Criteria for LLM Safety Testingcited by 0
  32. 2026Rethinking Code Performance Benchmarks for LLMscited by 0
  33. 2026Quantize with Confidence? An Empirical Study of Quantization for Code Generationcited by 0
  34. 2026Studying quantization trade-offs for efficient inference deployment in machine translationcited by 0
  35. 2026Memorization Diagnostics for Code LLMs Should be Scale-Awarecited by 0
  36. 2026Towards Fully Automated Medical Imaging Code Generation via Validation-based Context Engineeringcited by 0