mlsanityData to 7 Oct 2026

NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts

arXiv 2405.04520 ↗First posted 7 May 2024cs.CL cs.LG cs.SEcited by 19

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have manifested strong ability to generate codes for productive activities. However, current benchmarks for code synthesis, such as HumanEval, MBPP, and DS-1000, are predominantly oriented towards introductory tasks on algorithm and data science, insufficiently satisfying challenging requirements prevalent in real-world coding. To fill this gap, we propose NaturalCodeBench (NCB), a challenging code benchmark designed to mirror the complexity and variety of scenarios in real coding tasks. NCB comprises 402 high-quality problems in Python and Java, meticulously selected from natural user queries from online coding services, covering 6 different domains. Noting the extraordinary difficulty in creating testing cases for real-world queries, we also introduce a semi-automated pipeline to enhance the efficiency of test case construction. Comparing with manual solutions, it achieves an efficiency increase of more than 4 times. Our systematic experiments on 39 LLMs find that performance gaps on NCB between models with close HumanEval scores could still be significant, indicating a lack of focus on practical code synthesis scenarios or over-specified optimization on HumanEval. On the other hand, even the best-performing GPT-4 is still far from satisfying on NCB. The evaluation toolkit and development set are available at https://github.com/THUDM/NaturalCodeBench.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 44 of its 54 references that are in the map. First posted on arXiv 7 May 2024.

Builds on

44 of 54 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  11. 2023Mistral 7Bcited by 3,065
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,105
32 more
  1. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  2. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  3. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  4. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  5. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  7. 2021Measuring Coding Challenge Competence With APPScited by 879
  8. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  9. 2023StarCoder: may the source be with you!cited by 790
  10. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  11. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  12. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  13. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  14. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  15. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  16. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  17. 2020Unsupervised Translation of Programming Languagescited by 262
  18. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  19. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  20. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  21. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 150
  22. 2022Practical Program Repair in the Era of Large Pre-trained Language Modelscited by 133
  23. 2022Few-shot training LLMs for project-specific code-summarizationcited by 128
  24. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 124
  25. 2022Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Modelscited by 102
  26. 2023Impact of Code Language Models on Automated Program Repaircited by 85
  27. 2021Leveraging Automated Unit Tests for Unsupervised Code Translationcited by 82
  28. 2022Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learningcited by 76
  29. 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
  30. 2023Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repaircited by 55
  31. 2019Does BLEU Score Work for Code Migration?cited by 49
  32. 2022Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testingcited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 8; 2026: 2.

19 papers.

Sort:

By area:

  1. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  2. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 173
  3. 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 99
  4. 2024AutoGLM: Autonomous Foundation Agents for GUIscited by 96
  5. 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 66
  6. 2025DataSciBench: An LLM Agent Benchmark for Data Sciencecited by 53
  7. 2024The Limits of Inference Scaling Through Resamplingcited by 51
  8. 2025Seed-Coder: Let the Code Model Curate Data for Itselfcited by 46
  9. 2024Top General Performance = Top Domain Performance? DomainCodeBench: A Multi-domain Code Generation Benchmarkcited by 13
  10. 2026Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Modelcited by 12
  11. 2024FVEval: Understanding Language Model Capabilities in Formal Verification of Digital Hardwarecited by 9
  12. 2024CursorCore: Assist Programming through Aligning Anythingcited by 6
7 more
  1. 2024Programming with AI: Evaluating ChatGPT, Gemini, AlphaCode, and GitHub Copilot for Programmerscited by 5
  2. 2025IFEvalCode: Controlled Code Generationcited by 5
  3. 2026I-WebGenBench : Evaluating Interactivity in LLM-Generated Scientific Web Applicationscited by 3
  4. 2024Diagnosing and Remedying Knowledge Deficiencies in LLMs via Label-free Curricular Meaningful Learningcited by 2
  5. 2024SPDZCoder: Combining Expert Knowledge with LLMs for Generating Privacy-Computing Codecited by 1
  6. 2025CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluationcited by 1
  7. 2025MateInfoUB: A Real-World Benchmark for Testing LLMs in Competitive, Multilingual, and Multimodal Educational Taskscited by 1