mlsanityData to 14 Oct 2026

MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation

arXiv 2405.11430 ↗First posted 19 May 2024cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

Recent advancements in large language models (LLMs) have greatly improved code generation, specifically at the function level. For instance, GPT-4o has achieved a 91.0% pass rate on HumanEval. However, this draws into question the adequacy of existing benchmarks in thoroughly assessing function-level code generation capabilities. Our study analyzed two common benchmarks, HumanEval and MBPP, and found that these might not thoroughly evaluate LLMs' code generation capacities due to limitations in quality, difficulty, and granularity. To resolve this, we introduce the Mostly Hard Python Problems (MHPP) dataset, consisting of 210 unique human-curated problems. By focusing on the combination of natural language and code reasoning, MHPP gauges LLMs' abilities to comprehend specifications and restrictions, engage in multi-step reasoning, and apply coding knowledge effectively. Initial evaluations of 26 LLMs using MHPP showed many high-performing models on HumanEval failed to achieve similar success on MHPP. Moreover, MHPP highlighted various previously undiscovered limitations within various LLMs, leading us to believe that it could pave the way for a better understanding of LLMs' capabilities and limitations. MHPP, evaluation pipeline, and leaderboard can be found in https://github.com/SparksofAGI/MHPP.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 51 of its 71 references that are in the map. First posted on arXiv 19 May 2024.

Builds on

51 of 71 references.

References · 51

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  11. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  12. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
38 more
  1. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  2. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  3. 2021Measuring Coding Challenge Competence With APPScited by 902
  4. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  5. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  6. 2023Textbooks Are All You Needcited by 569
  7. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  8. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  9. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  10. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  11. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  12. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  13. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  14. 2020Unsupervised Translation of Programming Languagescited by 263
  15. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  16. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 194
  17. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  18. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  19. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 142
  20. 2022Few-shot training LLMs for project-specific code-summarizationcited by 129
  21. 2023InferFix: End-to-End Program Repair with LLMscited by 114
  22. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  23. 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
  24. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 103
  25. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 91
  26. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  27. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 87
  28. 2023Impact of Code Language Models on Automated Program Repaircited by 85
  29. 2024Instruction Tuning for Secure Code Generationcited by 68
  30. 2022CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Contextcited by 65
  31. 2021AVATAR: A Parallel Corpus for Java-Python Program Translationcited by 59
  32. 2019JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generationcited by 49
  33. 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39
  34. 2023Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPTcited by 36
  35. 2022Training and Evaluating a Jupyter Notebook Data Science Assistantcited by 31
  36. 2021CoDesc: A Large Code-Description Parallel Datasetcited by 23
  37. 2021Type4Py: Practical Deep Similarity Learning-Based Type Inference for Pythoncited by 19
  38. 2023CodeFuse-13B: A Pretrained Multi-lingual Code Large Language Modelcited by 14

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 9; 2026: 10.

26 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  3. 2024Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Surveycited by 83
  4. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 53
  5. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Searchcited by 53
  6. 2025Seed-Coder: Let the Code Model Curate Data for Itselfcited by 49
  7. 2025Uncertainty-Guided Chain-of-Thought for Code Generation with LLMscited by 32
  8. 2024EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimizationcited by 30
  9. 2024MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMscited by 30
  10. 2025EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Codecited by 27
  11. 2024EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuningcited by 17
  12. 2024CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharingcited by 12
14 more
  1. 2024Measuring the Influence of Incorrect Code on Test Generationcited by 11
  2. 2026CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Modelscited by 5
  3. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  4. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  5. 2026CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generationcited by 4
  6. 2025ChessArena: A Chess Testbed for Evaluating Strategic Reasoning Capabilities of Large Language Modelscited by 3
  7. 2026Exposing Weaknesses of Large Reasoning Models through Graph Algorithm Problemscited by 3
  8. 2025Understanding Chain-of-Thought Effectiveness in Code Generation: An Empirical and Information-Theoretic Analysiscited by 2
  9. 2026FireBench: Evaluating Instruction Following in Enterprise and API-Driven LLM Applicationscited by 2
  10. 2026Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languagescited by 1
  11. 2026IntentCoding: Amplifying User Intent in Code Generationcited by 0
  12. 2026Structural Anchors and Reasoning Fragility:Understanding CoT Robustness in LLM4Codecited by 0
  13. 2026From Dead Code and Static Requirements to Working Engines: Software Revival with Coding Agentscited by 0
  14. 2026CodeWalk: Generating Coding Benchmarks by Walking a Problem Graphcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CodeEval: A pedagogical approach for targeted evaluation of code-trained Large Language Models
  2. 2024Low-Cost Language Models: Survey and Performance Evaluation on Python Code Generation
  3. 2024HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation
  4. 2023L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models
  5. 2025Holistic Evaluation of State-of-the-Art LLMs for Code Generation
  6. 2026Evaluating LLMs Code Reasoning Under Real-World Context
  7. 2025Prompt engineering and framework: implementation to increase code reliability based guideline for LLMs
  8. 2026CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models