# HumanEval on Latest GPT Models -- 2024

arXiv: https://arxiv.org/abs/2402.14852 · HTML: https://mlsanity.com/paper/2402.14852/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 13 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?](https://mlsanity.com/paper/2302.03495/) (2023)
- [Hierarchical Neural Program Synthesis](https://mlsanity.com/paper/2303.06018/) (2023)
- [Enhancing Robot Program Synthesis Through Environmental Context](https://mlsanity.com/paper/2312.08250/) (2023)

## Built on by (most cited first, 7 of 7)

- [SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference](https://mlsanity.com/paper/2510.10302/) (2025)
- [Breaking Model Lock-in: Cost-Efficient Zero-Shot LLM Routing via a Universal Latent Space](https://mlsanity.com/paper/2601.06220/) (2026)
- [From GPT-3 to GPT-5: Mapping their capabilities, scope, limitations, and consequences](https://mlsanity.com/paper/2604.10332/) (2026)
- [AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment](https://mlsanity.com/paper/2510.22593/) (2025)
- [EnterpriseLab: A Full-Stack Platform for developing and deploying agents in Enterprises](https://mlsanity.com/paper/2603.21630/) (2026)
- [A-ProS: Towards Reliable Autonomous Programming Through Multi-Model Feedback](https://mlsanity.com/paper/2605.18073/) (2026)
- [AutoGEEval: A Multimodal and Automated Framework for Geospatial Code Generation on GEE with Large Language Models](https://mlsanity.com/paper/2505.12900/) (2025)

## Abstract (arXiv)

In 2023, we are using the latest models of GPT-4 to advance program synthesis. The large language models have significantly improved the state-of-the-art for this purpose. To make these advancements more accessible, we have created a repository that connects these models to Huamn Eval. This dataset was initally developed to be used with a language model called CODEGEN on natural and programming language data. The utility of these trained models is showcased by demonstrating their competitive performance in zero-shot Python code generation on HumanEval tasks compared to previous state-of-the-art solutions. Additionally, this gives way to developing more multi-step paradigm synthesis. This benchmark features 160 diverse problem sets factorized into multistep prompts that our analysis shows significantly improves program synthesis over single-turn inputs. All code is open source at https://github.com/daniel442li/gpt-human-eval .