# The First Prompt Counts the Most! An Evaluation of Large Language Models on Iterative Example-Based Code Generation

arXiv: https://arxiv.org/abs/2411.06774 · HTML: https://mlsanity.com/paper/2411.06774/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2024.

## Built on by (most cited first, 7 of 7)

- [A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System](https://mlsanity.com/paper/2510.09721/) (2025)
- [The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss](https://mlsanity.com/paper/2512.08374/) (2025)
- [Programming by Examples Meets Historical Linguistics: A Large Language Model Based Approach to Sound Law Induction](https://mlsanity.com/paper/2501.16524/) (2025)
- [Routing, Cascades, and User Choice for LLMs](https://mlsanity.com/paper/2602.09902/) (2026)
- [From I/O to Code with Discovery Agent](https://mlsanity.com/paper/2605.15334/) (2026)
- [AutoPass: Evidence-Guided LLM Agents for Compiler Performance Tuning](https://mlsanity.com/paper/2606.20373/) (2026)
- [Do Uncertainty Signals Help? A Systematic Study of Uncertainty-Aware Decoding with Rollback Mechanisms](https://mlsanity.com/paper/2608.14653/) (2026)

## Abstract (arXiv)

The capabilities of Large Language Models (LLMs) in code generation have been extensively studied, particularly for implementing target functionalities from natural-language descriptions. Alternatively, input-output (I/O) examples provide an accessible, unambiguous, and flexible way to describe functionalities. However, their inherent diversity, opaqueness, and incompleteness impose greater challenges for understanding and implementing the target requirements. Therefore, generating code from I/O examples (i.e., example-based code generation) provides a new perspective, allowing us to additionally evaluate LLMs' capability to infer target functionalities from limited information and to process new-form requirements. However, related research about LLMs in example-based code generation remains largely unexplored. To fill this gap, this paper presents the first comprehensive study on example-based code generation using LLMs. We adopt an iterative evaluation framework and formalize the objective of example-based code generation as two sequential sub-objectives: generating code conforming to the given examples and generating code that successfully implements the target functionalities from (iteratively) given examples. We assess six state-of-the-art LLMs using a new benchmark of 172 diverse target functionalities. The results demonstrate that when requirements are described using iterative I/O examples rather than natural language, the LLMs' score decreases by over 60%, and the vast majority (even over 95%) of successfully implemented functionalities are achieved in the first round of the iterations. Furthermore, we also find that combining I/O examples with even imprecise and fragmental natural language descriptions greatly improves LLM performance, and the selection of initial I/O examples can also influence the score, suggesting opportunities for prompt optimization.