# From Output to Evaluation: Does Raw Instruction-Tuned Code LLMs Output Suffice for Fill-in-the-Middle Code Generation?

arXiv: https://arxiv.org/abs/2505.18789 · HTML: https://mlsanity.com/paper/2505.18789/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 11 references that are in the map.
- First posted on arXiv 24 May 2025.

## Builds on (most cited first, 8 of 8)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [Efficient Training of Language Models to Fill in the Middle](https://mlsanity.com/paper/2207.14255/) (2022)
- [SelfCodeAlign: Self-Alignment for Code Generation](https://mlsanity.com/paper/2410.24198/) (2024)
- [Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks](https://mlsanity.com/paper/2403.04814/) (2024)
- [Meet in the Middle: A New Pre-training Paradigm](https://mlsanity.com/paper/2303.07295/) (2023)
- [Planning-Aware Code Infilling via Horizon-Length Prediction](https://mlsanity.com/paper/2410.03103/) (2024)
- [Improving FIM Code Completions via Context & Curriculum Based Learning](https://mlsanity.com/paper/2412.16589/) (2024)

## Built on by (most cited first, 1 of 1)

- [Plan-and-Patch: Diffusion Language Models for Agentic Planning](https://mlsanity.com/paper/2610.10786/) (2026)

## Abstract (arXiv)

Post-processing is crucial for the automatic evaluation of LLMs in fill-in-the-middle (FIM) code generation due to the frequent presence of extraneous code in raw outputs. This extraneous generation suggests a lack of awareness regarding output boundaries, requiring truncation for effective evaluation. The determination of an optimal truncation strategy, however, often proves intricate, particularly when the scope includes several programming languages. This study investigates the necessity of post-processing instruction-tuned LLM outputs. Our findings reveal that supervised fine-tuning significantly enhances FIM code generation, enabling LLMs to generate code that seamlessly integrates with the surrounding context. Evaluating our fine-tuned textttQwen2.5-Coder (base and instruct) models on HumanEval Infilling and SAFIM benchmarks demonstrates improved performances without post-processing, especially when the middle consist of complete lines. However, post-processing of the LLM outputs remains necessary when the middle is a random span of code.