# Rethinking Repetition Problems of LLMs in Code Generation

arXiv: https://arxiv.org/abs/2505.10402 · HTML: https://mlsanity.com/paper/2505.10402/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 1 of its 47 references that are in the map.
- First posted on arXiv 15 May 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 1 of 1)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)

## Built on by (most cited first, 6 of 6)

- [Saber: An Efficient Sampling with Adaptive Acceleration and Backtracking Enhanced Remasking for Diffusion Language Model](https://mlsanity.com/paper/2510.18165/) (2025)
- [Lookahead-then-Verify: Reliable Constrained Decoding for Diffusion LLMs under Context-Free Grammars](https://mlsanity.com/paper/2602.00612/) (2026)
- [CodeReasoner: Enhancing the Code Reasoning Ability with Reinforcement Learning](https://mlsanity.com/paper/2507.17548/) (2025)
- [SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs](https://mlsanity.com/paper/2604.07737/) (2026)
- [Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansion](https://mlsanity.com/paper/2605.22579/) (2026)
- [Escaping Mode Collapse in LLM Generation via Geometric Regulation](https://mlsanity.com/paper/2605.00435/) (2026)

## Abstract (arXiv)

With the advent of neural language models, the performance of code generation has been significantly boosted. However, the problem of repetitions during the generation process continues to linger. Previous work has primarily focused on content repetition, which is merely a fraction of the broader repetition problem in code generation. A more prevalent and challenging problem is structural repetition. In structural repetition, the repeated code appears in various patterns but possesses a fixed structure, which can be inherently reflected in grammar. In this paper, we formally define structural repetition and propose an efficient decoding approach called RPG, which stands for Repetition Penalization based on Grammar, to alleviate the repetition problems in code generation for LLMs. Specifically, RPG first leverages grammar rules to identify repetition problems during code generation, and then strategically decays the likelihood of critical tokens that contribute to repetitions, thereby mitigating them in code generation. To facilitate this study, we construct a new dataset CodeRepetEval to comprehensively evaluate approaches for mitigating the repetition problems in code generation. Extensive experimental results demonstrate that RPG substantially outperforms the best-performing baselines on CodeRepetEval dataset as well as HumanEval and MBPP benchmarks, effectively reducing repetitions and enhancing the quality of generated code.