# When to Stop? Towards Efficient Code Generation in LLMs with Excess Token Prevention

arXiv: https://arxiv.org/abs/2407.20042 · HTML: https://mlsanity.com/paper/2407.20042/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jul 2024.

## Built on by (most cited first, 21 of 21)

- [LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation](https://mlsanity.com/paper/2409.20550/) (2024)
- [Agents in Software Engineering: Survey, Landscape, and Vision](https://mlsanity.com/paper/2409.09030/) (2024)
- [What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyond](https://mlsanity.com/paper/2503.20589/) (2025)
- [An Empirical Study of Agent Developer Practices in AI Agent Frameworks](https://mlsanity.com/paper/2512.01939/) (2025)
- [Marking Code Without Breaking It: Code Watermarking for Detecting LLM-Generated Code](https://mlsanity.com/paper/2502.18851/) (2025)
- [Towards an Understanding of Context Utilization in Code Intelligence](https://mlsanity.com/paper/2504.08734/) (2025)
- [AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completion](https://mlsanity.com/paper/2601.19697/) (2026)
- [EfficientUICoder: A Bidirectional Token Compression Framework for Efficient MLLM-Based UI Code Generation](https://mlsanity.com/paper/2509.12159/) (2025)
- [Evaluating Quantized Large Language Models for Code Generation on Low-Resource Language Benchmarks](https://mlsanity.com/paper/2410.14766/) (2024)
- [Integrating Expert Knowledge into Logical Programs via LLMs](https://mlsanity.com/paper/2502.12275/) (2025)
- [Breaking the Loop: Detecting and Mitigating Denial-of-Service Vulnerabilities in Large Language Models](https://mlsanity.com/paper/2503.00416/) (2025)
- [Green Prompting: Characterizing Prompt-driven Energy Costs of LLM Inference](https://mlsanity.com/paper/2503.10666/) (2025)
- [ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generation](https://mlsanity.com/paper/2601.09703/) (2026)
- [DRAINCODE: Stealthy Energy Consumption Attacks on Retrieval-Augmented Code Generation via Context Poisoning](https://mlsanity.com/paper/2601.20615/) (2026)
- [Optimizing Token Consumption in LLMs: A Nano Surge Approach for Code Reasoning Efficiency](https://mlsanity.com/paper/2504.15989/) (2025)
- [PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agents](https://mlsanity.com/paper/2607.18859/) (2026)
- [SolAgent: A Specialized Multi-Agent Framework for Solidity Code Generation](https://mlsanity.com/paper/2601.23009/) (2026)
- [Benchmarking Web API Integration Code Generation](https://mlsanity.com/paper/2509.20172/) (2025)
- [RRT*former: Environment-Aware Sampling-Based Motion Planning using Transformer](https://mlsanity.com/paper/2511.15414/) (2025)
- [HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding](https://mlsanity.com/paper/2607.22389/) (2026)
- [Adaptive Critical Token-Aware Retrieval for Repository-Level Code Generation](https://mlsanity.com/paper/2609.01601/) (2026)

## Abstract (arXiv)

Code generation aims to automatically generate code snippets that meet given natural language requirements and plays an important role in software development. Although Code LLMs have shown excellent performance in this domain, their long generation time poses a signification limitation in practice use. In this paper, we first conduct an in-depth preliminary study with different Code LLMs on code generation tasks and identify a significant efficiency issue, i.e., continual generation of excess tokens. It harms the developer productivity and leads to huge computational wastes. To address it, we introduce CodeFast, an inference acceleration approach for Code LLMs on code generation. The key idea of CodeFast is to terminate the inference process in time when unnecessary excess tokens are detected. First, we propose an automatic data construction framework to obtain training data. Then, we train a unified lightweight model GenGuard applicable to multiple programming languages to predict whether to terminate inference at the current step. Finally, we enhance Code LLM with GenGuard to accelerate its inference in code generation tasks. We conduct extensive experiments with CodeFast on five representative Code LLMs across four widely used code generation datasets. Experimental results show that (1) CodeFast can significantly improve the inference speed of various Code LLMs in code generation, ranging form 34% to 452%, without compromising the quality of generated code. (2) CodeFast is stable across different parameter settings and can generalize to untrained datasets. Our code and data are available at https://github.com/DeepSoftwareAnalytics/CodeFast