# ROCODE: Integrating Backtracking Mechanism and Program Analysis in Large Language Models for Code Generation

arXiv: https://arxiv.org/abs/2411.07112 · HTML: https://mlsanity.com/paper/2411.07112/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2024.

## Built on by (most cited first, 13 of 13)

- [AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities](https://mlsanity.com/paper/2508.11126/) (2025)
- [CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment](https://mlsanity.com/paper/2510.18471/) (2025)
- [Saber: An Efficient Sampling with Adaptive Acceleration and Backtracking Enhanced Remasking for Diffusion Language Model](https://mlsanity.com/paper/2510.18165/) (2025)
- [Multi-agent Collaboration with State Management](https://mlsanity.com/paper/2605.20563/) (2026)
- [Multi-Agent Systems for Dataset Adaptation in Software Engineering: Capabilities, Limitations, and Future Directions](https://mlsanity.com/paper/2511.21380/) (2025)
- [Knowledge-Graph-Driven Data Synthesis for Low-Resource Software Development: A HarmonyOS Case Study](https://mlsanity.com/paper/2512.00380/) (2025)
- [Autoregressive, Yet Revisable: In Decoding Revision for Secure Code Generation](https://mlsanity.com/paper/2602.01187/) (2026)
- [Coding with Eyes: Visual Feedback Unlocks Reliable GUI Code Generating and Debugging](https://mlsanity.com/paper/2604.19750/) (2026)
- [ConceptCoder: Improve Code Reasoning via Concept Learning](https://mlsanity.com/paper/2603.23470/) (2026)
- [Beyond the Autoregressive Horizon: A Comprehensive Survey of Diffusion Models, World Modelling, and State Space Models for Code](https://mlsanity.com/paper/2606.23690/) (2026)
- [ExeCRE: Execution-Consistency Guided Reliability Estimation for Self-Correcting Code Generation](https://mlsanity.com/paper/2608.04439/) (2026)
- [Do Uncertainty Signals Help? A Systematic Study of Uncertainty-Aware Decoding with Rollback Mechanisms](https://mlsanity.com/paper/2608.14653/) (2026)
- [SPROUT: Scaling Agentic Reinforcement Learning for Long-Horizon Software Engineering](https://mlsanity.com/paper/2610.14405/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have achieved impressive performance in code generation recently, offering programmers revolutionary assistance in software development. However, due to the auto-regressive nature of LLMs, they are susceptible to error accumulation during code generation. Once an error is produced, LLMs can merely continue to generate the subsequent code conditioned on it, given their inability to adjust previous outputs. Existing LLM-based approaches typically consider post-revising after code generation, leading to the challenging resolution of accumulated errors and the significant wastage of resources. Ideally, LLMs should rollback and resolve the occurred error in time during code generation, rather than proceed on the basis of the error and wait for post-revising after generation. In this paper, we propose ROCODE, which integrates the backtracking mechanism and program analysis into LLMs for code generation. Specifically, we employ program analysis to perform incremental error detection during the generation process. When an error is detected, the backtracking mechanism is triggered to priming rollback strategies and constraint regeneration, thereby eliminating the error early and ensuring continued generation on the correct basis. Experiments on multiple code generation benchmarks show that ROCODE can significantly reduce the errors generated by LLMs, with a compilation pass rate of 99.1%. The test pass rate is improved by up to 23.8% compared to the best baseline approach. Compared to the post-revising baseline, the token cost is reduced by 19.3%. Moreover, our approach is model-agnostic and achieves consistent improvements across nine representative LLMs.