# Feedback Loops and Code Perturbations in LLM-based Software Engineering: A Case Study on a C-to-Rust Translation System

arXiv: https://arxiv.org/abs/2512.02567 · HTML: https://mlsanity.com/paper/2512.02567/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 2 Dec 2025.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [TextBugger: Generating Adversarial Text Against Real-world Applications](https://mlsanity.com/paper/1812.05271/) (2018)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?](https://mlsanity.com/paper/2502.12115/) (2025)
- [ReCode: Robustness Evaluation of Code Generation Models](https://mlsanity.com/paper/2212.10264/) (2022)
- [Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples](https://mlsanity.com/paper/1803.01128/) (2018)
- [Automated Unit Test Improvement using Large Language Models at Meta](https://mlsanity.com/paper/2402.09171/) (2024)
- [Bad Characters: Imperceptible NLP Attacks](https://mlsanity.com/paper/2106.09898/) (2021)
- [On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot](https://mlsanity.com/paper/2302.00438/) (2023)
- [MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks](https://mlsanity.com/paper/2311.07463/) (2023)
- [On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models](https://mlsanity.com/paper/1903.06620/) (2019)
- [Code Translation with Compiler Representations](https://mlsanity.com/paper/2207.03578/) (2022)
- [Towards Translating Real-World Code with LLMs: A Study of Translating to Rust](https://mlsanity.com/paper/2405.11514/) (2024)
- [Evaluating Agent-based Program Repair at Google](https://mlsanity.com/paper/2501.07531/) (2025)
- [How much does AI impact development speed? An enterprise-based randomized controlled trial](https://mlsanity.com/paper/2410.12944/) (2024)
- [The King is Naked: on the Notion of Robustness for Natural Language Processing](https://mlsanity.com/paper/2112.07605/) (2021)
- [Challenging Machine Learning-based Clone Detectors via Semantic-preserving Code Transformations](https://mlsanity.com/paper/2111.10793/) (2021)
- [Classification robustness to common optical aberrations](https://mlsanity.com/paper/2308.15499/) (2023)
- [Scalable, Validated Code Translation of Entire Projects using Large Language Models](https://mlsanity.com/paper/2412.08035/) (2024)
- [Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2201.00768/) (2022)
- [Assured LLM-Based Software Engineering](https://mlsanity.com/paper/2402.04380/) (2024)
- [Evaluating Neural Machine Comprehension Model Robustness to Noisy Inputs and Adversarial Attacks](https://mlsanity.com/paper/2005.00190/) (2020)
- [Enhancing Robustness of AI Offensive Code Generators via Data Augmentation](https://mlsanity.com/paper/2306.05079/) (2023)
- [Robustness and invariance properties of image classifiers](https://mlsanity.com/paper/2209.02408/) (2022)
- [Formal Runtime Error Detection During Development in the Automotive Industry](https://mlsanity.com/paper/2310.16468/) (2023)

## Abstract (arXiv)

The advent of strong generative AI has a considerable impact on various software engineering tasks such as code repair, test generation, or language translation. While tools like GitHub Copilot are already in widespread use in interactive settings, automated approaches require a higher level of reliability before being usable in industrial practice. In this paper, we focus on three aspects that directly influence the quality of the results: a) the effect of automated feedback loops, b) the choice of Large Language Model (LLM), and c) the influence of behavior-preserving code changes. We study the effect of these three variables on an automated C-to-Rust translation system. Code translation from C to Rust is an attractive use case in industry due to Rust's safety guarantees. The translation system is based on a generate-and-check pattern, in which Rust code generated by the LLM is automatically checked for compilability and behavioral equivalence with the original C code. For negative checking results, the LLM is re-prompted in a feedback loop to repair its output. These checks also allow us to evaluate and compare the respective success rates of the translation system when varying the three variables. Our results show that without feedback loops LLM selection has a large effect on translation success. However, when the translation system uses feedback loops the differences across models diminish. We observe this for the average performance of the system as well as its robustness under code perturbations. Finally, we also identify that diversity provided by code perturbations can even result in improved system performance.