mlsanityData to 7 Oct 2026

Feedback Loops and Code Perturbations in LLM-based Software Engineering: A Case Study on a C-to-Rust Translation System

arXiv 2512.02567 ↗First posted 2 Dec 2025cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The advent of strong generative AI has a considerable impact on various software engineering tasks such as code repair, test generation, or language translation. While tools like GitHub Copilot are already in widespread use in interactive settings, automated approaches require a higher level of reliability before being usable in industrial practice. In this paper, we focus on three aspects that directly influence the quality of the results: a) the effect of automated feedback loops, b) the choice of Large Language Model (LLM), and c) the influence of behavior-preserving code changes. We study the effect of these three variables on an automated C-to-Rust translation system. Code translation from C to Rust is an attractive use case in industry due to Rust's safety guarantees. The translation system is based on a generate-and-check pattern, in which Rust code generated by the LLM is automatically checked for compilability and behavioral equivalence with the original C code. For negative checking results, the LLM is re-prompted in a feedback loop to repair its output. These checks also allow us to evaluate and compare the respective success rates of the translation system when varying the three variables. Our results show that without feedback loops LLM selection has a large effect on translation success. However, when the translation system uses feedback loops the differences across models diminish. We observe this for the average performance of the system as well as its robustness under code perturbations. Finally, we also identify that diversity provided by code perturbations can even result in improved system performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 48 references that are in the map. First posted on arXiv 2 Dec 2025.

Builds on

30 of 48 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2024Phi-4 Technical Reportcited by 675
  5. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  6. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  7. 2022Do Users Write More Insecure Code with AI Assistants?cited by 186
  8. 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 178
  9. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  10. 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 99
  11. 2018Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examplescited by 93
  12. 2024Automated Unit Test Improvement using Large Language Models at Metacited by 73
18 more
  1. 2021Bad Characters: Imperceptible NLP Attackscited by 69
  2. 2023On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilotcited by 64
  3. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  4. 2019On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Modelscited by 56
  5. 2022Code Translation with Compiler Representationscited by 42
  6. 2024Towards Translating Real-World Code with LLMs: A Study of Translating to Rustcited by 35
  7. 2025Evaluating Agent-based Program Repair at Googlecited by 35
  8. 2024How much does AI impact development speed? An enterprise-based randomized controlled trialcited by 19
  9. 2021The King is Naked: on the Notion of Robustness for Natural Language Processingcited by 16
  10. 2021Challenging Machine Learning-based Clone Detectors via Semantic-preserving Code Transformationscited by 15
  11. 2023Classification robustness to common optical aberrationscited by 13
  12. 2024Scalable, Validated Code Translation of Entire Projects using Large Language Modelscited by 12
  13. 2022Robust Natural Language Processing: Recent Advances, Challenges, and Future Directionscited by 9
  14. 2024Assured LLM-Based Software Engineeringcited by 5
  15. 2020Evaluating Neural Machine Comprehension Model Robustness to Noisy Inputs and Adversarial Attackscited by 3
  16. 2023Enhancing Robustness of AI Offensive Code Generators via Data Augmentationcited by 3
  17. 2022Robustness and invariance properties of image classifierscited by 1
  18. 2023Formal Runtime Error Detection During Development in the Automotive Industrycited by 1

Built on by

Not cited yet.