# Better Together? An Evaluation of AI-Supported Code Translation

arXiv: https://arxiv.org/abs/2202.07682 · HTML: https://mlsanity.com/paper/2202.07682/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Feb 2022.

## Built on by (most cited first, 15 of 15)

- [CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation](https://mlsanity.com/paper/2310.04951/) (2023)
- [Design Principles for Generative AI Applications](https://mlsanity.com/paper/2401.14484/) (2024)
- [Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward](https://mlsanity.com/paper/2406.07765/) (2024)
- [Code Translation with Compiler Representations](https://mlsanity.com/paper/2207.03578/) (2022)
- [Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice](https://mlsanity.com/paper/2404.14901/) (2024)
- [Toward General Design Principles for Generative AI Applications](https://mlsanity.com/paper/2301.05578/) (2023)
- [Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools](https://mlsanity.com/paper/2412.14732/) (2024)
- [Rocks Coding, Not Development--A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks](https://mlsanity.com/paper/2402.05650/) (2024)
- [The Impact of Prompt Programming on Function-Level Code Generation](https://mlsanity.com/paper/2412.20545/) (2024)
- [Human-AI Experience in Integrated Development Environments: A Systematic Literature Review](https://mlsanity.com/paper/2503.06195/) (2025)
- [A Joint Learning Model with Variational Interaction for Multilingual Program Translation](https://mlsanity.com/paper/2408.14515/) (2024)
- [The Impact of LLM-Assistants on Software Developer Productivity: A Systematic Review and Mapping Study](https://mlsanity.com/paper/2507.03156/) (2025)
- [Function-to-Style Guidance of LLMs for Code Translation](https://mlsanity.com/paper/2507.11083/) (2025)
- [Advancing GenAI Assisted Programming--A Comparative Study on Prompt Efficiency and Code Quality Between GPT-4 and GLM-4](https://mlsanity.com/paper/2402.12782/) (2024)
- [ACT: Bridging the Gap in Code Translation through Synthetic Data Generation & Adaptive Training](https://mlsanity.com/paper/2507.16478/) (2025)

## Abstract (arXiv)

Generative machine learning models have recently been applied to source code, for use cases including translating code between programming languages, creating documentation from code, and auto-completing methods. Yet, state-of-the-art models often produce code that is erroneous or incomplete. In a controlled study with 32 software engineers, we examined whether such imperfect outputs are helpful in the context of Java-to-Python code translation. When aided by the outputs of a code translation model, participants produced code with fewer errors than when working alone. We also examined how the quality and quantity of AI translations affected the work process and quality of outcomes, and observed that providing multiple translations had a larger impact on the translation process than varying the quality of provided translations. Our results tell a complex, nuanced story about the benefits of generative code models and the challenges software engineers face when working with their outputs. Our work motivates the need for intelligent user interfaces that help software engineers effectively work with generative code models in order to understand and evaluate their outputs and achieve superior outcomes to working alone.