# Skeleton-Guided-Translation: A Benchmarking Framework for Code Repository Translation with Fine-Grained Quality Evaluation

arXiv: https://arxiv.org/abs/2501.16050 · HTML: https://mlsanity.com/paper/2501.16050/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 27 Jan 2025.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Unsupervised Translation of Programming Languages](https://mlsanity.com/paper/2006.03511/) (2020)
- [CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks](https://mlsanity.com/paper/2105.12655/) (2021)
- [Multi-task Sequence to Sequence Learning](https://mlsanity.com/paper/1511.06114/) (2015)
- [Leveraging Automated Unit Tests for Unsupervised Code Translation](https://mlsanity.com/paper/2110.06773/) (2021)
- [AVATAR: A Parallel Corpus for Java-Python Program Translation](https://mlsanity.com/paper/2108.11590/) (2021)
- [CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation](https://mlsanity.com/paper/2310.04951/) (2023)
- [xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval](https://mlsanity.com/paper/2303.03004/) (2023)
- [Adversarial Robustness with Semi-Infinite Constrained Learning](https://mlsanity.com/paper/2110.15767/) (2021)
- [StructCoder: Structure-Aware Transformer for Code Generation](https://mlsanity.com/paper/2206.05239/) (2022)
- [On the Evaluation of Neural Code Translation: Taxonomy and Benchmark](https://mlsanity.com/paper/2308.08961/) (2023)
- [RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation](https://mlsanity.com/paper/2412.17744/) (2024)
- [CoTran: An LLM-based Code Translator using Reinforcement Learning with Feedback from Compiler and Symbolic Execution](https://mlsanity.com/paper/2306.06755/) (2023)
- [VERT: Verified Equivalent Rust Transpilation with Large Language Models as Few-Shot Learners](https://mlsanity.com/paper/2404.18852/) (2024)
- [Rectifier: Code Translation with Corrector via LLMs](https://mlsanity.com/paper/2407.07472/) (2024)
- [ContrastRepair: Enhancing Conversation-Based Automated Program Repair via Contrastive Test Case Pairs](https://mlsanity.com/paper/2403.01971/) (2024)
- [Unraveling the Potential of Large Language Models in Code Translation: How Far Are We?](https://mlsanity.com/paper/2410.09812/) (2024)
- [Enhancing Code Translation in Language Models with Few-Shot Learning via Retrieval-Augmented Generation](https://mlsanity.com/paper/2407.19619/) (2024)
- [Syzygy: Dual Code-Test C to (safe) Rust Translation using LLMs and Dynamic Analysis](https://mlsanity.com/paper/2412.14234/) (2024)
- [Explain-then-Translate: An Analysis on Improving Program Translation with Self-generated Explanations](https://mlsanity.com/paper/2311.07070/) (2023)
- [Automatically Testing Functional Properties of Code Translation Models](https://mlsanity.com/paper/2309.12813/) (2023)

## Built on by (most cited first, 3 of 3)

- [Unlocking LLM Repair Capabilities Through Cross-Language Translation and Multi-Agent Refinement](https://mlsanity.com/paper/2503.22512/) (2025)
- [Program Skeletons for Automated Program Translation](https://mlsanity.com/paper/2504.07483/) (2025)
- [ORCA: Evaluating LLMs on Data Science Code Translation](https://mlsanity.com/paper/2609.30749/) (2026)

## Abstract (arXiv)

The advancement of large language models has intensified the need to modernize enterprise applications and migrate legacy systems to secure, versatile languages. However, existing code translation benchmarks primarily focus on individual functions, overlooking the complexities involved in translating entire repositories, such as maintaining inter-module coherence and managing dependencies. While some recent repository-level translation benchmarks attempt to address these challenges, they still face limitations, including poor maintainability and overly coarse evaluation granularity, which make them less developer-friendly. We introduce Skeleton-Guided-Translation, a framework for repository-level Java to C# code translation with fine-grained quality evaluation. It uses a two-step process: first translating the repository's structural "skeletons", then translating the full repository guided by these skeletons. Building on this, we present TRANSREPO-BENCH, a benchmark of high quality open-source Java repositories and their corresponding C# skeletons, including matching unit tests and build configurations. Our unit tests are fixed and can be applied across multiple or incremental translations without manual adjustments, enhancing automation and scalability in evaluations. Additionally, we develop fine-grained evaluation metrics that assess translation quality at the individual test case level, addressing traditional binary metrics' inability to distinguish when build failures cause all tests to fail. Evaluations using TRANSREPO-BENCH highlight key challenges and advance more accurate repository level code translation.