# RustRepoTrans: Repository-level Code Translation Benchmark Targeting Rust

arXiv: https://arxiv.org/abs/2411.13990 · HTML: https://mlsanity.com/paper/2411.13990/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2024.

## Built on by (most cited first, 6 of 6)

- [EffiReasonTrans: RL-Optimized Reasoning for Code Translation](https://mlsanity.com/paper/2510.18863/) (2025)
- [MatchFixAgent: Language-Agnostic Autonomous Repository-Level Code Translation Validation and Repair](https://mlsanity.com/paper/2509.16187/) (2025)
- [Documentation-Guided Agentic Codebase Migration from C to Rust](https://mlsanity.com/paper/2605.14634/) (2026)
- [Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence](https://mlsanity.com/paper/2605.29054/) (2026)
- [Translator vs. Challenger: Adversarial Agentic Learning for C-to-Rust Translation](https://mlsanity.com/paper/2609.15381/) (2026)
- [ORCA: Evaluating LLMs on Data Science Code Translation](https://mlsanity.com/paper/2609.30749/) (2026)

## Abstract (arXiv)

Recent advancements in large language models (LLMs) have demonstrated impressive capabilities in code translation, typically evaluated using benchmarks like CodeTransOcean and RepoTransBench. However, dependency-free benchmarks fail to capture real-world complexities by focusing primarily on simple function-level translations and overlooking repository-level context (e.g., dependencies). Full-repository translation benchmarks significantly exceed the current capabilities of existing models, resulting in performance bottlenecks that fail to provide actionable insights for guiding model development. Furthermore, existing benchmarks do not account for the scenario of incrementally translating new or modified modules from the source to the target language, which demands careful handling of repository-level contexts such as dependencies, cross-module references, and architectural divergence. Moreover, LLMs' effectiveness in translating to newer, low-resource languages like Rust remains largely underexplored. To address these gaps, we introduce RustRepoTrans, the first repository-level context code translation benchmark targeting incremental translation, comprising 375 tasks translating into Rust from C, Java, and Python. Using this benchmark, we evaluate seven representative LLMs, analyzing their errors to assess limitations in complex translation scenarios. Among them, DeepSeek-R1 performs best with 51.5% Pass@1, excelling in both basic functionality and additional translation abilities, such as noise robustness and syntactical difference identification. However, even DeepSeek-R1 experiences a 22.2% performance drop (Pass@1 from 73.7% to 51.5%) when handling repository-level context compared to previous benchmarks without such context.