From Translation to Superset: Benchmark-Driven Evolution of a Production AI Agent from Rust to Python
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Cross-language migration of large software systems is a persistent engineering challenge, particularly when the source codebase evolves rapidly. We present a methodology for LLM-assisted continuous code translation in which a large language model translates a production Rust codebase (648K LOC, 65 crates) into Python (41K LOC, 28 modules), with public agent benchmarks as the objective function driving iterative refinement. Our subject system is Codex CLI, a production AI coding agent. We demonstrate that: (1) the Python port resolves 59/80 SWE-bench Verified tasks (73.8%) versus Rust's 56/80 (70.0%), and achieves 42.5% on Terminal-Bench versus Rust's 47.5%, confirming near-parity on real-world agentic tasks; (2) benchmark-driven debugging, revealing API protocol mismatches, environment pollution, a silent WebSocket failure mode, and an API 400 crash, is more effective than static testing alone; (3) the architecture supports continuous upstream synchronisation via an LLM-assisted diff-translate-test loop; and (4) the Python port has evolved into a capability superset with 30 feature-flagged extensions (multi-agent orchestration, semantic memory, guardian safety, cost tracking) absent from Rust, while preserving strict parity mode for comparison. Our evaluation shows that for LLM-based agents where API latency dominates, Python's expressiveness yields a 15.9x code reduction with negligible performance cost, while the benchmark-as-objective-function methodology provides a principled framework for growing a cross-language port from parity into an extended platform.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 6 of its 17 references that are in the map. First posted on arXiv 13 Apr 2026.
Builds on
6 of 17 references.
Closest ancestors · 2
Same direction, newest first
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May
- 2023ChatDev: Communicative Agents for Software Development16 Jul
Other references · 4
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2018An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translationcited by 106
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026HARBOR: Automated Harness Optimizationcited by 9
- 2026HARBOR: Automated Harness Optimization22 Apr 2026
Similar, not cited
Close in wording, no citation either way
- 2024MOSS: Enabling Code-Driven Evolution and Context Management for AI Agents
- 2026Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation
- 2025FreshBrew: A Benchmark for Evaluating AI Agents on Java Code Migration
- 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
- 2026ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development
- 2026AgentFactory: A Self-Evolving Framework Through Executable Subagent Accumulation and Reuse
- 2025NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents
- 2026RepoMod-Bench: A Benchmark for Code Repository Modernization via Implementation-Agnostic Testing