mlsanityData to 14 Oct 2026

CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation

arXiv 2310.04951 ↗First posted 8 Oct 2023cs.AI cs.PLcited by 57

Not on the map. Why?

Abstract (arXiv)

Recent code translation techniques exploit neural machine translation models to translate source code from one programming language to another to satisfy production compatibility or to improve efficiency of codebase maintenance. Most existing code translation datasets only focus on a single pair of popular programming languages. To advance research on code translation and meet diverse requirements of real-world applications, we construct CodeTransOcean, a large-scale comprehensive benchmark that supports the largest variety of programming languages for code translation. CodeTransOcean consists of three novel multilingual datasets, namely, MultilingualTrans supporting translations between multiple popular programming languages, NicheTrans for translating between niche programming languages and popular ones, and LLMTrans for evaluating executability of translated code by large language models (LLMs). CodeTransOcean also includes a novel cross-framework dataset, DLTrans, for translating deep learning code across different frameworks. We develop multilingual modeling approaches for code translation and demonstrate their great potential in improving the translation quality of both low-resource and high-resource language pairs and boosting the training efficiency. We also propose a novel evaluation metric Debugging Success Rate@K for program-level code translation. Last but not least, we evaluate LLM ChatGPT on our datasets and investigate its potential for fuzzy execution predictions. We build baselines for CodeTransOcean and analyze challenges of code translation for guiding future research. The CodeTransOcean datasets and code are publicly available at https://github.com/WeixiangYAN/CodeTransOcean.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Builds on 50 of its 78 references that are in the map. First posted on arXiv 8 Oct 2023.

Builds on

50 of 78 references.

References · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  8. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  9. 2023Teaching Large Language Models to Self-Debugcited by 988
  10. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  11. 2021Measuring Coding Challenge Competence With APPScited by 902
  12. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
38 more
  1. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  2. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 609
  3. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  4. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
  5. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  6. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  7. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  8. 2023Language Models can Solve Computer Taskscited by 372
  9. 2021Unified Pre-training for Program Understanding and Generationcited by 366
  10. 2023GPTScore: Evaluate as You Desirecited by 361
  11. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  12. 2020Unsupervised Translation of Programming Languagescited by 263
  13. 2019Massively Multilingual Neural Machine Translationcited by 246
  14. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 200
  15. 2016Transfer Learning for Low-Resource Neural Machine Translationcited by 188
  16. 2019SPoC: Search-based Pseudocode to Codecited by 176
  17. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 176
  18. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 160
  19. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  20. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  21. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  22. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 102
  23. 2018Tree-to-tree Neural Networks for Program Translationcited by 93
  24. 2021Leveraging Automated Unit Tests for Unsupervised Code Translationcited by 82
  25. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 76
  26. 2023CodeScore: Evaluating Code Generation by Learning Code Executioncited by 67
  27. 2023Large Language Models are Diverse Role-Players for Summarization Evaluationcited by 62
  28. 2021AVATAR: A Parallel Corpus for Java-Python Program Translationcited by 59
  29. 2017Transfer Learning across Low-Resource, Related Languages for Neural Machine Translationcited by 58
  30. 2022XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligencecited by 56
  31. 2022NatGen: Generative pre-training by "Naturalizing" source codecited by 54
  32. 2023DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agentscited by 52
  33. 2019Does BLEU Score Work for Code Migration?cited by 49
  34. 2020Multi-task Learning for Multilingual Neural Machine Translationcited by 46
  35. 2021Perfection Not Required? Human-AI Partnerships in Code Translationcited by 42
  36. 2023Exploring ChatGPT's Ability to Rank Content: A Preliminary Study on Consistency with Human Preferencescited by 25
  37. 2022AixBench: A Code Generation Benchmark Datasetcited by 22
  38. 2022Better Together? An Evaluation of AI-Supported Code Translationcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 13; 2025: 27; 2026: 14.

57 papers.

Sort:

By area:

  1. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 130
  2. 2025Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineeringcited by 67
  3. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  4. 2024CoIR: A Comprehensive Benchmark for Code Information Retrieval Modelscited by 60
  5. 2024MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problemscited by 40
  6. 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39
  7. 2025LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judgecited by 37
  8. 2024CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Executioncited by 31
  9. 2024CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?cited by 30
  10. 2024AlphaTrans: A Neuro-Symbolic Compositional Approach for Repository-Level Code Translation and Validationcited by 28
  11. 2024RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translationcited by 23
  12. 2025SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Mannercited by 20
38 more
  1. 2024Divide-and-Conquer Meets Consensus: Unleashing the Power of Functions in Code Generationcited by 19
  2. 2025CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generationcited by 19
  3. 2024UniCoder: Scaling Code Large Language Model via Universal Codecited by 18
  4. 2024VersiCode: Towards Version-controllable Code Generationcited by 17
  5. 2024AcTracer: Active Testing of Large Language Model via Multi-Stage Samplingcited by 17
  6. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  7. 2023LLM-SQL-Solver: Can LLMs Determine SQL Equivalence?cited by 10
  8. 2025LLM-Assisted Translation of Legacy FORTRAN Codes to C++: A Cross-Platform Studycited by 7
  9. 2026AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completioncited by 7
  10. 2026CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Codingcited by 7
  11. 2025Knowledge Augmented Complex Problem Solving with Large Language Models: A Surveycited by 6
  12. 2025REFINE: Enhancing Program Repair Agents through Context-Aware Patch Refinementcited by 6
  13. 2025Turning the Tide: Repository-based Code Reflectioncited by 5
  14. 2025IFEvalCode: Controlled Code Generationcited by 5
  15. 2024A Joint Learning Model with Variational Interaction for Multilingual Program Translationcited by 4
  16. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 4
  17. 2025Function-to-Style Guidance of LLMs for Code Translationcited by 4
  18. 2025Skeleton-Guided-Translation: A Benchmarking Framework for Code Repository Translation with Fine-Grained Quality Evaluationcited by 3
  19. 2025Fully Autonomous Programming using Iterative Multi-Agent Debugging with Large Language Modelscited by 3
  20. 2025StarFlow: Generating Structured Workflow Outputs From Sketch Imagescited by 3
  21. 2025CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programmingcited by 3
  22. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  23. 2025Project-Level C-to-Rust Translation via Pointer Knowledge Graphscited by 3
  24. 2025C2LLM Technical Report: A New Frontier in Code Retrieval via Adaptive Cross-Attention Poolingcited by 3
  25. 2026DRAINCODE: Stealthy Energy Consumption Attacks on Retrieval-Augmented Code Generation via Context Poisoningcited by 3
  26. 2025MigGPT: Harnessing Large Language Models for Automated Migration of Out-of-Tree Linux Kernel Patches Across Versionscited by 2
  27. 2026ReCodeAgent: A Multi-agent Workflow for Language-Agnostic Translation and Validation of Large-Scale Repositoriescited by 2
  28. 2026Large Language Models for Multilingual Code Intelligence: A Surveycited by 2
  29. 2026PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agentscited by 2
  30. 2024SPDZCoder: Combining Expert Knowledge with LLMs for Generating Privacy-Computing Codecited by 1
  31. 2025A Comparative Study of Specialized LLMs as Dense Retrieverscited by 1
  32. 2025When Many-Shot Prompting Fails: An Empirical Study of LLM Code Translationcited by 1
  33. 2025M2G-Eval: Enhancing and Evaluating Multi-granularity Multilingual Code Generationcited by 1
  34. 2026ITPEval: Benchmarking Formal Translation Across Interactive Theorem Proverscited by 1
  35. 2025A Pilot Study on LLM-Based Agentic Translation from Android to iOS: Pitfalls and Insightscited by 0
  36. 2025Learning Bug Context for PyTorch-to-JAX Translation with LLMscited by 0
  37. 2025Automated Snippet-Alignment Data Augmentation for Code Translationcited by 0
  38. 2025SoK: Understanding (New) Security Issues Across AI4Code Use Casescited by 0

50 most cited of 57 All 57 (JSON)