mlsanityData to 14 Oct 2026

CODEMENV: Benchmarking Large Language Models on Code Migration

arXiv 2506.00894 ↗First posted 1 Jun 2025cs.SE cs.AI cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have shown remarkable capabilities across various software engineering tasks; however, their effectiveness in code migration, adapting code to run in different environments, remains insufficiently studied. In this work, we introduce CODEMENV: Code Migration Across Environment, a new benchmark specifically designed to assess LLMs' abilities in code migration scenarios. CODEMENV consists of 922 examples spanning 19 Python and Java packages, and covers three core tasks: (1) identifying functions incompatible with specific versions, (2) detecting changes in function definitions, and (3) adapting code to target environments. Experimental evaluation with seven LLMs on CODEMENV yields an average pass@1 rate of 26.50%, with GPT-4O achieving the highest score at 43.84%. Key findings include: (i) LLMs tend to be more proficient with newer function versions, which aids in migrating legacy code, and (ii) LLMs sometimes exhibit logical inconsistencies by identifying function changes irrelevant to the intended migration environment. The datasets are available at https://github.com/xdshen-ai/Benchmark-of-Code-Migration.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 23 of its 33 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

23 of 33 references.

References · 23

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  8. 2022Mass-Editing Memory in a Transformercited by 1,001
  9. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  10. 2021Measuring Coding Challenge Competence With APPScited by 902
  11. 2024A Survey on Large Language Models for Code Generationcited by 658
  12. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
11 more
  1. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 300
  2. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 246
  3. 2022DocPrompting: Generating Code by Retrieving the Docscited by 153
  4. 2024Locate-then-edit for Multi-hop Factual Recall under Knowledge Editingcited by 37
  5. 2024Multi-hop Question Answering under Temporal Knowledge Editingcited by 36
  6. 2024Towards Translating Real-World Code with LLMs: A Study of Translating to Rustcited by 35
  7. 2024EVOR: Evolving Retrieval for Code Generationcited by 22
  8. 2025Migrating Code At Scale With LLMs At Googlecited by 14
  9. 2024Model Editing for LLMs4Code: How Far are We?cited by 10
  10. 2024Automatic Library Migration Using Large Language Models: First Resultscited by 9
  11. 2024TransAgent: Enhancing LLM-Based Code Translation via Fine-Grained Execution Alignmentcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 8.

13 papers.

Sort:

By area:

  1. 2025Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMscited by 21
  2. 2026Predicting LLM Output Length via Entropy-Guided Representationscited by 9
  3. 2025The Compositional Architecture of Regret in Large Language Modelscited by 3
  4. 2025GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilitiescited by 2
  5. 2026Environment-in-the-Loop: Rethinking Code Migration with LLM-based Agentscited by 2
  6. 2026Evaluating Language Models on Cross-Language Code Functional Equivalencecited by 2
  7. 2025Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Imagescited by 1
  8. 2025BackportBench: A Multilingual Benchmark for Automated Backporting of Patchescited by 1
  9. 2026TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Taskscited by 1
  10. 2026RepoRescue: An Empirical Study of LLM Agents on Whole-Repository Compatibility Rescuecited by 1
  11. 2026Toward Executable Repository-Level Code Generation via Environment Alignmentcited by 0
  12. 2026Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalencecited by 0
1 more
  1. 2026ORCA: Evaluating LLMs on Data Science Code Translationcited by 0