mlsanityData to 14 Oct 2026

TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks

arXiv 2601.22597 ↗First posted 30 Jan 2026cs.SE cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

With the advancement of automated software engineering, research focus is increasingly shifting toward practical tasks reflecting the day-to-day work of software engineers. Among these tasks, software migration, a critical process of adapting code to evolving environments, has been largely overlooked. In this study, we introduce TimeMachine-bench, a benchmark designed to evaluate software migration in real-world Python projects. Our benchmark consists of GitHub repositories whose tests begin to fail in response to dependency updates. The construction process is fully automated, enabling live updates of the benchmark. Furthermore, we curated a human-verified subset to ensure problem solvability. We evaluated agent-based baselines built on top of 11 models, including both strong open-weight and state-of-the-art LLMs on this verified subset. Our results indicated that, while LLMs show some promise for migration tasks, they continue to face substantial reliability challenges, including spurious solutions that exploit low test coverage and unnecessary edits stemming from suboptimal tool-use strategies. Our dataset and implementation are available at https://github.com/tohoku-nlp/timemachine-bench.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 55 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

22 of 55 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  6. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  7. 2023Teaching Large Language Models to Self-Debugcited by 988
  8. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  9. 2021Measuring Coding Challenge Competence With APPScited by 902
  10. 2024A Survey on Large Language Models for Code Generationcited by 658
  11. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  12. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
10 more
  1. 2023Software Testing with Large Language Models: Survey, Landscape, and Visioncited by 166
  2. 2022Automating Code Review Activities by Large-Scale Pre-trainingcited by 98
  3. 2022Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programmingcited by 91
  4. 2025Repo2Run: Automated Building Executable Environment for Code Repository at Scalecited by 39
  5. 2024LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completioncited by 32
  6. 2023Refactoring Programs Using Large Language Models with Few-Shot Examplescited by 28
  7. 2025CODEMENV: Benchmarking Large Language Models on Code Migrationcited by 13
  8. 2024LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generationcited by 12
  9. 2025MigrationBench: Repository-Level Code Migration Benchmark from Java 8cited by 11
  10. 2024Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Modelscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026ORCA: Evaluating LLMs on Data Science Code Translationcited by 0