# LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation

arXiv: https://arxiv.org/abs/2412.04478 · HTML: https://mlsanity.com/paper/2412.04478/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 5 references that are in the map.
- First posted on arXiv 19 Nov 2024.

## Builds on (most cited first, 2 of 2)

- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)

## Built on by (most cited first, 12 of 12)

- [DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale](https://mlsanity.com/paper/2501.13699/) (2025)
- [Environment-Aware Code Generation: How far are We?](https://mlsanity.com/paper/2601.12262/) (2026)
- [What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistants](https://mlsanity.com/paper/2605.30777/) (2026)
- [GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities](https://mlsanity.com/paper/2507.12367/) (2025)
- [Automating Database-Native Function Code Synthesis with LLMs](https://mlsanity.com/paper/2604.06231/) (2026)
- [TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks](https://mlsanity.com/paper/2601.22597/) (2026)
- [Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions](https://mlsanity.com/paper/2605.06279/) (2026)
- [Knowledge boundary probing and demand-guided intervention for LLM-based power system code generation](https://mlsanity.com/paper/2605.31478/) (2026)
- [Towards Knowledge Alignment in Code LLMs: Contrastive Unlearning for Evolving APIs](https://mlsanity.com/paper/2606.30810/) (2026)
- [Toward Executable Repository-Level Code Generation via Environment Alignment](https://mlsanity.com/paper/2604.03622/) (2026)
- [Diagnosing Knowledge Gaps in LLM Tool Use: An Agentic Benchmark for Novel API Acquisition](https://mlsanity.com/paper/2606.03657/) (2026)
- [LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models](https://mlsanity.com/paper/2606.25402/) (2026)

## Abstract (arXiv)

Recent advancements in code completion models have primarily focused on local file contexts. However, these studies do not fully capture the complexity of real-world software development, which often requires the use of rapidly-evolving public libraries. To fill the gap, we introduce LibEvolutionEval, a detailed study requiring an understanding of library evolution to perform in-line code completion accurately. LibEvolutionEval provides a version-specific code-completion task comprised of eight libraries (torch, torchvision, scipy, pil, tqdm, pyyaml, matplotlib, and pandas) as they evolve over the year along with a detailed analysis of the evolution of two popular and well-maintained public libraries: PyTorch and Matplotlib. We evaluate popular public models and find that public library evolution significantly influences model performance. We explored mitigation methods by studying how retrieved version-specific library documentation and prompting can improve the model's capability in handling these fast-evolving packages, paving a promising future path in better handling fast-evolving libraries.