# LibRec: Benchmarking Retrieval-Augmented LLMs for Library Migration Recommendations

arXiv: https://arxiv.org/abs/2508.09791 · HTML: https://mlsanity.com/paper/2508.09791/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 25 of its 48 references that are in the map.
- First posted on arXiv 13 Aug 2025.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 25 of 25)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [ChatUniTest: A Framework for LLM-Based Test Generation](https://mlsanity.com/paper/2305.04764/) (2023)
- [Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning](https://mlsanity.com/paper/2304.11384/) (2023)
- [Retrieve and Refine: Exemplar-based Neural Comment Generation](https://mlsanity.com/paper/1910.10419/) (2019)
- [Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories](https://mlsanity.com/paper/2503.03586/) (2025)
- [Do Developers Update Their Library Dependencies? An Empirical Study on the Impact of Security Advisories on Library Migration](https://mlsanity.com/paper/1709.04621/) (2017)
- [EditSum: A Retrieve-and-Edit Framework for Source Code Summarization](https://mlsanity.com/paper/2308.13775/) (2023)
- [From Commit Message Generation to History-Aware Commit Message Completion](https://mlsanity.com/paper/2308.07655/) (2023)
- [When LLMs Meet API Documentation: Can Retrieval Augmentation Aid Code Generation Just as It Helps Developers?](https://mlsanity.com/paper/2503.15231/) (2025)
- [An Empirical Study of Usages, Updates and Risks of Third-Party Libraries in Java Projects](https://mlsanity.com/paper/2002.11028/) (2020)
- [Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Tests](https://mlsanity.com/paper/2408.11710/) (2024)
- [Selecting third-party libraries: The practitioners' perspective](https://mlsanity.com/paper/2005.12574/) (2020)
- [STEAM: Simulating the InTeractive BEhavior of ProgrAMmers for Automatic Bug Fixing](https://mlsanity.com/paper/2308.14460/) (2023)
- [Is ChatGPT a Good Software Librarian? An Exploratory Study on the Use of ChatGPT for Software Library Recommendations](https://mlsanity.com/paper/2408.05128/) (2024)
- [Characterizing Python Library Migrations](https://mlsanity.com/paper/2207.01124/) (2022)
- [Is this GitHub Project Maintained? Measuring the Level of Maintenance Activity of Open-Source Projects](https://mlsanity.com/paper/2003.04755/) (2020)
- [ComSum: Commit Messages Summarization and Meaning Preservation](https://mlsanity.com/paper/2108.10763/) (2021)
- [APIGen: Generative API Method Recommendation](https://mlsanity.com/paper/2401.15843/) (2024)
- [DDPT: Diffusion-Driven Prompt Tuning for Large Language Model Code Generation](https://mlsanity.com/paper/2504.04351/) (2025)
- [CCISolver: End-to-End Detection and Repair of Method-Level Code-Comment Inconsistency](https://mlsanity.com/paper/2506.20558/) (2025)
- [Where to Go Now? Finding Alternatives for Declining Packages in the npm Ecosystem](https://mlsanity.com/paper/2308.08667/) (2023)

## Abstract (arXiv)

In this paper, we propose LibRec, a novel framework that integrates the capabilities of LLMs with retrieval-augmented generation(RAG) techniques to automate the recommendation of alternative libraries. The framework further employs in-context learning to extract migration intents from commit messages to enhance the accuracy of its recommendations. To evaluate the effectiveness of LibRec, we introduce LibEval, a benchmark designed to assess the performance in the library migration recommendation task. LibEval comprises 2,888 migration records associated with 2,368 libraries extracted from 2,324 Python repositories. Each migration record captures source-target library pairs, along with their corresponding migration intents and intent types. Based on LibEval, we evaluated the effectiveness of ten popular LLMs within our framework, conducted an ablation study to examine the contributions of key components within our framework, explored the impact of various prompt strategies on the framework's performance, assessed its effectiveness across various intent types, and performed detailed failure case analyses.