mlsanityData to 14 Oct 2026

LibRec: Benchmarking Retrieval-Augmented LLMs for Library Migration Recommendations

arXiv 2508.09791 ↗First posted 13 Aug 2025cs.SE cs.AIcited by 0

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

In this paper, we propose LibRec, a novel framework that integrates the capabilities of LLMs with retrieval-augmented generation(RAG) techniques to automate the recommendation of alternative libraries. The framework further employs in-context learning to extract migration intents from commit messages to enhance the accuracy of its recommendations. To evaluate the effectiveness of LibRec, we introduce LibEval, a benchmark designed to assess the performance in the library migration recommendation task. LibEval comprises 2,888 migration records associated with 2,368 libraries extracted from 2,324 Python repositories. Each migration record captures source-target library pairs, along with their corresponding migration intents and intent types. Based on LibEval, we evaluated the effectiveness of ten popular LLMs within our framework, conducted an ablation study to examine the contributions of key components within our framework, explored the impact of various prompt strategies on the framework's performance, assessed its effectiveness across various intent types, and performed detailed failure case analyses.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 25 of its 48 references that are in the map. First posted on arXiv 13 Aug 2025.

Builds on

25 of 48 references.

References · 25

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  3. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  4. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  5. 2024A Survey on Large Language Models for Code Generationcited by 658
  6. 2023ChatUniTest: A Framework for LLM-Based Test Generationcited by 96
  7. 2023Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learningcited by 72
  8. 2019Retrieve and Refine: Exemplar-based Neural Comment Generationcited by 36
  9. 2025Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositoriescited by 31
  10. 2017Do Developers Update Their Library Dependencies? An Empirical Study on the Impact of Security Advisories on Library Migrationcited by 23
  11. 2023EditSum: A Retrieve-and-Edit Framework for Source Code Summarizationcited by 18
  12. 2023From Commit Message Generation to History-Aware Commit Message Completioncited by 16
13 more
  1. 2025When LLMs Meet API Documentation: Can Retrieval Augmentation Aid Code Generation Just as It Helps Developers?cited by 13
  2. 2020An Empirical Study of Usages, Updates and Risks of Third-Party Libraries in Java Projectscited by 12
  3. 2024Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Testscited by 11
  4. 2020Selecting third-party libraries: The practitioners' perspectivecited by 9
  5. 2023STEAM: Simulating the InTeractive BEhavior of ProgrAMmers for Automatic Bug Fixingcited by 9
  6. 2024Is ChatGPT a Good Software Librarian? An Exploratory Study on the Use of ChatGPT for Software Library Recommendationscited by 7
  7. 2022Characterizing Python Library Migrationscited by 4
  8. 2020Is this GitHub Project Maintained? Measuring the Level of Maintenance Activity of Open-Source Projectscited by 3
  9. 2021ComSum: Commit Messages Summarization and Meaning Preservationcited by 3
  10. 2024APIGen: Generative API Method Recommendationcited by 3
  11. 2025DDPT: Diffusion-Driven Prompt Tuning for Large Language Model Code Generationcited by 2
  12. 2025CCISolver: End-to-End Detection and Repair of Method-Level Code-Comment Inconsistencycited by 2
  13. 2023Where to Go Now? Finding Alternatives for Declining Packages in the npm Ecosystemcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RAGe: A Retrieval-Augmented Generation Evaluation Framework
  2. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
  3. 2025EncouRAGe: Evaluating RAG Local, Fast, and Reliable
  4. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routing
  5. 2024Long Context RAG Performance of Large Language Models
  6. 2024Automatic Library Migration Using Large Language Models: First Results
  7. 2025SEARA: An Automated Approach for Obtaining Optimal Retrievers
  8. 2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generation