# Is ChatGPT a Good Software Librarian? An Exploratory Study on the Use of ChatGPT for Software Library Recommendations

arXiv: https://arxiv.org/abs/2408.05128 · HTML: https://mlsanity.com/paper/2408.05128/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 42 references that are in the map.
- First posted on arXiv 9 Aug 2024.

## Builds on (most cited first, 4 of 4)

- [Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation](https://mlsanity.com/paper/2308.10335/) (2023)
- [Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2402.11702/) (2024)
- [A Preliminary Analysis on the Code Generation Capabilities of GPT-3.5 and Bard AI Models for Java Functions](https://mlsanity.com/paper/2305.09402/) (2023)
- [Evaluation of the Programming Skills of Large Language Models](https://mlsanity.com/paper/2405.14388/) (2024)

## Built on by (most cited first, 7 of 7)

- [A Study of LLMs' Preferences for Libraries and Programming Languages](https://mlsanity.com/paper/2503.17181/) (2025)
- [A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models](https://mlsanity.com/paper/2409.15228/) (2024)
- [Library Hallucinations in LLM-Generated Code: A Risk Analysis Grounded in Developer Queries](https://mlsanity.com/paper/2509.22202/) (2025)
- [HFuzzer: Testing Large Language Models for Package Hallucinations via Phrase-based Fuzzing](https://mlsanity.com/paper/2509.23835/) (2025)
- [Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code](https://mlsanity.com/paper/2605.05267/) (2026)
- [LibRec: Benchmarking Retrieval-Augmented LLMs for Library Migration Recommendations](https://mlsanity.com/paper/2508.09791/) (2025)
- [Skills That Don't Exist: A Large-Scale Study of Hallucinated Skill Recommendation in LLM Agents](https://mlsanity.com/paper/2607.12340/) (2026)

## Abstract (arXiv)

Software libraries play a critical role in the functionality, efficiency, and maintainability of software systems. As developers increasingly rely on Large Language Models (LLMs) to streamline their coding processes, the effectiveness of these models in recommending appropriate libraries becomes crucial yet remains largely unexplored. In this paper, we assess the effectiveness of ChatGPT as a software librarian and identify areas for improvement. We conducted an empirical study using GPT-3.5 Turbo to generate Python code for 10,000 Stack Overflow questions. Our findings show that ChatGPT uses third-party libraries nearly 10% more often than human developers, favoring widely adopted and well-established options. However, 14.2% of the recommended libraries had restrictive copyleft licenses, which were not explicitly communicated by ChatGPT. Additionally, 6.5% of the libraries did not work out of the box, leading to potential developer confusion and wasted time. While ChatGPT can be an effective software librarian, it should be improved by providing more explicit information on maintainability metrics and licensing. We recommend that developers implement rigorous dependency management practices and double-check library licenses before integrating LLM-generated code into their projects.