# How Robust are LLM-Generated Library Imports? An Empirical Study using Stack Overflow

arXiv: https://arxiv.org/abs/2507.10818 · HTML: https://mlsanity.com/paper/2507.10818/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 52 references that are in the map.
- First posted on arXiv 14 Jul 2025.

## Builds on (most cited first, 22 of 22)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [Evaluation of OpenAI o1: Opportunities and Challenges of AGI](https://mlsanity.com/paper/2409.18486/) (2024)
- [The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development](https://mlsanity.com/paper/2302.07080/) (2023)
- [A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges](https://mlsanity.com/paper/2303.17125/) (2023)
- [Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation](https://mlsanity.com/paper/2308.10335/) (2023)
- [An Empirical Comparison of Dependency Network Evolution in Seven Software Packaging Ecosystems](https://mlsanity.com/paper/1710.04936/) (2017)
- [StackEval: Benchmarking LLMs in Coding Assistance](https://mlsanity.com/paper/2412.05288/) (2024)
- [Automating Dependency Updates in Practice: An Exploratory Study on GitHub Dependabot](https://mlsanity.com/paper/2206.07230/) (2022)
- [An Empirical Study of Usages, Updates and Risks of Third-Party Libraries in Java Projects](https://mlsanity.com/paper/2002.11028/) (2020)
- [Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2402.11702/) (2024)
- [Selecting third-party libraries: The practitioners' perspective](https://mlsanity.com/paper/2005.12574/) (2020)
- [A Comprehensive Study of Bloated Dependencies in the Maven Ecosystem](https://mlsanity.com/paper/2001.07808/) (2020)
- [Dependency Smells in JavaScript Projects](https://mlsanity.com/paper/2010.14573/) (2020)
- [Coverage-Based Debloating for Java Bytecode](https://mlsanity.com/paper/2008.08401/) (2020)
- [What are the characteristics of highly-selected packages? A case study on the npm ecosystem](https://mlsanity.com/paper/2204.04562/) (2022)
- [Not All Dependencies are Equal: An Empirical Study on Production Dependencies in NPM](https://mlsanity.com/paper/2207.14711/) (2022)
- [Dependency Update Strategies and Package Characteristics](https://mlsanity.com/paper/2305.15675/) (2023)
- [Evaluation of the Programming Skills of Large Language Models](https://mlsanity.com/paper/2405.14388/) (2024)
- [DepOwl: Detecting Dependency Bugs to Prevent Compatibility Failures](https://mlsanity.com/paper/2102.08543/) (2021)
- [An Overview and Catalogue of Dependency Challenges in Open Source Software Package Registries](https://mlsanity.com/paper/2409.18884/) (2024)

## Built on by (most cited first, 2 of 2)

- [Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions](https://mlsanity.com/paper/2605.06279/) (2026)
- [LLM Agents for Generating Microservice-based Applications: how complex is your specification?](https://mlsanity.com/paper/2508.20119/) (2025)

## Abstract (arXiv)

Software libraries are central to the functionality, security, and maintainability of modern code. As developers increasingly turn to Large Language Models (LLMs) to assist with programming tasks, understanding how these models recommend libraries is essential. In this paper, we conduct an empirical study of six state-of-the-art LLMs, both proprietary and open-source, by prompting them to solve real-world Python problems sourced from Stack Overflow. We analyze the types of libraries they import, the characteristics of those libraries, and the extent to which the recommendations are usable out of the box. Our results show that LLMs predominantly favour third-party libraries over standard ones, and often recommend mature, popular, and permissively licensed dependencies. However, we also identify gaps in usability: 4.6% of the libraries could not be resolved automatically due to structural mismatches between import names and installable packages, and only two models (out of six) provided installation guidance. While the generated code is technically valid, the lack of contextual support places the burden of manually resolving dependencies on the user. Our findings offer actionable insights for both developers and researchers, and highlight opportunities to improve the reliability and usability of LLM-generated code in the context of software dependencies.