mlsanityData to 14 Oct 2026

How Robust are LLM-Generated Library Imports? An Empirical Study using Stack Overflow

arXiv 2507.10818 ↗First posted 14 Jul 2025cs.SE cs.AI cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Software libraries are central to the functionality, security, and maintainability of modern code. As developers increasingly turn to Large Language Models (LLMs) to assist with programming tasks, understanding how these models recommend libraries is essential. In this paper, we conduct an empirical study of six state-of-the-art LLMs, both proprietary and open-source, by prompting them to solve real-world Python problems sourced from Stack Overflow. We analyze the types of libraries they import, the characteristics of those libraries, and the extent to which the recommendations are usable out of the box. Our results show that LLMs predominantly favour third-party libraries over standard ones, and often recommend mature, popular, and permissively licensed dependencies. However, we also identify gaps in usability: 4.6% of the libraries could not be resolved automatically due to structural mismatches between import names and installable packages, and only two models (out of six) provided installation guidance. While the generated code is technically valid, the lack of contextual support places the burden of manually resolving dependencies on the user. Our findings offer actionable insights for both developers and researchers, and highlight opportunities to improve the reliability and usability of LLM-generated code in the context of software dependencies.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 22 of its 52 references that are in the map. First posted on arXiv 14 Jul 2025.

Builds on

22 of 52 references.

References · 22

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  4. 2024Evaluation of OpenAI o1: Opportunities and Challenges of AGIcited by 127
  5. 2023The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Developmentcited by 96
  6. 2023A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challengescited by 75
  7. 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 73
  8. 2017An Empirical Comparison of Dependency Network Evolution in Seven Software Packaging Ecosystemscited by 20
  9. 2024StackEval: Benchmarking LLMs in Coding Assistancecited by 15
  10. 2022Automating Dependency Updates in Practice: An Exploratory Study on GitHub Dependabotcited by 13
  11. 2020An Empirical Study of Usages, Updates and Risks of Third-Party Libraries in Java Projectscited by 12
  12. 2024Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generationcited by 11
10 more
  1. 2020Selecting third-party libraries: The practitioners' perspectivecited by 9
  2. 2020A Comprehensive Study of Bloated Dependencies in the Maven Ecosystemcited by 5
  3. 2020Dependency Smells in JavaScript Projectscited by 4
  4. 2020Coverage-Based Debloating for Java Bytecodecited by 3
  5. 2022What are the characteristics of highly-selected packages? A case study on the npm ecosystemcited by 3
  6. 2022Not All Dependencies are Equal: An Empirical Study on Production Dependencies in NPMcited by 3
  7. 2023Dependency Update Strategies and Package Characteristicscited by 2
  8. 2024Evaluation of the Programming Skills of Large Language Modelscited by 2
  9. 2021DepOwl: Detecting Dependency Bugs to Prevent Compatibility Failurescited by 1
  10. 2024An Overview and Catalogue of Dependency Challenges in Open Source Software Package Registriescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.