mlsanityData to 14 Oct 2026

A Study of LLMs' Preferences for Libraries and Programming Languages

arXiv 2503.17181 ↗First posted 21 Mar 2025cs.SE cs.AI cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Despite the rapid progress of large language models (LLMs) in code generation, existing evaluations focus on functional correctness or syntactic validity, overlooking how LLMs make critical design choices such as which library or programming language to use. To fill this gap, we perform the first empirical study of LLMs' preferences for libraries and programming languages when generating code, covering eight diverse LLMs. We observe a strong tendency to overuse widely adopted libraries such as NumPy; in up to 45% of cases, this usage is not required and deviates from the ground-truth solutions. The LLMs we study also show a significant preference toward Python as their default language. For high-performance project initialisation tasks where Python is not the optimal language, it remains the dominant choice in 58% of cases, and Rust is not used once. These results highlight how LLMs prioritise familiarity and popularity over suitability and task-specific optimality; underscoring the need for targeted fine-tuning, data diversification, and evaluation benchmarks that explicitly measure language and library selection fidelity.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 51 of its 108 references that are in the map. First posted on arXiv 21 Mar 2025.

Builds on

51 of 108 references.

References · 51

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  7. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  8. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  9. 2021Measuring Coding Challenge Competence With APPScited by 902
  10. 2024A Survey on Large Language Models for Code Generationcited by 658
  11. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  12. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
38 more
  1. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  2. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  3. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 98
  4. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 91
  5. 2024Bias in Large Language Models: Origin, Evaluation, and Mitigationcited by 77
  6. 2021NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentationcited by 73
  7. 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 73
  8. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  9. 2024A Survey on Evaluating Large Language Models in Code Generation Taskscited by 44
  10. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 37
  11. 2023Investigating and Designing for Trust in AI-powered Code Generation Toolscited by 34
  12. 2024On Leakage of Code Generation Evaluation Datasetscited by 32
  13. 2025A Closer Look at System Prompt Robustnesscited by 32
  14. 2024How Beginning Programmers and Code LLMs (Mis)read Each Othercited by 31
  15. 2024An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Projectcited by 28
  16. 2024Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seekingcited by 26
  17. 2023Uncovering and Quantifying Social Biases in Code Generationcited by 25
  18. 2024Bias Unveiled: Investigating Social Bias in LLM-Generated Codecited by 25
  19. 2022AixBench: A Code Generation Benchmark Datasetcited by 22
  20. 2023Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflowcited by 22
  21. 2024Benchmarks and Metrics for Evaluations of Code Generation: A Critical Reviewcited by 20
  22. 2025Auditing Prompt Caching in Language Model APIscited by 20
  23. 2024Can We Trust Large Language Models Generated Code? A Framework for In-Context Learning, Security Patterns, and Code Evaluations Across Diverse LLMscited by 19
  24. 2024Exploring Multi-Lingual Bias of Large Code Models in Code Generationcited by 17
  25. 2025Conversational AI as a Coding Assistant: Understanding Programmers' Interactions with and Expectations from Large Language Models for Codingcited by 11
  26. 2024Compositional API Recommendation for Library-Oriented Code Generationcited by 10
  27. 2025The Invisible Hand: Unveiling Provider Bias in Large Language Models for Code Generationcited by 10
  28. 2025Zero-Shot Verification-guided Chain of Thoughtscited by 10
  29. 2020Selecting third-party libraries: The practitioners' perspectivecited by 9
  30. 2024PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMscited by 9
  31. 2024An Empirical Study on Developers Shared Conversations with ChatGPT in GitHub Pull Requests and Issuescited by 9
  32. 2024Trained Without My Consent: Detecting Code Inclusion In Language Models Trained on Codecited by 8
  33. 2025Studying How Configurations Impact Code Generation in LLMs: the Case of ChatGPTcited by 8
  34. 2023A Qualitative Study on the Implementation Design Decisions of Developerscited by 7
  35. 2024Is ChatGPT a Good Software Librarian? An Exploratory Study on the Use of ChatGPT for Software Library Recommendationscited by 7
  36. 2024A Systematic Evaluation of Large Code Models in API Suggestion: When, Which, and Howcited by 7
  37. 2023Towards Causal Analysis of Empirical Software Engineering Data: The Impact of Programming Languages on Coding Competitionscited by 4
  38. 2021Performance vs Programming Effort between Rust and C on Multicore Architectures: Case Study in N-Bodycited by 2

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 14.

20 papers.

Sort:

By area:

  1. 2026Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomycited by 6
  2. 2025Assessing the Impact of Code Changes on the Fault Localizability of Large Language Modelscited by 4
  3. 2025Code Reasoning for Software Engineering Tasks: A Survey and A Call to Actioncited by 4
  4. 2025The Future of Generative AI in Software Engineering: A Vision from Industry and Academia in the European GENIUS Projectcited by 4
  5. 2026CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generationcited by 4
  6. 2026FlexSQL: Flexible Exploration and Execution Make Better Text-to-SQL Agentscited by 4
  7. 2025Taxonomy of migration scenarios for Qiskit refactoring using LLMscited by 2
  8. 2026FormalRewardBench: A Benchmark for Formal Theorem Proving Reward Modelscited by 2
  9. 2025Library Hallucinations in LLM-Generated Code: A Risk Analysis Grounded in Developer Queriescited by 1
  10. 2026Steering Code LLMs with Activation Directions for Language and Library Controlcited by 1
  11. 2026Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languagescited by 1
  12. 2026An Exploratory Study of Agent Plans for Agentic AI Coding Tools in Open-Source Softwarecited by 1
8 more
  1. 2025Rethinking Technology Stack Selection with AI Coding Proficiencycited by 0
  2. 2026From If-Statements to ML Pipelines: Revisiting Bias in Code-Generationcited by 0
  3. 2026Quality and Security Signals in AI-Generated Python Refactoring Pull Requestscited by 0
  4. 2026Do LLMs Favor Their Providers? Measuring Vertical Integration Bias in Code Generationcited by 0
  5. 2026The Impact of Configuring Agentic AI Coding Tools on Build-vs-Buy Decisions: A Study Protocolcited by 0
  6. 2026Using Probabilistic Programs to Train Inductive Reasoning in Large Language Modelscited by 0
  7. 2026Do AI Coding Assistants Check Before They Install? A Pre-Registered Demand-Side Audit of Trust Signals in the Research Software Supply Chaincited by 0
  8. 2026Can Agents Design Libraries for Agents?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs
  2. 2026Subjective Code Preferences in Experts and Large Language Models ↗
  3. 2025code_transformed: The Influence of Large Language Models on Code
  4. 2025Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation
  5. 2025Evaluating Large Language Models for Functional and Maintainable Code in Industrial Settings: A Case Study at ASML
  6. 2026When LLMs Lag Behind: Knowledge Conflicts from Evolving APIs in Code Generation
  7. 2025Holistic Evaluation of State-of-the-Art LLMs for Code Generation
  8. 2025AI-Generated Code Is Not Reproducible (Yet): An Empirical Study of Dependency Gaps in LLM-Based Coding Agents