# An Empirical Study on Using Large Language Models to Analyze Software Supply Chain Security Failures

arXiv: https://arxiv.org/abs/2308.04898 · HTML: https://mlsanity.com/paper/2308.04898/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 100 references that are in the map.
- First posted on arXiv 9 Aug 2023.

## Builds on (most cited first, 14 of 14)

- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design](https://mlsanity.com/paper/2303.07839/) (2023)
- [Backstabber's Knife Collection: A Review of Open Source Software Supply Chain Attacks](https://mlsanity.com/paper/2005.09535/) (2020)
- [Taxonomy of Attacks on Open-Source Software Supply Chains](https://mlsanity.com/paper/2204.04008/) (2022)
- [Small World with High Risks: A Study of Security Threats in the npm Ecosystem](https://mlsanity.com/paper/1902.09217/) (2019)
- [What are Weak Links in the npm Supply Chain?](https://mlsanity.com/paper/2112.10165/) (2021)
- [Time Pressure in Software Engineering: A Systematic Review](https://mlsanity.com/paper/1901.05771/) (2019)
- [Vulnerable Open Source Dependencies: Counting Those That Matter](https://mlsanity.com/paper/1808.09753/) (2018)
- [NLP-assisted software testing: A systematic mapping of the literature](https://mlsanity.com/paper/1806.00696/) (2018)
- [Reflecting on Recurring Failures in IoT Development](https://mlsanity.com/paper/2206.13560/) (2022)
- [Reflections on Software Failure Analysis](https://mlsanity.com/paper/2209.02930/) (2022)
- [Incorporating Failure Knowledge into Design Decisions for IoT Systems: A Controlled Experiment on Novices](https://mlsanity.com/paper/2206.13562/) (2022)
- [Software supply chain: review of attacks, risk assessment strategies and security controls](https://mlsanity.com/paper/2305.14157/) (2023)

## Built on by (most cited first, 11 of 11)

- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [InvAgent: A Large Language Model based Multi-Agent System for Inventory Management in Supply Chains](https://mlsanity.com/paper/2407.11384/) (2024)
- [Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Code](https://mlsanity.com/paper/2312.14856/) (2023)
- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [A Survey of Large Language Models in Cybersecurity](https://mlsanity.com/paper/2402.16968/) (2024)
- [HuggingGraph: Understanding the Supply Chain of LLM Ecosystem](https://mlsanity.com/paper/2507.14240/) (2025)
- [Integrating Large Language Models with Internet of Things Applications](https://mlsanity.com/paper/2410.19223/) (2024)
- [Proof of Time: A Benchmark for Evaluating Scientific Idea Judgments](https://mlsanity.com/paper/2601.07606/) (2026)
- [Leveraging Large Language Models to Detect npm Malicious Packages](https://mlsanity.com/paper/2403.12196/) (2024)
- [Securing Educational LLMs: A Generalised Taxonomy of Attacks on LLMs and DREAD Risk Assessment](https://mlsanity.com/paper/2508.08629/) (2025)
- [Towards the Anonymization of the Language Modeling](https://mlsanity.com/paper/2501.02407/) (2025)

## Abstract (arXiv)

As we increasingly depend on software systems, the consequences of breaches in the software supply chain become more severe. High-profile cyber attacks like those on SolarWinds and ShadowHammer have resulted in significant financial and data losses, underlining the need for stronger cybersecurity. One way to prevent future breaches is by studying past failures. However, traditional methods of analyzing these failures require manually reading and summarizing reports about them. Automated support could reduce costs and allow analysis of more failures. Natural Language Processing (NLP) techniques such as Large Language Models (LLMs) could be leveraged to assist the analysis of failures. In this study, we assessed the ability of Large Language Models (LLMs) to analyze historical software supply chain breaches. We used LLMs to replicate the manual analysis of 69 software supply chain security failures performed by members of the Cloud Native Computing Foundation (CNCF). We developed prompts for LLMs to categorize these by four dimensions: type of compromise, intent, nature, and impact. GPT 3.5s categorizations had an average accuracy of 68% and Bard had an accuracy of 58% over these dimensions. We report that LLMs effectively characterize software supply chain failures when the source articles are detailed enough for consensus among manual analysts, but cannot yet replace human analysts. Future work can improve LLM performance in this context, and study a broader range of articles and failures.