# On the (In)Security of LLM App Stores

arXiv: https://arxiv.org/abs/2407.08422 · HTML: https://mlsanity.com/paper/2407.08422/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 60 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Builds on (most cited first, 21 of 21)

- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- [ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation](https://mlsanity.com/paper/2310.17389/) (2023)
- [On the Risk of Misinformation Pollution with Large Language Models](https://mlsanity.com/paper/2305.13661/) (2023)
- [PLeak: Prompt Leaking Attacks against Large Language Model Applications](https://mlsanity.com/paper/2405.06823/) (2024)
- [LLM Platform Security: Applying a Systematic Evaluation Framework to OpenAI's ChatGPT Plugins](https://mlsanity.com/paper/2309.10254/) (2023)
- [Malla: Demystifying Real-world Large Language Model Integrated Malicious Services](https://mlsanity.com/paper/2401.03315/) (2024)
- [On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused?](https://mlsanity.com/paper/2310.01581/) (2023)
- [Large Language Model Supply Chain: A Research Agenda](https://mlsanity.com/paper/2404.12736/) (2024)
- [A First Look at GPT Apps: Landscape and Vulnerability](https://mlsanity.com/paper/2402.15105/) (2024)
- [LLM App Store Analysis: A Vision and Roadmap](https://mlsanity.com/paper/2404.12737/) (2024)
- [Opening A Pandora's Box: Things You Should Know in the Era of Custom GPTs](https://mlsanity.com/paper/2401.00905/) (2023)
- [AbuseGPT: Abuse of Generative AI ChatBots to Create Smishing Campaigns](https://mlsanity.com/paper/2402.09728/) (2024)
- [Unveiling the Misuse Potential of Base Large Language Models via In-Context Learning](https://mlsanity.com/paper/2404.10552/) (2024)
- [GPT Store Mining and Analysis](https://mlsanity.com/paper/2405.10210/) (2024)
- [GPTZoo: A Large-scale Dataset of GPTs for the Research Community](https://mlsanity.com/paper/2405.15630/) (2024)
- [GPT in Sheep's Clothing: The Risk of Customized GPTs](https://mlsanity.com/paper/2401.09075/) (2024)
- [GPTs Window Shopping: An analysis of the Landscape of Custom ChatGPT Models](https://mlsanity.com/paper/2405.10547/) (2024)

## Built on by (most cited first, 14 of 14)

- [Position: Privacy Is Not Just Memorization!](https://mlsanity.com/paper/2510.01645/) (2025)
- [System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User Injection](https://mlsanity.com/paper/2505.06493/) (2025)
- [Malicious LLM-Based Conversational AI Makes Users Reveal Personal Information](https://mlsanity.com/paper/2506.11680/) (2025)
- [LLM Applications: Current Paradigms and the Next Frontier](https://mlsanity.com/paper/2503.04596/) (2025)
- [RedTeamLLM: an Agentic AI framework for offensive security](https://mlsanity.com/paper/2505.06913/) (2025)
- [When Agents Do Not Stop: Uncovering Infinite Agentic Loops in LLM Agents](https://mlsanity.com/paper/2607.01641/) (2026)
- [FDLLM: A Dedicated Detector for Black-Box LLMs Fingerprinting](https://mlsanity.com/paper/2501.16029/) (2025)
- [PromptCOS: Towards Content-only System Prompt Copyright Auditing for LLMs](https://mlsanity.com/paper/2509.03117/) (2025)
- [LLM App Squatting and Cloning](https://mlsanity.com/paper/2411.07518/) (2024)
- [AgentFlow: Building Agent Dependency Graphs for Static Analysis of Agent Programs](https://mlsanity.com/paper/2607.01640/) (2026)
- [A Large-Scale Empirical Analysis of Custom GPTs' Vulnerabilities in the OpenAI Ecosystem](https://mlsanity.com/paper/2505.08148/) (2025)
- [LaQual: An Automated Framework for LLM App Quality Evaluation](https://mlsanity.com/paper/2508.18636/) (2025)
- [Beyond Jailbreak: Unveiling Risks in LLM Applications Arising from Blurred Capability Boundaries](https://mlsanity.com/paper/2511.17874/) (2025)
- [Do No Harm? Hallucination and Actor-Level Abuse in Web-Deployed Medical Large Language Models](https://mlsanity.com/paper/2605.20591/) (2026)

## Abstract (arXiv)

LLM app stores have seen rapid growth, leading to the proliferation of numerous custom LLM apps. However, this expansion raises security concerns. In this study, we propose a three-layer concern framework to identify the potential security risks of LLM apps, i.e., LLM apps with abusive potential, LLM apps with malicious intent, and LLM apps with exploitable vulnerabilities. Over five months, we collected 786,036 LLM apps from six major app stores: GPT Store, FlowGPT, Poe, Coze, Cici, and Character.AI. Our research integrates static and dynamic analysis, the development of a large-scale toxic word dictionary (i.e., ToxicDict) comprising over 31,783 entries, and automated monitoring tools to identify and mitigate threats. We uncovered that 15,146 apps had misleading descriptions, 1,366 collected sensitive personal information against their privacy policies, and 15,996 generated harmful content such as hate speech, self-harm, extremism, etc. Additionally, we evaluated the potential for LLM apps to facilitate malicious activities, finding that 616 apps could be used for malware generation, phishing, etc. Our findings highlight the urgent need for robust regulatory frameworks and enhanced enforcement mechanisms.