# A Large-Scale Empirical Analysis of Custom GPTs' Vulnerabilities in the OpenAI Ecosystem

arXiv: https://arxiv.org/abs/2505.08148 · HTML: https://mlsanity.com/paper/2505.08148/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 43 references that are in the map.
- First posted on arXiv 13 May 2025.

## Builds on (most cited first, 13 of 13)

- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy](https://mlsanity.com/paper/2307.00691/) (2023)
- [Text Summarization Using Large Language Models: A Comparative Study of MPT-7b-instruct, Falcon-7b-instruct, and OpenAI Chat-GPT Models](https://mlsanity.com/paper/2310.10449/) (2023)
- [Lost in Translation: Large Language Models in Non-English Content Analysis](https://mlsanity.com/paper/2306.07377/) (2023)
- [Fine-Tuning and Prompt Engineering for Large Language Models-based Code Review Automation](https://mlsanity.com/paper/2402.00905/) (2024)
- [Negotiating with LLMS: Prompt Hacks, Skill Gaps, and Reasoning Deficits](https://mlsanity.com/paper/2312.03720/) (2023)
- [A First Look at GPT Apps: Landscape and Vulnerability](https://mlsanity.com/paper/2402.15105/) (2024)
- [Attacks on Third-Party APIs of Large Language Models](https://mlsanity.com/paper/2404.16891/) (2024)
- [On the (In)Security of LLM App Stores](https://mlsanity.com/paper/2407.08422/) (2024)
- [LLM App Store Analysis: A Vision and Roadmap](https://mlsanity.com/paper/2404.12737/) (2024)
- [Opening A Pandora's Box: Things You Should Know in the Era of Custom GPTs](https://mlsanity.com/paper/2401.00905/) (2023)
- [GPT Store Mining and Analysis](https://mlsanity.com/paper/2405.10210/) (2024)
- [GPTs Window Shopping: An analysis of the Landscape of Custom ChatGPT Models](https://mlsanity.com/paper/2405.10547/) (2024)

## Built on by (most cited first, 1 of 1)

- [Toward Continuous Assurance for the Democratization of AI Agent Creation in Industry](https://mlsanity.com/paper/2607.21495/) (2026)

## Abstract (arXiv)

Millions of users leverage generative pretrained transformer (GPT)-based language models developed by leading model providers for a wide range of tasks. To support enhanced user interaction and customization, many platforms-such as OpenAI-now enable developers to create and publish tailored model instances, known as custom GPTs, via dedicated repositories or application stores. These custom GPTs empower users to browse and interact with specialized applications designed to meet specific needs. However, as custom GPTs see growing adoption, concerns regarding their security vulnerabilities have intensified. Existing research on these vulnerabilities remains largely theoretical, often lacking empirical, large-scale, and statistically rigorous assessments of associated risks. In this study, we analyze 14,904 custom GPTs to assess their susceptibility to seven exploitable threats, such as roleplay-based attacks, system prompt leakage, phishing content generation, and malicious code synthesis, across various categories and popularity tiers within the OpenAI marketplace. We introduce a multi-metric ranking system to examine the relationship between a custom GPT's popularity and its associated security risks. Our findings reveal that over 95% of custom GPTs lack adequate security protections. The most prevalent vulnerabilities include roleplay-based vulnerabilities (96.51%), system prompt leakage (92.20%), and phishing (91.22%). Furthermore, we demonstrate that OpenAI's foundational models exhibit inherent security weaknesses, which are often inherited or amplified in custom GPTs. These results highlight the urgent need for enhanced security measures and stricter content moderation to ensure the safe deployment of GPT-based applications.