# LLM Platform Security: Applying a Systematic Evaluation Framework to OpenAI's ChatGPT Plugins

arXiv: https://arxiv.org/abs/2309.10254 · HTML: https://mlsanity.com/paper/2309.10254/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 91 references that are in the map.
- First posted on arXiv 19 Sep 2023.

## Builds on (most cited first, 16 of 16)

- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks](https://mlsanity.com/paper/2302.05733/) (2023)
- [Co-Writing with Opinionated Language Models Affects Users' Views](https://mlsanity.com/paper/2302.00560/) (2023)
- [Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs](https://mlsanity.com/paper/2307.10490/) (2023)
- [Ethical Frameworks and Computer Security Trolley Problems: Foundations for Conversations](https://mlsanity.com/paper/2302.14326/) (2023)
- [Tracking, Profiling, and Ad Targeting in the Alexa Echo Smart Speaker Ecosystem](https://mlsanity.com/paper/2204.10920/) (2022)
- [Cookie Synchronization: Everything You Always Wanted to Know But Were Afraid to Ask](https://mlsanity.com/paper/1805.10505/) (2018)
- [CanaryTrap: Detecting Data Misuse by Third-Party Apps on Online Social Networks](https://mlsanity.com/paper/2006.15794/) (2020)
- [EmPoWeb: Empowering Web Applications with Browser Extensions](https://mlsanity.com/paper/1901.03397/) (2019)
- [Experimental Security Analysis of the App Model in Business Collaboration Platforms](https://mlsanity.com/paper/2203.04427/) (2022)

## Built on by (most cited first, 30 of 51)

- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models](https://mlsanity.com/paper/2312.14197/) (2023)
- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems](https://mlsanity.com/paper/2403.04960/) (2024)
- [System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspective](https://mlsanity.com/paper/2409.19091/) (2024)
- [A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems](https://mlsanity.com/paper/2402.18649/) (2024)
- [(A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advice](https://mlsanity.com/paper/2402.01864/) (2024)
- ["It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents](https://mlsanity.com/paper/2309.11653/) (2023)
- [ACE: A Security Architecture for LLM-Integrated App Systems](https://mlsanity.com/paper/2504.20984/) (2025)
- [Can LLMs Follow Simple Rules?](https://mlsanity.com/paper/2311.04235/) (2023)
- [SPML: A DSL for Defending Language Models Against Prompt Attacks](https://mlsanity.com/paper/2402.11755/) (2024)
- [Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges](https://mlsanity.com/paper/2510.23883/) (2025)
- [Formal Analysis and Supply Chain Security for Agentic AI Skills](https://mlsanity.com/paper/2603.00195/) (2026)
- [Real AI Agents with Fake Memories: Fatal Context Manipulation Attacks on Web3 Agents](https://mlsanity.com/paper/2503.16248/) (2025)
- [Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Models](https://mlsanity.com/paper/2402.01749/) (2024)
- [Les Dissonances: Cross-Tool Harvesting and Polluting in Pool-of-Tools Empowered LLM Agents](https://mlsanity.com/paper/2504.03111/) (2025)
- [Towards Automating Data Access Permissions in AI Agents](https://mlsanity.com/paper/2511.17959/) (2025)
- [AI Risk Management Should Incorporate Both Safety and Security](https://mlsanity.com/paper/2405.19524/) (2024)
- [Attacks on Third-Party APIs of Large Language Models](https://mlsanity.com/paper/2404.16891/) (2024)
- [LLM Whisperer: An Inconspicuous Attack to Bias LLM Responses](https://mlsanity.com/paper/2406.04755/) (2024)
- [Agent-Sentry: Bounding LLM Agents via Execution Provenance](https://mlsanity.com/paper/2603.22868/) (2026)
- [On the (In)Security of LLM App Stores](https://mlsanity.com/paper/2407.08422/) (2024)
- [An In-Depth Investigation of Data Collection in LLM App Ecosystems](https://mlsanity.com/paper/2408.13247/) (2024)
- [System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User Injection](https://mlsanity.com/paper/2505.06493/) (2025)
- [A Security Risk Taxonomy for Prompt-Based Interaction With Large Language Models](https://mlsanity.com/paper/2311.11415/) (2023)
- [HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents?](https://mlsanity.com/paper/2604.15415/) (2026)
- [SandboxEval: Towards Securing Test Environment for Untrusted Code](https://mlsanity.com/paper/2504.00018/) (2025)
- [Review of Generative AI Methods in Cybersecurity](https://mlsanity.com/paper/2403.08701/) (2024)
- [A Survey on Responsible Generative AI: What to Generate and What Not](https://mlsanity.com/paper/2404.05783/) (2024)
- [LLM Applications: Current Paradigms and the Next Frontier](https://mlsanity.com/paper/2503.04596/) (2025)

## Abstract (arXiv)

Large language model (LLM) platforms, such as ChatGPT, have recently begun offering an app ecosystem to interface with third-party services on the internet. While these apps extend the capabilities of LLM platforms, they are developed by arbitrary third parties and thus cannot be implicitly trusted. Apps also interface with LLM platforms and users using natural language, which can have imprecise interpretations. In this paper, we propose a framework that lays a foundation for LLM platform designers to analyze and improve the security, privacy, and safety of current and future third-party integrated LLM platforms. Our framework is a formulation of an attack taxonomy that is developed by iteratively exploring how LLM platform stakeholders could leverage their capabilities and responsibilities to mount attacks against each other. As part of our iterative process, we apply our framework in the context of OpenAI's plugin (apps) ecosystem. We uncover plugins that concretely demonstrate the potential for the types of issues that we outline in our attack taxonomy. We conclude by discussing novel challenges and by providing recommendations to improve the security, privacy, and safety of present and future LLM-based computing platforms.