mlsanityData to 14 Oct 2026

Towards Safer Chatbots: Automated Policy Compliance Evaluation of Custom GPTs

arXiv 2502.01436 ↗First posted 3 Feb 2025cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

User-configured chatbots built on top of large language models are increasingly available through centralized marketplaces such as OpenAI's GPT Store. While these platforms enforce usage policies intended to prevent harmful or inappropriate behavior, the scale and opacity of customized chatbots make systematic policy enforcement challenging. As a result, policy-violating chatbots continue to remain publicly accessible despite existing review processes. This paper presents a fully automated method for evaluating the compliance of Custom GPTs with its marketplace usage policy using black-box interaction. The method combines large-scale GPT discovery, policy-driven red-teaming prompts, and automated compliance assessment using an LLM-as-a-judge. We focus on three policy-relevant domains explicitly addressed in OpenAI's usage policies: Romantic, Cybersecurity, and Academic GPTs. We validate our compliance assessment component against a human-annotated ground-truth dataset, achieving an F1 score of 0.975 for binary policy violation detection. We then apply the method in a large-scale empirical study of 782 Custom GPTs retrieved from the GPT Store. The results show that 58.7% of the evaluated GPTs exhibit at least one policy-violating response, with substantial variation across policy domains. A comparison with the base models (GPT-4 and GPT-4o) indicates that most violations originate from model-level behavior, while customization tends to amplify these tendencies rather than create new failure modes. Our findings reveal limitations in current review mechanisms for user-configured chatbots and demonstrate the feasibility of scalable, behavior-based policy compliance evaluation.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 27 of its 57 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

27 of 57 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  6. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  7. 2019Defending Against Neural Fake Newscited by 559
  8. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  9. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  10. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  11. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  12. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 134
15 more
  1. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  2. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  3. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 92
  4. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Modelscited by 86
  5. 2023Safety Assessment of Chinese Large Language Modelscited by 85
  6. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 77
  7. 2023Assessing Prompt Injection Risks in 200+ Custom GPTscited by 55
  8. 2018Polisis: Automated Analysis and Presentation of Privacy Policies Using Deep Learningcited by 50
  9. 2024Learning From Failure: Integrating Negative Examples when Fine-tuning Large Language Models as Agentscited by 50
  10. 2023Knowledge Sanitization of Large Language Modelscited by 37
  11. 2024From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguardscited by 17
  12. 2024A First Look at GPT Apps: Landscape and Vulnerabilitycited by 16
  13. 2024Lessons From an App Update at Replika AI: Identity Discontinuity in Human-AI Relationshipscited by 15
  14. 2023A Security Risk Taxonomy for Prompt-Based Interaction With Large Language Modelscited by 12
  15. 2024Online Safety Analysis for LLMs: a Benchmark, an Assessment, and a Path Forwardcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Policy Compliance of User Requests in Natural Language for AI Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Privacy and Security Threat for OpenAI GPTs
  2. 2024RogueGPT: dis-ethical tuning transforms ChatGPT4 into a Rogue AI in 158 Words
  3. 2026GroupGPT: A Token-efficient and Privacy-preserving Agentic Framework for Multi-User Chat Assistant
  4. 2023Regulating ChatGPT and other Large Generative AI Models
  5. 2025A Proposal for Evaluating the Operational Risk for ChatBots based on Large Language Models
  6. 2025Understanding Privacy Norms Around LLM-Based Chatbots: A Contextual Integrity Perspective
  7. 2025When AI Meets the Web: Prompt Injection Risks in Third-Party AI Chatbot Plugins
  8. 2025RICoTA: Red-teaming of In-the-wild Conversation with Test Attempts