# SafeGPT: Preventing Data Leakage and Unethical Outputs in Enterprise LLM Use

arXiv: https://arxiv.org/abs/2601.06366 · HTML: https://mlsanity.com/paper/2601.06366/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 38 references that are in the map.
- First posted on arXiv 10 Jan 2026.

## Builds on (most cited first, 16 of 16)

- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails](https://mlsanity.com/paper/2310.10501/) (2023)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [Deceiving Google's Perspective API Built for Detecting Toxic Comments](https://mlsanity.com/paper/1702.08138/) (2017)
- [Defending Pre-trained Language Models as Few-shot Learners against Backdoor Attacks](https://mlsanity.com/paper/2309.13256/) (2023)
- [Towards Robust Toxic Content Classification](https://mlsanity.com/paper/1912.06872/) (2019)
- [Zodiac: A Cardiologist-Level LLM Framework for Multi-Agent Diagnostics](https://mlsanity.com/paper/2410.02026/) (2024)
- [POLAR: Automating Cyber Threat Prioritization through LLM-Powered Assessment](https://mlsanity.com/paper/2510.01552/) (2025)
- [PromptFix: Few-shot Backdoor Removal via Adversarial Prompt Tuning](https://mlsanity.com/paper/2406.04478/) (2024)
- [Schema-Grounded LLM Extraction for FHIR Patient Digital Twins](https://mlsanity.com/paper/2601.05847/) (2026)

## Built on by (most cited first, 3 of 3)

- [Generating Leakage-Free Benchmarks for Robust RAG Evaluation](https://mlsanity.com/paper/2605.08838/) (2026)
- [Persona-Execution Separation: An Architecture Pattern for Evolving LLM Agents under Execution Audit](https://mlsanity.com/paper/2608.27427/) (2026)
- [Polar: LLM-Powered Synthesis of Real-World Cyber Evidence for Prioritization and Mitigation](https://mlsanity.com/paper/2610.07298/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are transforming enterprise workflows but introduce security and ethics challenges when employees inadvertently share confidential data or generate policy-violating content. This paper proposes SafeGPT, a two-sided guardrail system preventing sensitive data leakage and unethical outputs. SafeGPT integrates input-side detection/redaction, output-side moderation/reframing, and human-in-the-loop feedback. Experiments demonstrate SafeGPT effectively reduces data leakage risk and biased outputs while maintaining satisfaction.