# LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?

arXiv: https://arxiv.org/abs/2307.10719 · HTML: https://mlsanity.com/paper/2307.10719/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 59 references that are in the map.
- First posted on arXiv 20 Jul 2023.

## Builds on (most cited first, 29 of 29)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://mlsanity.com/paper/2305.13860/) (2023)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [Can AI-Generated Text be Reliably Detected?](https://mlsanity.com/paper/2303.11156/) (2023)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [Large Language Models as Tool Makers](https://mlsanity.com/paper/2305.17126/) (2023)
- [Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks](https://mlsanity.com/paper/2302.05733/) (2023)
- [Fundamental Limitations of Alignment in Large Language Models](https://mlsanity.com/paper/2304.11082/) (2023)
- [On the Tool Manipulation Capability of Open-source Large Language Models](https://mlsanity.com/paper/2305.16504/) (2023)
- [Sponge Examples: Energy-Latency Attacks on Neural Networks](https://mlsanity.com/paper/2006.03463/) (2020)
- [Bad Characters: Imperceptible NLP Attacks](https://mlsanity.com/paper/2106.09898/) (2021)
- [Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examples](https://mlsanity.com/paper/2209.02128/) (2022)
- [Memory Augmented Large Language Models are Computationally Universal](https://mlsanity.com/paper/2301.04589/) (2023)
- [On the Impossible Safety of Large AI Models](https://mlsanity.com/paper/2209.15259/) (2022)
- [Impossibility Results in AI: A Survey](https://mlsanity.com/paper/2109.00484/) (2021)
- [When Vision Fails: Text Attacks Against ViT and OCR](https://mlsanity.com/paper/2306.07033/) (2023)

## Built on by (most cited first, 30 of 49)

- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [StruQ: Defending Against Prompt Injection with Structured Queries](https://mlsanity.com/paper/2402.06363/) (2024)
- [Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks](https://mlsanity.com/paper/2310.10844/) (2023)
- [LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet](https://mlsanity.com/paper/2408.15221/) (2024)
- [Identifying and Mitigating the Security Risks of Generative AI](https://mlsanity.com/paper/2308.14840/) (2023)
- [Coercing LLMs to do and reveal (almost) anything](https://mlsanity.com/paper/2402.14020/) (2024)
- [Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities](https://mlsanity.com/paper/2308.12833/) (2023)
- [Safeguarding Large Language Models: A Survey](https://mlsanity.com/paper/2406.02622/) (2024)
- [JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models](https://mlsanity.com/paper/2407.01599/) (2024)
- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs](https://mlsanity.com/paper/2406.09324/) (2024)
- [JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework](https://mlsanity.com/paper/2410.12855/) (2024)
- [Can Language Models be Instructed to Protect Personal Information?](https://mlsanity.com/paper/2310.02224/) (2023)
- [Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)](https://mlsanity.com/paper/2407.14937/) (2024)
- [Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models](https://mlsanity.com/paper/2403.04786/) (2024)
- [ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings](https://mlsanity.com/paper/2402.16006/) (2024)
- [Adversaries Can Misuse Combinations of Safe Models](https://mlsanity.com/paper/2406.14595/) (2024)
- [MOSSBench: Is Your Multimodal Language Model Oversensitive to Safe Queries?](https://mlsanity.com/paper/2406.17806/) (2024)
- [Navigating LLM Ethics: Advancements, Challenges, and Future Directions](https://mlsanity.com/paper/2406.18841/) (2024)
- [Make Them Spill the Beans! Coercive Knowledge Extraction from (Production) LLMs](https://mlsanity.com/paper/2312.04782/) (2023)
- [Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?](https://mlsanity.com/paper/2404.06644/) (2024)
- [Breach By A Thousand Leaks: Unsafe Information Leakage in `Safe' AI Responses](https://mlsanity.com/paper/2407.02551/) (2024)
- [Security Attacks on LLM-based Code Completion Tools](https://mlsanity.com/paper/2408.11006/) (2024)
- [Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI](https://mlsanity.com/paper/2409.15398/) (2024)
- [Steering the CensorShip: Uncovering Representation Vectors for LLM "Thought" Control](https://mlsanity.com/paper/2504.17130/) (2025)
- [What Large Language Models Do Not Talk About: An Empirical Study of Moderation and Censorship Practices](https://mlsanity.com/paper/2504.03803/) (2025)
- [Whose LLM is it Anyway? Linguistic Comparison and LLM Attribution for GPT-3.5, GPT-4 and Bard](https://mlsanity.com/paper/2402.14533/) (2024)
- [Moderating Model Marketplaces: Platform Governance Puzzles for AI Intermediaries](https://mlsanity.com/paper/2311.12573/) (2023)
- [A Survey on Responsible Generative AI: What to Generate and What Not](https://mlsanity.com/paper/2404.05783/) (2024)

## Abstract (arXiv)

Large language models (LLMs) have exhibited impressive capabilities in comprehending complex instructions. However, their blind adherence to provided instructions has led to concerns regarding risks of malicious use. Existing defence mechanisms, such as model fine-tuning or output censorship using LLMs, have proven to be fallible, as LLMs can still generate problematic responses. Commonly employed censorship approaches treat the issue as a machine learning problem and rely on another LM to detect undesirable content in LLM outputs. In this paper, we present the theoretical limitations of such semantic censorship approaches. Specifically, we demonstrate that semantic censorship can be perceived as an undecidable problem, highlighting the inherent challenges in censorship that arise due to LLMs' programmatic and instruction-following capabilities. Furthermore, we argue that the challenges extend beyond semantic censorship, as knowledgeable attackers can reconstruct impermissible outputs from a collection of permissible ones. As a result, we propose that the problem of censorship needs to be reevaluated; it should be treated as a security problem which warrants the adaptation of security-based approaches to mitigate potential risks.