# Discovering Forbidden Topics in Language Models

arXiv: https://arxiv.org/abs/2505.17441 · HTML: https://mlsanity.com/paper/2505.17441/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 65 references that are in the map.
- First posted on arXiv 23 May 2025.

## Builds on (most cited first, 30 of 37)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://mlsanity.com/paper/2305.13860/) (2023)
- [WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs](https://mlsanity.com/paper/2406.18495/) (2024)
- [Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks](https://mlsanity.com/paper/2404.02151/) (2024)
- [Safety Alignment Should Be Made More Than Just a Few Tokens Deep](https://mlsanity.com/paper/2406.05946/) (2024)
- [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021)
- [Tackling Climate Change with Machine Learning](https://mlsanity.com/paper/1906.05433/) (2019)
- [Red-Teaming the Stable Diffusion Safety Filter](https://mlsanity.com/paper/2210.04610/) (2022)
- [Bias and Debias in Recommender System: A Survey and Future Directions](https://mlsanity.com/paper/2010.03240/) (2020)
- [WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models](https://mlsanity.com/paper/2406.18510/) (2024)
- [AI Deception: A Survey of Examples, Risks, and Potential Solutions](https://mlsanity.com/paper/2308.14752/) (2023)
- [Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark](https://mlsanity.com/paper/2304.03279/) (2023)
- [Large Language Models can Strategically Deceive their Users when Put Under Pressure](https://mlsanity.com/paper/2311.07590/) (2023)
- [OpenXAI: Towards a Transparent Evaluation of Model Explanations](https://mlsanity.com/paper/2206.11104/) (2022)
- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [Foundation Models and Fair Use](https://mlsanity.com/paper/2303.15715/) (2023)
- [Copyright Violations and Large Language Models](https://mlsanity.com/paper/2310.13771/) (2023)
- [The Art of Saying No: Contextual Noncompliance in Language Models](https://mlsanity.com/paper/2407.12043/) (2024)
- [Evaluating Language-Model Agents on Realistic Autonomous Tasks](https://mlsanity.com/paper/2312.11671/) (2023)
- [Saving Face: Investigating the Ethical Concerns of Facial Recognition Auditing](https://mlsanity.com/paper/2001.00964/) (2020)
- [Large Language Models Reflect the Ideology of their Creators](https://mlsanity.com/paper/2410.18417/) (2024)

## Built on by (most cited first, 9 of 9)

- [Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences](https://mlsanity.com/paper/2510.13900/) (2025)
- [Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectives](https://mlsanity.com/paper/2511.06626/) (2025)
- [Large Reasoning Models Learn Better Alignment from Flawed Thinking](https://mlsanity.com/paper/2510.00938/) (2025)
- [Prefill Awareness in Large Language Models](https://mlsanity.com/paper/2606.12747/) (2026)
- [How Alignment Routes: Localizing, Scaling, and Controlling Policy Circuits in Language Models](https://mlsanity.com/paper/2604.04385/) (2026)
- [Matching Ranks Over Probability Yields Truly Deep Safety Alignment](https://mlsanity.com/paper/2512.05518/) (2025)
- [AuAu: A Benchmark for Auditing Authoritarian Alignment in Large Language Models](https://mlsanity.com/paper/2606.16127/) (2026)
- [Detecting Hidden Behaviors in LLMs via Activation-matched Finetuning](https://mlsanity.com/paper/2609.00351/) (2026)
- [Alignment of LRMs via Counter-Aligned Few-Shot Conversation Exposure](https://mlsanity.com/paper/2609.27763/) (2026)

## Abstract (arXiv)

Refusal discovery is the task of identifying the full set of topics that a language model refuses to discuss. We introduce this new problem setting and develop a refusal discovery method, Iterated Prefill Crawler (IPC), that uses token prefilling to find forbidden topics. We benchmark IPC on Tulu-3-8B, an open-source model with public safety tuning data. Our crawler manages to retrieve 31 out of 36 topics within a budget of 1000 prompts. Next, we scale the crawler to a frontier model using the prefilling option of Claude-Haiku. Finally, we crawl three widely used open-weight models: Llama-3.3-70B and two of its variants finetuned for reasoning: DeepSeek-R1-70B and Perplexity-R1-1776-70B. DeepSeek-R1-70B reveals patterns consistent with censorship tuning: The model exhibits "thought suppression" behavior that indicates memorization of CCP-aligned responses. Although Perplexity-R1-1776-70B is robust to censorship, IPC elicits CCP-aligned refusals answers in the quantized model. Our findings highlight the critical need for refusal discovery methods to detect biases, boundaries, and alignment failures of AI systems.