# Forbidden Facts: An Investigation of Competing Objectives in Llama-2

arXiv: https://arxiv.org/abs/2312.08793 · HTML: https://mlsanity.com/paper/2312.08793/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 15 references that are in the map.
- First posted on arXiv 14 Dec 2023.

## Builds on (most cited first, 10 of 10)

- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Copy Suppression: Comprehensively Understanding an Attention Head](https://mlsanity.com/paper/2310.04625/) (2023)

## Built on by (most cited first, 4 of 4)

- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [When Parts Are Greater Than Sums: Individual LLM Components Can Outperform Full Models](https://mlsanity.com/paper/2406.13131/) (2024)
- [How Do Language Models Understand Tables? A Mechanistic Analysis of Cell Location](https://mlsanity.com/paper/2602.08548/) (2026)
- [Align, Unify, Suppress, Route: A Coherentist View of Transformer Computation](https://mlsanity.com/paper/2608.22034/) (2026)

## Abstract (arXiv)

LLMs often face competing pressures (for example helpfulness vs. harmlessness). To understand how models resolve such conflicts, we study Llama-2-chat models on the forbidden fact task. Specifically, we instruct Llama-2 to truthfully complete a factual recall statement while forbidding it from saying the correct answer. This often makes the model give incorrect answers. We decompose Llama-2 into 1000+ components, and rank each one with respect to how useful it is for forbidding the correct answer. We find that in aggregate, around 35 components are enough to reliably implement the full suppression behavior. However, these components are fairly heterogeneous and many operate using faulty heuristics. We discover that one of these heuristics can be exploited via a manually designed adversarial attack which we call The California Attack. Our results highlight some roadblocks standing in the way of being able to successfully interpret advanced ML systems. Project website available at https://forbiddenfacts.github.io .