# Extracting Unlearned Information from LLMs with Activation Steering

arXiv: https://arxiv.org/abs/2411.02631 · HTML: https://mlsanity.com/paper/2411.02631/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 21 of its 25 references that are in the map.
- First posted on arXiv 4 Nov 2024.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)
- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 21 of 21)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [Who's Harry Potter? Approximate Unlearning in LLMs](https://mlsanity.com/paper/2310.02238/) (2023)
- [Linear Representations of Sentiment in Large Language Models](https://mlsanity.com/paper/2310.15154/) (2023)
- [Eight Methods to Evaluate Robust Unlearning in LLMs](https://mlsanity.com/paper/2402.16835/) (2024)
- [Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks](https://mlsanity.com/paper/2309.17410/) (2023)
- [JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs](https://mlsanity.com/paper/2402.05668/) (2024)
- [Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space](https://mlsanity.com/paper/2402.09063/) (2024)
- [DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Models](https://mlsanity.com/paper/2310.20138/) (2023)
- [Right to be Forgotten in the Era of Large Language Models: Implications, Challenges, and Solutions](https://mlsanity.com/paper/2307.03941/) (2023)
- [A Language Model's Guide Through Latent Space](https://mlsanity.com/paper/2402.14433/) (2024)
- [On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused?](https://mlsanity.com/paper/2310.01581/) (2023)

## Built on by (most cited first, 21 of 21)

- [OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics](https://mlsanity.com/paper/2506.12618/) (2025)
- [A Comprehensive Survey of Machine Unlearning Techniques for Large Language Models](https://mlsanity.com/paper/2503.01854/) (2025)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [The Rogue Scalpel: Activation Steering Compromises LLM Safety](https://mlsanity.com/paper/2509.22067/) (2025)
- [LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarks](https://mlsanity.com/paper/2504.10185/) (2025)
- [Distillation Robustifies Unlearning](https://mlsanity.com/paper/2506.06278/) (2025)
- [Silenced Biases: The Dark Side LLMs Learned to Refuse](https://mlsanity.com/paper/2511.03369/) (2025)
- [Improving LLM Unlearning Robustness via Random Perturbations](https://mlsanity.com/paper/2501.19202/) (2025)
- [Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMs](https://mlsanity.com/paper/2507.04219/) (2025)
- [Evaluate Bias without Manual Test Sets: A Concept Representation Perspective for LLMs](https://mlsanity.com/paper/2505.15524/) (2025)
- [Forgetting to Forget: Attention Sink as A Gateway for Backdooring LLM Unlearning](https://mlsanity.com/paper/2510.17021/) (2025)
- [A Survey on Unlearning in Large Language Models](https://mlsanity.com/paper/2510.25117/) (2025)
- [Align-then-Unlearn: Embedding Alignment for LLM Unlearning](https://mlsanity.com/paper/2506.13181/) (2025)
- [Position: The Term "Machine Unlearning" Is Overused in LLMs](https://mlsanity.com/paper/2606.27379/) (2026)
- [The Safety Gap Toolkit: Evaluating Hidden Dangers of Open-Source Models](https://mlsanity.com/paper/2507.11544/) (2025)
- [AUVIC: Adversarial Unlearning of Visual Concepts for Multi-modal Large Language Models](https://mlsanity.com/paper/2511.11299/) (2025)
- [Reflections and New Directions for Human-Centered Large Language Models](https://mlsanity.com/paper/2605.06901/) (2026)
- [Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility](https://mlsanity.com/paper/2606.03291/) (2026)
- [Measure, Don't Optimize: Forecasting Recovery in LLM Unlearning](https://mlsanity.com/paper/2608.11408/) (2026)
- [Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it](https://mlsanity.com/paper/2609.36612/) (2026)
- [Mitigating Private Data Leakage in LLMs with Whiteout](https://mlsanity.com/paper/2610.02418/) (2026)

## Abstract (arXiv)

An unintended consequence of the vast pretraining of Large Language Models (LLMs) is the verbatim memorization of fragments of their training data, which may contain sensitive or copyrighted information. In recent years, unlearning has emerged as a solution to effectively remove sensitive knowledge from models after training. Yet, recent work has shown that supposedly deleted information can still be extracted by malicious actors through various attacks. Still, current attacks retrieve sets of possible candidate generations and are unable to pinpoint the output that contains the actual target information. We propose activation steering as a method for exact information retrieval from unlearned LLMs. We introduce a novel approach to generating steering vectors, named Anonymized Activation Steering. Additionally, we develop a simple word frequency method to pinpoint the correct answer among a set of candidates when retrieving unlearned information. Our evaluation across multiple unlearning techniques and datasets demonstrates that activation steering successfully recovers general knowledge (e.g., widely known fictional characters) while revealing limitations in retrieving specific information (e.g., details about non-public individuals). Overall, our results demonstrate that exact information retrieval from unlearned models is possible, highlighting a severe vulnerability of current unlearning techniques.