# Step-by-Step Reasoning Attack: Revealing 'Erased' Knowledge in Large Language Models

arXiv: https://arxiv.org/abs/2506.17279 · HTML: https://mlsanity.com/paper/2506.17279/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 14 Jun 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 24 of 24)

- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://mlsanity.com/paper/2404.05868/) (2024)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [In-Context Unlearning: Language Models as Few Shot Unlearners](https://mlsanity.com/paper/2310.07579/) (2023)
- [Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs](https://mlsanity.com/paper/2407.15549/) (2024)
- [Eight Methods to Evaluate Robust Unlearning in LLMs](https://mlsanity.com/paper/2402.16835/) (2024)
- [Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks](https://mlsanity.com/paper/2309.17410/) (2023)
- [Tamper-Resistant Safeguards for Open-Weight LLMs](https://mlsanity.com/paper/2408.00761/) (2024)
- [An Adversarial Perspective on Machine Unlearning for AI Safety](https://mlsanity.com/paper/2409.18025/) (2024)
- [Machine Unlearning of Pre-trained Large Language Models](https://mlsanity.com/paper/2402.15159/) (2024)
- [Guardrail Baselines for Unlearning in LLMs](https://mlsanity.com/paper/2403.03329/) (2024)
- [Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning](https://mlsanity.com/paper/2406.13356/) (2024)
- [DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Models](https://mlsanity.com/paper/2310.20138/) (2023)
- [Do Unlearning Methods Remove Information from Language Model Weights?](https://mlsanity.com/paper/2410.08827/) (2024)
- [A Closer Look at Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2410.08109/) (2024)
- [UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models](https://mlsanity.com/paper/2402.10052/) (2024)
- [Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localization](https://mlsanity.com/paper/2410.12949/) (2024)
- [On Effects of Steering Latent Representation for Large Language Model Unlearning](https://mlsanity.com/paper/2408.06223/) (2024)
- [Intrinsic Test of Unlearning Using Parametric Knowledge Traces](https://mlsanity.com/paper/2406.11614/) (2024)
- [Large Language Models Relearn Removed Concepts](https://mlsanity.com/paper/2401.01814/) (2024)
- [WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models](https://mlsanity.com/paper/2410.17509/) (2024)
- [RKLD: Reverse KL-Divergence-based Knowledge Distillation for Unlearning Personal Information in Large Language Models](https://mlsanity.com/paper/2406.01983/) (2024)
- [UnStar: Unlearning with Self-Taught Anti-Sample Reasoning for LLMs](https://mlsanity.com/paper/2410.17050/) (2024)

## Built on by (most cited first, 6 of 6)

- [A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models](https://mlsanity.com/paper/2509.03871/) (2025)
- [Auditing Reasoning-Trace Memorization Claims after Unlearning with Head-Conditioned Canaries](https://mlsanity.com/paper/2605.18891/) (2026)
- [Downgrade to Upgrade: Optimizer Simplification Enhances Robustness in LLM Unlearning](https://mlsanity.com/paper/2510.00761/) (2025)
- [Rethinking Reasoning: A Survey on Reasoning-based Backdoors in LLMs](https://mlsanity.com/paper/2510.07697/) (2025)
- [Copyright Detective: A Forensic System to Evidence LLMs Flickering Copyright Leakage Risks](https://mlsanity.com/paper/2602.05252/) (2026)
- [Extracting Forgotten Prompts from Targeted Unlearned Models](https://mlsanity.com/paper/2609.03662/) (2026)

## Abstract (arXiv)

Knowledge erasure in large language models (LLMs) is important for ensuring compliance with data and AI regulations, safeguarding user privacy, mitigating bias, and misinformation. Existing unlearning methods aim to make the process of knowledge erasure more efficient and effective by removing specific knowledge while preserving overall model performance, especially for retained information. However, it has been observed that the unlearning techniques tend to suppress and leave the knowledge beneath the surface, thus making it retrievable with the right prompts. In this work, we demonstrate that textitstep-by-step reasoning can serve as a backdoor to recover this hidden information. We introduce a step-by-step reasoning-based black-box attack, Sleek, that systematically exposes unlearning failures. We employ a structured attack framework with three core components: (1) an adversarial prompt generation strategy leveraging step-by-step reasoning built from LLM-generated queries, (2) an attack mechanism that successfully recalls erased content, and exposes unfair suppression of knowledge intended for retention and (3) a categorization of prompts as direct, indirect, and implied, to identify which query types most effectively exploit unlearning weaknesses. Through extensive evaluations on four state-of-the-art unlearning techniques and two widely used LLMs, we show that existing approaches fail to ensure reliable knowledge removal. Of the generated adversarial prompts, 62.5% successfully retrieved forgotten Harry Potter facts from WHP-unlearned Llama, while 50% exposed unfair suppression of retained knowledge. Our work highlights the persistent risks of information leakage, emphasizing the need for more robust unlearning strategies for erasure.