# Leak-Resistant Unlearning: A New Benchmark for Evaluating Multi-Hop Reasoning Consistency and Recovery Robustness

arXiv: https://arxiv.org/abs/2608.04519 · HTML: https://mlsanity.com/paper/2608.04519/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 16 of its 38 references that are in the map.
- First posted on arXiv 5 Aug 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 16 of 16)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://mlsanity.com/paper/2404.05868/) (2024)
- [MUSE: Machine Unlearning Six-Way Evaluation for Language Models](https://mlsanity.com/paper/2407.06460/) (2024)
- [UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AI](https://mlsanity.com/paper/2407.00106/) (2024)
- [The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigation](https://mlsanity.com/paper/2507.05578/) (2025)
- [Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearning](https://mlsanity.com/paper/2601.03190/) (2026)
- [Breaking Chains: Unraveling the Links in Multi-Hop Knowledge Unlearning](https://mlsanity.com/paper/2410.13274/) (2024)
- [Understanding the Dilemma of Unlearning for Large Language Models](https://mlsanity.com/paper/2509.24675/) (2025)
- [CAP: Controllable Alignment Prompting for Unlearning in LLMs](https://mlsanity.com/paper/2604.21251/) (2026)
- [What the "Spotless" Mind Remembers: How Knowledge Entanglement Shapes What Leaks After Unlearning in LLMs](https://mlsanity.com/paper/2510.25732/) (2025)
- [GONE: Structural Knowledge Unlearning via Neighborhood-Expanded Distribution Shaping](https://mlsanity.com/paper/2603.12275/) (2026)

## Abstract (arXiv)

Benchmarking machine unlearning methods is critical to understand whether sensitive knowledge is removed from large language models (LLMs) or not. Current unlearning benchmarks include mainly single-hop questions and a narrow set of multi-hop questions. Although effective, they still face two challenges. (1) Knowledge is not isolated, whereby diverse multi-hop reasoning paths can potentially induce knowledge leakage than normal queries. (2) Unlearning may be fragile: unlearned knowledge can be partially recovered through recovery attacks such as lightweight post-unlearning adaptation, making static evaluation insufficient. Therefore, in this paper, we introduce unlearning as a novel benchmark to understand robust LLM knowledge removal across diverse reasoning paths and recovery attacks. We experiment with this benchmark on 3 models, 6 unlearning methods, and 2 carefully curated datasets. Results show that existing methods are vulnerable to multi-hop reasoning paths and recovery attacks. We further explore the trade-off among forget quality, robustness, and model utility for LLM unlearning.