# LLM Unlearning Evaluation with TRIAGE

arXiv: https://arxiv.org/abs/2609.32103 · HTML: https://mlsanity.com/paper/2609.32103/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 21 of its 60 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 21 of 21)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://mlsanity.com/paper/2404.05868/) (2024)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [Eight Methods to Evaluate Robust Unlearning in LLMs](https://mlsanity.com/paper/2402.16835/) (2024)
- [Do Unlearning Methods Remove Information from Language Model Weights?](https://mlsanity.com/paper/2410.08827/) (2024)
- [UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AI](https://mlsanity.com/paper/2407.00106/) (2024)
- [Are we making progress in unlearning? Findings from the first NeurIPS unlearning competition](https://mlsanity.com/paper/2406.09073/) (2024)
- [To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Models](https://mlsanity.com/paper/2405.03097/) (2024)
- [Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs](https://mlsanity.com/paper/2505.16831/) (2025)
- [On Effects of Steering Latent Representation for Large Language Model Unlearning](https://mlsanity.com/paper/2408.06223/) (2024)
- [Second-Order Information Matters: Revisiting Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2403.10557/) (2024)
- [BLUR: A Benchmark for LLM Unlearning Robust to Forget-Retain Overlap](https://mlsanity.com/paper/2506.15699/) (2025)
- [Improving LLM Unlearning Robustness via Random Perturbations](https://mlsanity.com/paper/2501.19202/) (2025)
- [Existing Large Language Model Unlearning Evaluations Are Inconclusive](https://mlsanity.com/paper/2506.00688/) (2025)
- [Is your algorithm unlearning or untraining?](https://mlsanity.com/paper/2604.07962/) (2026)
- [Align-then-Unlearn: Embedding Alignment for LLM Unlearning](https://mlsanity.com/paper/2506.13181/) (2025)

## Abstract (arXiv)

Large language models can memorize private or harmful information, motivating machine unlearning methods that remove targeted knowledge while preserving other capabilities. However, existing evaluations rely primarily on behavioral benchmarks, which assess whether a model appears to forget but provide limited insight into how unlearning changes the model or affects related knowledge. We introduce textitTRIAGE (textitTripartite Representation-internal Introspection for Adjacency Gap Evaluation), a benchmark-agnostic evaluation framework for characterizing these changes. TRIAGE uses diagonal approximations of the Fisher information and Hessian to measure changes in parameter sensitivity and local curvature, and utilizes a Forget / Adjacent-Retain / Generic-Retain partition to quantify an adjacency gap in semantically related knowledge. Based on the magnitude and distribution of these changes, TRIAGE further classifies each algorithm's update as no-op, partially localized, collateral dominant, or globally destructive. Across 12 unlearning methods, four language models, and the WMDP, TOFU, and MUSE benchmarks, we find that methods with similar behavioral forgetting can produce substantially different internal changes and patterns of collateral damage. These signatures also vary across models and benchmarks, indicating that the effects of unlearning are not determined solely by the unlearning algorithm. TRIAGE can be applied alongside existing unlearning benchmarks to complement behavioral evaluation with a model-internal view of how unlearning reshapes the model's parameter space and affects retained knowledge.