# Effective Skill Unlearning through Intervention and Abstention

arXiv: https://arxiv.org/abs/2503.21730 · HTML: https://mlsanity.com/paper/2503.21730/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 27 Mar 2025.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 19 of 19)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [Who's Harry Potter? Approximate Unlearning in LLMs](https://mlsanity.com/paper/2310.02238/) (2023)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [Understanding the Role of Individual Units in a Deep Neural Network](https://mlsanity.com/paper/2009.05041/) (2020)
- [Quark: Controllable Text Generation with Reinforced Unlearning](https://mlsanity.com/paper/2205.13636/) (2022)
- [In-Context Unlearning: Language Models as Few Shot Unlearners](https://mlsanity.com/paper/2310.07579/) (2023)
- [Unlearn What You Want to Forget: Efficient Unlearning for LLMs](https://mlsanity.com/paper/2310.20150/) (2023)
- [CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks](https://mlsanity.com/paper/2204.10965/) (2022)
- [Natural Language Descriptions of Deep Visual Features](https://mlsanity.com/paper/2201.11114/) (2022)
- [KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignment](https://mlsanity.com/paper/2305.06535/) (2023)
- [The Importance of Prompt Tuning for Automated Neuron Explanations](https://mlsanity.com/paper/2310.06200/) (2023)

## Built on by (most cited first, 9 of 9)

- [LLM Agents Already Know When to Call Tools -- Even Without Reasoning](https://mlsanity.com/paper/2605.09252/) (2026)
- [SoK: Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2506.09227/) (2025)
- [Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training](https://mlsanity.com/paper/2605.06076/) (2026)
- [ReflCtrl: Controlling LLM Reflection Efficiently via Representation Engineering](https://mlsanity.com/paper/2512.13979/) (2025)
- [Auditing Language Model Unlearning via Information Decomposition](https://mlsanity.com/paper/2601.15111/) (2026)
- [Reveal and Release: Iterative LLM Unlearning with Self-generated Data](https://mlsanity.com/paper/2509.14624/) (2025)
- [Steer2Edit: From Activation Steering to Component-Level Editing](https://mlsanity.com/paper/2602.09870/) (2026)
- [Layer-Targeted Multilingual Knowledge Erasure in Large Language Models](https://mlsanity.com/paper/2602.22562/) (2026)
- [Suppression Sticks, Locality Is Fragile: A Closed-Loop Target-and-Control Audit of Task-Vector Negation in VLA Policies](https://mlsanity.com/paper/2608.04692/) (2026)

## Abstract (arXiv)

Large language Models (LLMs) have demonstrated remarkable skills across various domains. Understanding the mechanisms behind their abilities and implementing controls over them is becoming increasingly important for developing better models. In this paper, we focus on skill unlearning in LLMs, specifically unlearning a particular skill while retaining their overall capabilities. We introduce two lightweight, training-free machine skill unlearning techniques for LLMs. First, we observe that the pre-activation distribution of neurons in each Feed-Forward Layer (FFL) differs when the model demonstrates different skills. Additionally, we find that queries triggering the same skill cluster within the FFL key space and can be separated from other queries using a hypercube. Based on these observations, we propose two lightweight, training-free skill unlearning methods via textitintervention and textitabstention respectively: textttNeuron Adjust and textttKey Space Detection. We evaluate our methods on unlearning math-solving, Python-coding, and comprehension skills across seven different languages. The results demonstrate their strong unlearning capabilities for the designated skills. Specifically, textttKey Space Detection achieves over 80% relative performance drop on the forgetting skill and less than 10% relative performance drop on other skills and the model's general knowledge (MMLU) for most unlearning tasks. Our code is available at https://github.com/Trustworthy-ML-Lab/effectiveskillunlearning