mlsanityData to 7 Oct 2026

Dissecting Language Models: Machine Unlearning via Selective Pruning

arXiv 2403.01267 ↗First posted 2 Mar 2024cs.LG cs.CLcited by 38

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Understanding and shaping the behaviour of Large Language Models (LLMs) is increasingly important as applications become more powerful and more frequently adopted. This paper introduces a machine unlearning method specifically designed for LLMs. We introduce a selective pruning method for LLMs that removes neurons based on their relative importance on a targeted capability compared to overall network performance. This approach is a compute- and data-efficient method for identifying and removing neurons that enable specific behaviours. Our findings reveal that both feed-forward and attention neurons in LLMs are specialized; that is, for specific tasks, certain neurons are more crucial than others. Code from all experiments is available at https://github.com/nickypro/selective-pruning

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 37 of its 47 references that are in the map. First posted on arXiv 2 Mar 2024.

Builds on

37 of 47 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening15 Aug
  2. 2023Towards Unbounded Machine Unlearning20 Feb
  3. 2022A Survey of Machine Unlearning6 Sep
  4. 2022Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher17 May
  5. 2022Markov Chain Monte Carlo-Based Machine Unlearning: Unlearning What Needs to be Forgotten28 Feb
  6. 2022Zero-Shot Machine Unlearning14 Jan
  7. 2021Fast Yet Effective Machine Unlearning17 Nov
  8. 2021Unrolling SGD: Understanding Factors Influencing Machine Unlearning27 Sep

Other references · 29

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  11. 2022Editing Models with Task Arithmeticcited by 1,076
  12. 2019Machine Unlearningcited by 1,020
17 more
  1. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  2. 2023Steering Language Models With Activation Engineeringcited by 824
  3. 2019Are Sixteen Heads Really Better than One?cited by 803
  4. 2022Galactica: A Large Language Model for Sciencecited by 736
  5. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  6. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  7. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  8. 2020What is the State of Neural Network Pruning?cited by 578
  9. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  10. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  11. 2023Unified Concept Editing in Diffusion Modelscited by 377
  12. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  13. 2020Amnesiac Machine Learningcited by 305
  14. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  15. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  16. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  17. 2020Learn to Forget: Machine Unlearning via Neuron Maskingcited by 56

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 19; 2026: 10.

38 papers.

Sort:

By area:

  1. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  2. 2024Machine Unlearning in Generative AI: A Surveycited by 49
  3. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 39
  4. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  5. 2024To Forget or Not? Towards Practical Knowledge Unlearning for Large Language Modelscited by 25
  6. 2024Dissecting Fine-Tuning Unlearning in Large Language Modelscited by 25
  7. 2025Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Modelscited by 25
  8. 2025Model Unlearning via Sparse Autoencoder Subspace Guided Projectionscited by 17
  9. 2024Mitigating Selection Bias with Node Pruning and Auxiliary Optionscited by 12
  10. 2025SoK: Machine Unlearning for Large Language Modelscited by 9
  11. 2024AND: Audio Network Dissection for Interpreting Deep Acoustic Modelscited by 7
  12. 2025SPIRIT: Patching Speech Language Models against Jailbreak Attackscited by 7
26 more
  1. 2024Forget Vectors at Play: Universal Input Perturbations Driving Machine Unlearning in Image Classificationcited by 6
  2. 2025ExAnte: A Benchmark for Ex-Ante Inference in Large Language Modelscited by 5
  3. 2025Module-Aware Parameter-Efficient Machine Unlearning on Transformerscited by 3
  4. 2026Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Typescited by 3
  5. 2026Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluationcited by 3
  6. 2025Pruning as a Defense: Reducing Memorization in Large Language Modelscited by 2
  7. 2025Exploring Incremental Unlearning: Techniques, Challenges, and Future Directionscited by 2
  8. 2025A Survey on Unlearning in Large Language Modelscited by 2
  9. 2025No Encore: Unlearning as Opt-Out in Music Generationcited by 1
  10. 2025Leverage Unlearning to Sanitize LLMscited by 1
  11. 2025ZK-APEX: Zero-Knowledge Approximate Personalized Unlearning with Executable Proofscited by 1
  12. 2025Robust MLLM Unlearning via Visual Knowledge Distillationcited by 1
  13. 2026Beyond Forgetting: Representation Misdirection Elicits Controllable Side Behaviors and Capabilitiescited by 1
  14. 2026Roots Beneath the Cut: Uncovering the Risk of Concept Revival in Pruning-Based Unlearning for Diffusion Modelscited by 1
  15. 2024Modularity in Transformers: Investigating Neuron Separability & Specializationcited by 0
  16. 2025Cyber for AI at SemEval-2025 Task 4: Forgotten but Not Lost: The Balancing Act of Selective Unlearning in Large Language Modelscited by 0
  17. 2025Sky of Unlearning (SoUL): Rewiring Federated Machine Unlearning via Selective Pruningcited by 0
  18. 2025Machine Unlearning for Robust DNNs: Attribution-Guided Partitioning and Neuron Pruning in Noisy Environmentscited by 0
  19. 2025Verifiable Unlearning on Edgecited by 0
  20. 2025TDHook: A Lightweight Framework for Interpretabilitycited by 0
  21. 2026Visualising Information Flow in Word Embeddings with Diffusion Tensor Imagingcited by 0
  22. 2026Relationship-Aware Safety Unlearning for Multimodal LLMscited by 0
  23. 2026Exploring the Limits of Pruning: Task-Specific Neurons, Model Collapse, and Recovery in Task-Specific Large Language Modelscited by 0
  24. 2026ICCU: In-Context Continual Unlearning via Pattern-Induced Refusal Rulescited by 0
  25. 2026Distance Is Not Enough: Forget-Retain Alignment Gap Predicts LLM Relearning Robustnesscited by 0
  26. 2026Cascade: Hierarchical Recoverability Control for Large Language Model Unlearningcited by 0