# Split, Unlearn, Merge: Leveraging Data Attributes for More Effective Unlearning in LLMs

arXiv: https://arxiv.org/abs/2406.11780 · HTML: https://mlsanity.com/paper/2406.11780/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 38 of its 52 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 30 of 38)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks](https://mlsanity.com/paper/1911.04933/) (2019)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Improving alignment of dialogue agents via targeted human judgements](https://mlsanity.com/paper/2209.14375/) (2022)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://mlsanity.com/paper/2404.05868/) (2024)
- [Making AI Forget You: Data Deletion in Machine Learning](https://mlsanity.com/paper/1907.05012/) (2019)
- [DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts](https://mlsanity.com/paper/2105.03023/) (2021)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification](https://mlsanity.com/paper/1903.04561/) (2019)
- [Amnesiac Machine Learning](https://mlsanity.com/paper/2010.10981/) (2020)
- [Are aligned neural networks adversarially aligned?](https://mlsanity.com/paper/2306.15447/) (2023)

## Built on by (most cited first, 18 of 18)

- [Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning](https://mlsanity.com/paper/2410.07163/) (2024)
- [An Adversarial Perspective on Machine Unlearning for AI Safety](https://mlsanity.com/paper/2409.18025/) (2024)
- [Exact Unlearning of Finetuning Data via Model Merging at Scale](https://mlsanity.com/paper/2504.04626/) (2025)
- [Continual Unlearning for Text-to-Image Diffusion Models: A Regularization Perspective](https://mlsanity.com/paper/2511.07970/) (2025)
- [UCD: Unlearning in LLMs via Contrastive Decoding](https://mlsanity.com/paper/2506.12097/) (2025)
- [Recent Advances in Attack and Defense Approaches of Large Language Models](https://mlsanity.com/paper/2409.03274/) (2024)
- [Mitigating Biases in Language Models via Bias Unlearning](https://mlsanity.com/paper/2509.25673/) (2025)
- [A Survey on Unlearning in Large Language Models](https://mlsanity.com/paper/2510.25117/) (2025)
- [Visual-Guided Key-Token Regularization for Multimodal Large Language Model Unlearning](https://mlsanity.com/paper/2601.22020/) (2026)
- [ZJUKLAB at SemEval-2025 Task 4: Unlearning via Model Merging](https://mlsanity.com/paper/2503.21088/) (2025)
- [Ready2Unlearn: A Learning-Time Approach for Preparing Models with Future Unlearning Readiness](https://mlsanity.com/paper/2505.10845/) (2025)
- [Revisiting the Past: Data Unlearning with Model State History](https://mlsanity.com/paper/2506.20941/) (2025)
- [Per-parameter Task Arithmetic for Unlearning in Large Language Models](https://mlsanity.com/paper/2601.22030/) (2026)
- [Layered Unlearning for Adversarial Relearning](https://mlsanity.com/paper/2505.09500/) (2025)
- [Deep Model Merging: The Sister of Neural Network Interpretability -- A Survey](https://mlsanity.com/paper/2410.12927/) (2024)
- [Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques](https://mlsanity.com/paper/2507.13629/) (2025)
- [EvoMU: Evolutionary Machine Unlearning](https://mlsanity.com/paper/2602.02139/) (2026)
- [Model Unlearning Objectives Vary for Distinct Language Functions](https://mlsanity.com/paper/2605.26454/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown to pose social and ethical risks such as generating toxic language or facilitating malicious use of hazardous knowledge. Machine unlearning is a promising approach to improve LLM safety by directly removing harmful behaviors and knowledge. In this paper, we propose "SPlit, UNlearn, MerGE" (SPUNGE), a framework that can be used with any unlearning method to amplify its effectiveness. SPUNGE leverages data attributes during unlearning by splitting unlearning data into subsets based on specific attribute values, unlearning each subset separately, and merging the unlearned models. We empirically demonstrate that SPUNGE significantly improves the performance of two recent unlearning methods on state-of-the-art LLMs while maintaining their general capabilities on standard academic benchmarks.