# ZJUKLAB at SemEval-2025 Task 4: Unlearning via Model Merging

arXiv: https://arxiv.org/abs/2503.21088 · HTML: https://mlsanity.com/paper/2503.21088/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 27 Mar 2025.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 25 of 25)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://mlsanity.com/paper/2404.05868/) (2024)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [Who's Harry Potter? Approximate Unlearning in LLMs](https://mlsanity.com/paper/2310.02238/) (2023)
- [Quark: Controllable Text Generation with Reinforced Unlearning](https://mlsanity.com/paper/2205.13636/) (2022)
- [Unlearn What You Want to Forget: Efficient Unlearning for LLMs](https://mlsanity.com/paper/2310.20150/) (2023)
- [Towards Safer Large Language Models through Machine Unlearning](https://mlsanity.com/paper/2402.10058/) (2024)
- [Model merging with SVD to tie the Knots](https://mlsanity.com/paper/2410.19735/) (2024)
- [DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling](https://mlsanity.com/paper/2406.11617/) (2024)
- [Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond](https://mlsanity.com/paper/2502.05374/) (2025)
- [AnyEdit: Edit Any Knowledge Encoded in Language Models](https://mlsanity.com/paper/2502.05628/) (2025)
- [Position: LLM Unlearning Benchmarks are Weak Measures of Progress](https://mlsanity.com/paper/2410.02879/) (2024)
- [SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)](https://mlsanity.com/paper/2406.17975/) (2024)
- [Open Problems in Machine Unlearning for AI Safety](https://mlsanity.com/paper/2501.04952/) (2025)
- [Erasing Conceptual Knowledge from Language Models](https://mlsanity.com/paper/2410.02760/) (2024)
- [Learning to Refuse: Towards Mitigating Privacy Risks in LLMs](https://mlsanity.com/paper/2407.10058/) (2024)
- [LUME: LLM Unlearning with Multitask Evaluations](https://mlsanity.com/paper/2502.15097/) (2025)
- [Split, Unlearn, Merge: Leveraging Data Attributes for More Effective Unlearning in LLMs](https://mlsanity.com/paper/2406.11780/) (2024)
- [MEOW: MEMOry Supervised LLM Unlearning Via Inverted Facts](https://mlsanity.com/paper/2409.11844/) (2024)
- [UnStar: Unlearning with Self-Taught Anti-Sample Reasoning for LLMs](https://mlsanity.com/paper/2410.17050/) (2024)
- [Unveiling Entity-Level Unlearning for Large Language Models: A Comprehensive Analysis](https://mlsanity.com/paper/2406.15796/) (2024)
- [On the Limitations and Prospects of Machine Unlearning for Generative AI](https://mlsanity.com/paper/2408.00376/) (2024)

## Built on by (most cited first, 2 of 2)

- [Agentic Unlearning: When LLM Agent Meets Machine Unlearning](https://mlsanity.com/paper/2602.17692/) (2026)
- [Mitigating Biases in Language Models via Bias Unlearning](https://mlsanity.com/paper/2509.25673/) (2025)

## Abstract (arXiv)

This paper presents the ZJUKLAB team's submission for SemEval-2025 Task 4: Unlearning Sensitive Content from Large Language Models. This task aims to selectively erase sensitive knowledge from large language models, avoiding both over-forgetting and under-forgetting issues. We propose an unlearning system that leverages Model Merging (specifically TIES-Merging), combining two specialized models into a more balanced unlearned model. Our system achieves competitive results, ranking second among 26 teams, with an online score of 0.944 for Task Aggregate and 0.487 for overall Aggregate. In this paper, we also conduct local experiments and perform a comprehensive analysis of the unlearning process, examining performance trajectories, loss dynamics, and weight perspectives, along with several supplementary experiments, to understand the effectiveness of our method. Furthermore, we analyze the shortcomings of our method and evaluation metrics, emphasizing that MIA scores and ROUGE-based metrics alone are insufficient to fully evaluate successful unlearning. Finally, we emphasize the need for more comprehensive evaluation methodologies and rethinking of unlearning objectives in future research. Code is available at https://github.com/zjunlp/unlearn/tree/main/semeval25.