mlsanityData to 14 Oct 2026

A mean teacher algorithm for unlearning of language models

arXiv 2504.13388 ↗First posted 18 Apr 2025cs.LG cs.CLcited by 0

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

One of the goals of language model unlearning is to reduce memorization of selected text instances while retaining the model's general abilities. Despite various proposed methods, reducing memorization of large datasets without noticeable degradation in model utility remains challenging. In this paper, we investigate the mean teacher algorithm (Tarvainen & Valpola, 2017), a simple proximal optimization method from continual learning literature that gradually modifies the teacher model. We show that the mean teacher can approximate a trajectory of a slow natural gradient descent (NGD), which inherently seeks low-curvature updates that are less likely to degrade the model utility. While slow NGD can suffer from vanishing gradients, we introduce a new unlearning loss called "negative log-unlikelihood" (NLUL) that avoids this problem. We show that the combination of mean teacher and NLUL improves some metrics on the MUSE benchmarks (Shi et al., 2024).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 28 of its 43 references that are in the map. First posted on arXiv 18 Apr 2025.

Builds on

28 of 43 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate29 Oct
  2. 2024LLM Unlearning via Loss Adjustment with Only Forget Data14 Oct
  3. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
  4. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progress3 Oct
  5. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
  6. 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference12 Jun
  7. 2024Large Language Model Unlearning via Embedding-Corrupted Prompts12 Jun
  8. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  9. 2024Guardrail Baselines for Unlearning in LLMs5 Mar
  10. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
  11. 2023Large Language Model Unlearning14 Oct
  12. 2023Who's Harry Potter? Approximate Unlearning in LLMs3 Oct
2 more
  1. 2022Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher17 May
  2. 2021Unrolling SGD: Understanding Factors Influencing Machine Unlearning27 Sep

Other references · 14

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2017Decoupled Weight Decay Regularizationcited by 1,831
  4. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  5. 2019Machine Unlearningcited by 1,061
  6. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  7. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 465
  8. 2014New insights and perspectives on the natural gradient methodcited by 443
  9. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  10. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  11. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
  12. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 117
2 more
  1. 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
  2. 2024Revisiting inverse Hessian vector products for calculating influence functionscited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models
  2. 2025SoK: Machine Unlearning for Large Language Models
  3. 2024Multi-Objective Large Language Model Unlearning
  4. 2025UniErase: Towards Balanced and Precise Unlearning in Language Models
  5. 2025Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs
  6. 2025BLUR: A Bi-Level Optimization Approach for LLM Unlearning
  7. 2026FIT to Forget: Robust Continual Unlearning for Large Language Models
  8. 2026Modeling LLM Unlearning as an Asymmetric Two-Task Learning Problem