mlsanityData to 14 Oct 2026

LLM Unlearning with LLM Beliefs

arXiv 2510.19422 ↗First posted 22 Oct 2025cs.LG cs.CLcited by 17

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Large language models trained on vast corpora inherently risk memorizing sensitive or harmful content, which may later resurface in their outputs. Prevailing unlearning methods generally rely on gradient ascent and its variants to lower the probability of specific target responses. However, we find that this strategy induces a critical side effect: probability mass is redistributed into high-likelihood regions, often corresponding to semantically related rephrasings of the targets. We refer to this as the squeezing effect, which explains why many methods yield merely spurious unlearning, a problem further obscured by automated metrics (e.g., ROUGE, truth ratio) that misreport actual success. To address this, we propose a bootstrapping (BS) framework that explicitly links the squeezing effect with the model's own high-confidence generations, namely its model beliefs. Since model beliefs inherently capture the very high-likelihood regions where probability mass is squeezed, incorporating them into the unlearning objective directly counters the squeezing effect. By jointly suppressing both target responses and model beliefs, BS-T (token) attenuates high-probability tokens, whereas BS-S (sequence) removes entire high-confidence generations, together achieving more thorough forgetting while preserving utility. Extensive experiments across diverse benchmarks with various model families confirm the effectiveness of our approach.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 65 of its 83 references that are in the map. First posted on arXiv 22 Oct 2025.

Builds on

65 of 83 references.

Closest ancestors · 33

Same direction, newest first

  1. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics14 Jun
  2. 2025Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awareness6 Jun
  3. 2025Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy31 May
  4. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning17 May
  5. 2025GRU: Mitigating the Trade-off between Unlearning and Retention for LLMs12 Mar
  6. 2025Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond26 Feb
  7. 2025A Comprehensive Survey of Machine Unlearning Techniques for Large Language Models22 Feb
  8. 2025LLM Unlearning via Neural Activation Redirection11 Feb
  9. 2024LLM Unlearning via Loss Adjustment with Only Forget Data14 Oct
  10. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
  11. 2024Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models20 Sep
  12. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
21 more
  1. 2024Soft Prompting for Unlearning in Large Language Models17 Jun
  2. 2024Towards Effective Evaluations and Comparisons for LLM Unlearning Methods13 Jun
  3. 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference12 Jun
  4. 2024Decoupling the Class Label and the Target Concept in Machine Unlearning12 Jun
  5. 2024Large Language Model Unlearning via Embedding-Corrupted Prompts12 Jun
  6. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning28 Apr
  7. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  8. 2024Eight Methods to Evaluate Robust Unlearning in LLMs26 Feb
  9. 2024UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models15 Feb
  10. 2024Rethinking Machine Unlearning for Large Language Models13 Feb
  11. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
  12. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMs31 Oct
  13. 2023Large Language Model Unlearning14 Oct
  14. 2023In-Context Unlearning: Language Models as Few Shot Unlearners11 Oct
  15. 2023Who's Harry Potter? Approximate Unlearning in LLMs3 Oct
  16. 2023Machine Unlearning: Solutions and Challenges14 Aug
  17. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Models4 Oct
  18. 2022Algorithms that Approximate Data Removal: New Results and Limitations25 Sep
  19. 2022Deep Unlearning via Randomized Conditionally Independent Hessians15 Apr
  20. 2021On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning22 Oct
  21. 2021Remember What You Want to Forget: Algorithms for Machine Unlearning4 Mar

Other references · 32

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  8. 2019Machine Unlearningcited by 1,061
  9. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  10. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 618
  11. 2019Certified Data Removal from Machine Learning Modelscited by 535
  12. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
20 more
  1. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  2. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 465
  3. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  4. 2020DeltaGrad: Rapid retraining of machine learning modelscited by 149
  5. 2021Graph Unlearningcited by 138
  6. 2020Machine Unlearning for Random Forestscited by 131
  7. 2024Learning Dynamics of LLM Finetuningcited by 95
  8. 2022Recommendation Unlearningcited by 80
  9. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 67
  10. 2024Large Language Model Safety: A Holistic Surveycited by 59
  11. 2024Machine Unlearning for Image-to-Image Generative Modelscited by 47
  12. 2022Harnessing Out-Of-Distribution Examples via Augmenting Content and Stylecited by 43
  13. 2024Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?cited by 41
  14. 2023Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learningcited by 27
  15. 2025Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Modelscited by 25
  16. 2024Score Forgetting Distillation: A Swift, Data-Free Method for Machine Unlearning in Diffusion Modelscited by 23
  17. 2025From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?cited by 23
  18. 2025EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forgetcited by 14
  19. 2025MTL-UE: Learning to Learn Nothing for Multi-Task Learningcited by 11
  20. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignmentcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 16.

17 papers.

Sort:

By area:

  1. 2026Contrastive Representation Shaping for LLM Unlearningcited by 7
  2. 2026Explainable LLM Unlearning Through Reasoningcited by 7
  3. 2026AEGIS: Adversarial Target-Guided Retention-Data-Free Robust Concept Erasure from Diffusion Modelscited by 5
  4. 2026Is your algorithm unlearning or untraining?cited by 4
  5. 2025What Is Preference Optimization Doing, and Why?cited by 2
  6. 2026Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearningcited by 2
  7. 2026BlindU: Blind Machine Unlearning without Revealing Erasing Datacited by 1
  8. 2026FeatDistill: A Feature Distillation Enhanced Multi-Expert Ensemble Framework for Robust AI-generated Image Detectioncited by 1
  9. 2026Protecting patient privacy in clinical foundation models: Technical and legal perspectivescited by 1
  10. 2026Unlearning What Matters: Token-Level Attribution for Precise Language Model Unlearningcited by 0
  11. 2026Graph Federated Unlearning for Privacy Preservationcited by 0
  12. 2026LLM Unlearning for Cyber Defense: A Survey on Methods, Challenges, and Emerging Threatscited by 0
5 more
  1. 2026Learning Dynamics of Continual Learning: A Unified View of Data Attribution, Forgetting, and Plasticity Losscited by 0
  2. 2026UnlearningSoup: Is Repeated Tuning Necessary for Large Language Model Unlearning?cited by 0
  3. 2026Unlearning Deceptive Behaviors in LLMs with Contrastive Forget Setscited by 0
  4. 2026Preemptive LLM Unlearning against Forbidden Capability Acquisition via Gradient Sealingcited by 0
  5. 2026LLM Persona Unlearningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Catastrophic Failure of LLM Unlearning via Quantization
  2. 2025LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarks
  3. 2026Unlearning in LLMs: Methods, Evaluation, and Open Challenges
  4. 2025Downgrade to Upgrade: Optimizer Simplification Enhances Robustness in LLM Unlearning
  5. 2024Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning
  6. 2025A Survey on Unlearning in Large Language Models
  7. 2024Unlearning in- vs. out-of-distribution data in LLMs under gradient-based method
  8. 2024Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methods