mlsanityData to 14 Oct 2026

A Probabilistic Perspective on Unlearning and Alignment for Large Language Models

arXiv 2410.03523 ↗First posted 4 Oct 2024cs.LG cs.AIcited by 21

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

Comprehensive evaluation of Large Language Models (LLMs) is an open research problem. Existing evaluations rely on deterministic point estimates generated via greedy decoding. However, we find that deterministic evaluations fail to capture the whole output distribution of a model, yielding inaccurate estimations of model capabilities. This is particularly problematic in critical contexts such as unlearning and alignment, where precise model evaluations are crucial. To remedy this, we introduce the first formal probabilistic evaluation framework for LLMs. Namely, we propose novel metrics with high probability guarantees concerning the output distribution of a model. Our metrics are application-independent and allow practitioners to make more reliable estimates about model capabilities before deployment. Our experimental analysis reveals that deterministic evaluations falsely indicate successful unlearning and alignment, whereas our probabilistic evaluations better capture model capabilities. We show how to overcome challenges associated with probabilistic outputs in a case study on unlearning by introducing (1) a novel loss based on entropy optimization, and (2) adaptive temperature scaling. We demonstrate that our approach significantly enhances unlearning in probabilistic settings on recent benchmarks. Overall, our proposed shift from point estimates to probabilistic evaluations of output distributions represents an important step toward comprehensive evaluations of LLMs. Code available at https://www.cs.cit.tum.de/daml/probabilistic-unlearning/.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 47 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

36 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr

Other references · 35

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  3. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  5. 2019Machine Unlearningcited by 1,061
  6. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  7. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 618
  8. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  9. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  10. 2019Certified Data Removal from Machine Learning Modelscited by 535
  11. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 533
  12. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
23 more
  1. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  2. 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 468
  3. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 437
  4. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 381
  5. 2021Remember What You Want to Forget: Algorithms for Machine Unlearningcited by 371
  6. 2020Descent-to-Delete: Gradient-Based Methods for Machine Unlearningcited by 311
  7. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 224
  8. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 215
  9. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  10. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 163
  11. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 158
  12. 2021Machine Unlearning via Algorithmic Stabilitycited by 130
  13. 2022Continual Learning and Private Unlearningcited by 130
  14. 2023ProPILE: Probing Privacy Leakage in Large Language Modelscited by 129
  15. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 113
  16. 2023KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignmentcited by 99
  17. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 94
  18. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 91
  19. 2023Adversarial Attacks and Defenses in Large Language Models: Old and New Threatscited by 49
  20. 2022Certified Graph Unlearningcited by 46
  21. 2020Certifying Confidence via Randomized Smoothingcited by 36
  22. 2022Repairing Neural Networks by Leaving the Right Past Behindcited by 27
  23. 2022Localized Randomized Smoothing for Collective Robustness Certificationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 7.

21 papers.

Sort:

By area:

  1. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 115
  2. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metricscited by 83
  3. 2026A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustnesscited by 20
  4. 2025LLM-Safety Evaluations Lack Robustnesscited by 18
  5. 2025REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objectivecited by 15
  6. 2025SoK: Machine Unlearning for Large Language Modelscited by 11
  7. 2025Leak@k: Unlearning Does Not Make LLMs Forget Under Probabilistic Decodingcited by 9
  8. 2026Estimating Tail Risks in Language Model Output Distributionscited by 9
  9. 2025Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectivescited by 8
  10. 2025Existing Large Language Model Unlearning Evaluations Are Inconclusivecited by 8
  11. 2025Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMscited by 7
  12. 2025Diffusion LLMs are Natural Adversaries for any LLMcited by 7
9 more
  1. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Researchcited by 6
  2. 2025Understanding the Dilemma of Unlearning for Large Language Modelscited by 4
  3. 2025Align-then-Unlearn: Embedding Alignment for LLM Unlearningcited by 3
  4. 2026Inference-time Unlearning Using Conformal Predictioncited by 2
  5. 2025Sequential Bayesian Evaluation of Large Language Model Behaviorcited by 1
  6. 2026Beyond Binary Rewards: A Comparative Study of Reward Design for Reinforcement Unlearningcited by 1
  7. 2026Closing the Distribution Gap in Adversarial Training for LLMscited by 0
  8. 2026From 'May' to 'Is': Certainty Distortion in Language Model Rewritingcited by 0
  9. 2026BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Comprehensive Survey of Machine Unlearning Techniques for Large Language Models
  2. 2026Unlearning in LLMs: Methods, Evaluation, and Open Challenges
  3. 2025Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models
  4. 2024Towards Effective Evaluations and Comparisons for LLM Unlearning Methods
  5. 2024A Closer Look at Machine Unlearning for Large Language Models
  6. 2025Do Large Language Models (Really) Need Statistical Foundations?
  7. 2025A Survey on Unlearning in Large Language Models
  8. 2025Pruning Laws for Large Language Models