mlsanityData to 14 Oct 2026

REMIND: Input Loss Landscapes Reveal Residual Memorization in Post-Unlearning LLMs

arXiv 2511.04228 ↗First posted 6 Nov 2025cs.CL cs.LGcited by 0

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Machine unlearning aims to remove the influence of specific training data from a model without requiring full retraining. This capability is crucial for ensuring privacy, safety, and regulatory compliance. Therefore, verifying whether a model has truly forgotten target data is essential for maintaining reliability and trustworthiness. However, existing evaluation methods often assess forgetting at the level of individual inputs. This approach may overlook residual influence present in semantically similar examples. Such influence can compromise privacy and lead to indirect information leakage. We propose REMIND (Residual Memorization In Neighborhood Dynamics), a novel evaluation method aiming to detect the subtle remaining influence of unlearned data and classify whether the data has been effectively forgotten. REMIND analyzes the model's loss over small input variations and reveals patterns unnoticed by single-point evaluations. We show that unlearned data yield flatter, less steep loss landscapes, while retained or unrelated data exhibit sharper, more volatile patterns. REMIND requires only query-based access, outperforms existing methods under similar constraints, and demonstrates robustness across different models, datasets, and paraphrased inputs, making it practical for real-world deployment. By providing a more sensitive and interpretable measure of unlearning effectiveness, REMIND provides a reliable framework to assess unlearning in language models. As a result, REMIND offers a novel perspective on memorization and unlearning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 18 of its 29 references that are in the map. First posted on arXiv 6 Nov 2025.

Builds on

18 of 29 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics14 Jun
  2. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
  3. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progress3 Oct
  4. 2024Erasing Conceptual Knowledge from Language Models3 Oct
  5. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
  6. 2024Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy2 Mar
  7. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan

Other references · 11

  1. 2020Extracting Training Data from Large Language Modelscited by 2,189
  2. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  3. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 258
  4. 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
  5. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  6. 2024ReCaLL: Membership Inference via Relative Conditional Log-Likelihoodscited by 64
  7. 2024ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddingscited by 32
  8. 2024Data Contamination Calibration for Black-box LLMscited by 24
  9. 2022Understanding and Combating Robust Overfitting via Input Loss Landscape Analysis and Regularizationcited by 22
  10. 2024Noisy Neighbors: Efficient membership inference attacks against LLMscited by 21
  11. 2024Adversarial Attacks on Large Language Models Using Regularized Relaxationcited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem
  2. 2025LLM Unlearning on Noisy Forget Sets: A Study of Incomplete, Rewritten, and Watermarked Data
  3. 2025Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs
  4. 2024Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning
  5. 2026Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs
  6. 2024Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearning
  7. 2026Trajectory-Guided Forget-Recover Network for Continual LLM Unlearning
  8. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigation