mlsanityData to 14 Oct 2026

Unlearning at Scale: State-Exact Trace-Preserving Deletion in Billion-Parameter Language Models

arXiv 2508.12220 ↗First posted 17 Aug 2025cs.LG cs.AI cs.CRcited by 1

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Can a prospectively instrumented training continuation reproduce a deletion counterfactual exactly after selected examples leave its replay dataset? We study a trace-preserving counterfactual that fixes recorded execution controls while assigning requested identifiers zero contribution. The guarantee is prospective: the original run must record this execution provenance and retain an eligible uncontaminated checkpoint. Under pinned single-GPU environments, replay from a token store materialized without the requested rows reconstructs a separately executed trace oracle bit-for-bit in model and optimizer state. Pythia 160M is exact across four deletion geometries; Pythia 2.8B matches all 2,775,208,960 model-state elements for a random 5% request; and Llama 3.2 1B is exact after omitting 400 of 4,000 TOFU examples from replay storage. These results establish billion-parameter state exactness. They do not establish cheap deletion, because dispersed requests can force nearly full replay. We release standardized TOFU/OpenUnlearning measurements as descriptive diagnostics only because the frozen campaign lacks the matched controls required for a causal behavioral claim.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 30 of its 41 references that are in the map. First posted on arXiv 17 Aug 2025.

Builds on

30 of 41 references.

Closest ancestors · 21

Same direction, newest first

  1. 2025Learning-Time Encoding Shapes Unlearning in LLMs18 Jun
  2. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics14 Jun
  3. 2025Certified Unlearning for Neural Networks8 Jun
  4. 2025System-Aware Unlearning Algorithms: Use Lesser, Forget Faster6 Jun
  5. 2025Machine Unlearning under Overparameterization28 May
  6. 2025Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs22 May
  7. 2024Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Research9 Dec
  8. 2024A Closer Look at Machine Unlearning for Large Language Models10 Oct
  9. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
  10. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
  11. 2024Towards Scalable Exact Machine Unlearning Using Parameter-Efficient Fine-Tuning24 Jun
  12. 2024Textual Unlearning Gives a False Sense of Unlearning19 Jun
9 more
  1. 2024Machine Unlearning of Pre-trained Large Language Models23 Feb
  2. 2024Rethinking Machine Unlearning for Large Language Models13 Feb
  3. 2024Fast Exact Unlearning for In-Context Learning Data for LLMs1 Feb
  4. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
  5. 2023In-Context Unlearning: Language Models as Few Shot Unlearners11 Oct
  6. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Models4 Oct
  7. 2021On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning22 Oct
  8. 2021Adaptive Machine Unlearning8 Jun
  9. 2021Remember What You Want to Forget: Algorithms for Machine Unlearning4 Mar

Other references · 9

  1. 2016Pointer Sentinel Mixture Modelscited by 3,096
  2. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  3. 2020Extracting Training Data from Large Language Modelscited by 2,189
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  5. 2019Machine Unlearningcited by 1,061
  6. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  7. 2019Certified Data Removal from Machine Learning Modelscited by 535
  8. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 465
  9. 2020Descent-to-Delete: Gradient-Based Methods for Machine Unlearningcited by 311

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Exact Record Omission in Delta Attention: A Transport Criterion, Its Cost, and a Replay Certificatecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Forgetting Without Restarting: Execution-State Unlearning for Stateful LLM Agents
  2. 2026Auditing Forgetting in Limited Memory Language Models
  3. 2026Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation
  4. 2026FIT to Forget: Robust Continual Unlearning for Large Language Models
  5. 2025The Erasure Illusion: Stress-Testing the Generalization of LLM Forgetting Evaluation
  6. 2026Aborted but Not Forgotten: KV-Cache Retention Breaks Rollback Consistency in Language Agents
  7. 2026Re-feeding Is Not Replaying: Measuring Replay Noise in Counterfactual Token-Credit Estimation
  8. 2026A Frozen 12B Beats Frontier Models on Verified Work: 100% Accuracy, 0 Tokens, Bit-Exact, Forever