# Erase at the Core: Representation Unlearning for Machine Unlearning

arXiv: https://arxiv.org/abs/2602.05375 · HTML: https://mlsanity.com/paper/2602.05375/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 27 of its 34 references that are in the map.
- First posted on arXiv 5 Feb 2026.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks](https://mlsanity.com/paper/1911.04933/) (2019)
- [SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation](https://mlsanity.com/paper/2310.12508/) (2023)
- [Deeply-Supervised Nets](https://mlsanity.com/paper/1409.5185/) (2014)
- [Unrolling SGD: Understanding Factors Influencing Machine Unlearning](https://mlsanity.com/paper/2109.13398/) (2021)
- [Coded Machine Unlearning](https://mlsanity.com/paper/2012.15721/) (2020)
- [Decoupled Distillation to Erase: A General Unlearning Method for Any Class-centric Tasks](https://mlsanity.com/paper/2503.23751/) (2025)
- [Machine Unlearning via Null Space Calibration](https://mlsanity.com/paper/2404.13588/) (2024)
- [Contrastive Deep Supervision](https://mlsanity.com/paper/2207.05306/) (2022)
- [Is Retain Set All You Need in Machine Unlearning? Restoring Performance of Unlearned Models with Out-Of-Distribution Images](https://mlsanity.com/paper/2404.12922/) (2024)
- [Are We Truly Forgetting? A Critical Re-examination of Machine Unlearning Evaluation Protocols](https://mlsanity.com/paper/2503.06991/) (2025)
- [Contrastive Unlearning: A Contrastive Approach to Machine Unlearning](https://mlsanity.com/paper/2401.10458/) (2024)
- [DUCK: Distance-based Unlearning via Centroid Kinematics](https://mlsanity.com/paper/2312.02052/) (2023)
- [From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularization](https://mlsanity.com/paper/2505.22310/) (2025)
- [An Information Theoretic Evaluation Metric For Strong Unlearning](https://mlsanity.com/paper/2405.17878/) (2024)
- [One-Point Contraction: Erasing Representational Separability toward Irreversible Deep Forgetting](https://mlsanity.com/paper/2507.07754/) (2025)

## Built on by (most cited first, 5 of 5)

- [Erased, but Not Gone: Output Forgetting Is Not True Forgetting](https://mlsanity.com/paper/2606.25001/) (2026)
- [The BatchNorm Illusion: Diagnosing Normalization Artifacts in Machine Unlearning Evaluation](https://mlsanity.com/paper/2609.08901/) (2026)
- [Gradient Concentration, Not Weight Saliency, Explains Representation-Level Class Unlearning](https://mlsanity.com/paper/2607.21353/) (2026)
- [Source-Free Class Relearning: Diagnosing Forgetting in Class Unlearning](https://mlsanity.com/paper/2609.02018/) (2026)
- [RA2: Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning](https://mlsanity.com/paper/2610.19385/) (2026)

## Abstract (arXiv)

Many approximate machine unlearning methods demonstrate strong logit-level forgetting -- such as near-zero accuracy on the forget set -- yet continue to preserve substantial information within their internal feature representations. We refer to this discrepancy as superficial forgetting. Recent studies indicate that most existing unlearning approaches primarily alter the final classifier, leaving intermediate representations largely unchanged and highly similar to those of the original model. To address this limitation, we introduce the Erase at the Core (EC), a framework designed to enforce forgetting throughout the entire network hierarchy. EC integrates multi-layer contrastive unlearning on the forget set with retain set preservation through deeply supervised learning. Concretely, EC attaches auxiliary modules to intermediate layers and applies both contrastive unlearning and cross-entropy losses at each supervision point, with layer-wise weighted losses. Experimental results show that EC not only achieves effective logit-level forgetting, but also substantially reduces representational similarity to the original model across intermediate layers. Furthermore, EC is model-agnostic and can be incorporated as a plug-in module into existing unlearning methods, improving representation-level forgetting while maintaining performance on the retain set.