mlsanityData to 14 Oct 2026

Contrastive Representation Shaping for LLM Unlearning

arXiv 2601.22028 ↗First posted 29 Jan 2026cs.LGcited by 7

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Most LLM unlearning methods aim to approximate retrain-from-scratch behaviors with minimal distribution shift, often via alignment-style objectives defined in the prediction space. While effective at reducing forgotten content generation, such approaches may act as suppression: forgotten concepts can persist in representations and remain entangled with retained knowledge. We introduce CLReg, a contrastive representation regularizer that identifies forget features while pushing them away from retain features, reducing forget--retain interference while empirically preserving the scale and shape of retain features. As light motivation for the mechanism, we provide a one-step analysis showing that CLReg decreases a simple entanglement proxy in the embedding space. Across unlearning benchmarks and LLMs of different sizes, CLReg decreases forget-retain representation entanglement to enhance mainstream unlearning methods without extra privacy risks, inspiring future unlearning work to remove forget concepts via representation shaping. Code is available at https://github.com/HaoranTang/CLReg.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 35 of its 39 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

35 of 39 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025LLM Unlearning with LLM Beliefs22 Oct
  2. 2025CoUn: Empowering Machine Unlearning via Contrastive Learning19 Sep
  3. 2025Sharpness-Aware Machine Unlearning16 Jun
  4. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics14 Jun
  5. 2025Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models5 Jun
  6. 2024LLM Unlearning via Loss Adjustment with Only Forget Data14 Oct
  7. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
  8. 2024Dissecting Fine-Tuning Unlearning in Large Language Models9 Oct
  9. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
  10. 2024What makes unlearning hard and what to do about it3 Jun
  11. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  12. 2024UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models15 Feb
6 more
  1. 2024Rethinking Machine Unlearning for Large Language Models13 Feb
  2. 2024Contrastive Unlearning: A Contrastive Approach to Machine Unlearning19 Jan
  3. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
  4. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation19 Oct
  5. 2023Large Language Model Unlearning14 Oct
  6. 2021Machine Unlearning of Features and Labels26 Aug

Other references · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  8. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  9. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  10. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 618
  11. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  12. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
5 more
  1. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  2. 2020Amnesiac Machine Learningcited by 319
  3. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  4. 2020Approximate Data Deletion from Machine Learning Modelscited by 255
  5. 2020Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Taskscited by 47

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlookcited by 43
  2. 2026LLM Unlearning for Cyber Defense: A Survey on Methods, Challenges, and Emerging Threatscited by 0
  3. 2026Gradient Concentration, Not Weight Saliency, Explains Representation-Level Class Unlearningcited by 0
  4. 2026Entangled Representations Amplify Collateral Damage in Unlearningcited by 0
  5. 2026Unlearning Deceptive Behaviors in LLMs with Contrastive Forget Setscited by 0
  6. 2026Learning What to Forget: Distributional Unlearning for LLM Representation Spacescited by 0
  7. 2026RA2: Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Machine Unlearning under Retain-Forget Entanglement
  2. 2025Towards Mitigating Excessive Forgetting in LLM Unlearning via Entanglement-Guidance with Proxy Constraint
  3. 2026Quantization-Robust Unlearning through the Lens of Retain-Forget Loss Landscapes Interaction
  4. 2026Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it
  5. 2026Robust LLM Unlearning Against Relearning Attacks: The Minor Components in Representations Matter
  6. 2025LLM Unlearning on Noisy Forget Sets: A Study of Incomplete, Rewritten, and Watermarked Data
  7. 2026Representation-Guided Parameter-Efficient LLM Unlearning
  8. 2026Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs