# CURE: Controlled Unlearning for Robust Embeddings -- Mitigating Conceptual Shortcuts in Pre-Trained Language Models

arXiv: https://arxiv.org/abs/2509.05230 · HTML: https://mlsanity.com/paper/2509.05230/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 22 of its 43 references that are in the map.
- First posted on arXiv 5 Sep 2025.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models](https://mlsanity.com/paper/2007.06778/) (2020)
- [Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual](https://mlsanity.com/paper/1908.10763/) (2019)
- [How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks](https://mlsanity.com/paper/1808.04926/) (2018)
- [Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation](https://mlsanity.com/paper/1905.05621/) (2019)
- [Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models](https://mlsanity.com/paper/2110.07736/) (2021)
- [Identifying Spurious Correlations for Robust Text Classification](https://mlsanity.com/paper/2010.02458/) (2020)
- [Explore Spurious Correlations at the Concept Level in Language Models for Text Classification](https://mlsanity.com/paper/2311.08648/) (2023)
- [Unifying Heterogeneous Electronic Health Records Systems via Text-Based Code Embedding](https://mlsanity.com/paper/2108.03625/) (2021)
- [Causal Intervention Improves Implicit Sentiment Analysis](https://mlsanity.com/paper/2208.09329/) (2022)
- [Topics to Avoid: Demoting Latent Confounds in Text Classification](https://mlsanity.com/paper/1909.00453/) (2019)
- [Less Learn Shortcut: Analyzing and Mitigating Learning of Spurious Feature-Label Correlation](https://mlsanity.com/paper/2205.12593/) (2022)
- [RAZOR: Sharpening Knowledge by Cutting Bias with Unsupervised Text Rewriting](https://mlsanity.com/paper/2412.07675/) (2024)

## Abstract (arXiv)

Pre-trained language models have achieved remarkable success across diverse applications but remain susceptible to spurious, concept-driven correlations that impair robustness and fairness. In this work, we introduce CURE, a novel and lightweight framework that systematically disentangles and suppresses conceptual shortcuts while preserving essential content information. Our method first extracts concept-irrelevant representations via a dedicated content extractor reinforced by a reversal network, ensuring minimal loss of task-relevant information. A subsequent controllable debiasing module employs contrastive learning to finely adjust the influence of residual conceptual cues, enabling the model to either diminish harmful biases or harness beneficial correlations as appropriate for the target task. Evaluated on the IMDB and Yelp datasets using three pre-trained architectures, CURE achieves an absolute improvement of +10 points in F1 score on IMDB and +2 points on Yelp, while introducing minimal computational overhead. Our approach establishes a flexible, unsupervised blueprint for combating conceptual biases, paving the way for more reliable and fair language understanding systems.