# Preserving Task-Relevant Information Under Linear Concept Removal

arXiv: https://arxiv.org/abs/2506.10703 · HTML: https://mlsanity.com/paper/2506.10703/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 45 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations](https://mlsanity.com/paper/2204.02937/) (2022)
- [Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting](https://mlsanity.com/paper/1901.09451/) (2019)
- [Improving Text Embeddings with Large Language Models](https://mlsanity.com/paper/2401.00368/) (2023)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [Adversarial Removal of Demographic Attributes from Text Data](https://mlsanity.com/paper/1808.06640/) (2018)
- [Controllable Invariance through Adversarial Feature Learning](https://mlsanity.com/paper/1705.11122/) (2017)
- [CausaLM: Causal Model Explanation Through Counterfactual Language Models](https://mlsanity.com/paper/2005.13407/) (2020)
- [Linear Adversarial Concept Erasure](https://mlsanity.com/paper/2201.12091/) (2022)
- [Places: An Image Database for Deep Scene Understanding](https://mlsanity.com/paper/1610.02055/) (2016)
- [Probing Classifiers are Unreliable for Concept Removal and Detection](https://mlsanity.com/paper/2207.04153/) (2022)
- [OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddings](https://mlsanity.com/paper/2007.00049/) (2020)
- [Contrastive Learning for Fair Representations](https://mlsanity.com/paper/2109.10645/) (2021)
- [Gold Doesn't Always Glitter: Spectral Removal of Linear and Nonlinear Guarded Attribute Information](https://mlsanity.com/paper/2203.07893/) (2022)
- [A Geometric Notion of Causal Probing](https://mlsanity.com/paper/2307.15054/) (2023)
- [Debiasing Algorithm through Model Adaptation](https://mlsanity.com/paper/2310.18913/) (2023)
- [Efficient fair PCA for fair representation learning](https://mlsanity.com/paper/2302.13319/) (2023)
- [Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark](https://mlsanity.com/paper/2406.01607/) (2024)
- [Better Hit the Nail on the Head than Beat around the Bush: Removing Protected Attributes with a Single Projection](https://mlsanity.com/paper/2212.04273/) (2022)

## Built on by (most cited first, 4 of 4)

- [Concept-Based Abductive and Contrastive Explanations for Behaviors of Vision Models](https://mlsanity.com/paper/2605.06640/) (2026)
- [Interpolation Is Not Invariance: Pair Count Is Not Coverage in Transformation Audits](https://mlsanity.com/paper/2609.14870/) (2026)
- [Measurement-Gated Provenance Attenuation for Frozen EEG Representations](https://mlsanity.com/paper/2609.32889/) (2026)
- [Operational Abstractions of Neural Network Concepts via Topological Representations](https://mlsanity.com/paper/2610.05017/) (2026)

## Abstract (arXiv)

Modern neural networks often encode unwanted concepts alongside task-relevant information, leading to fairness and interpretability concerns. Existing post-hoc approaches can remove undesired concepts but often degrade useful signals. We introduce SPLINCE-Simultaneous Projection for LINear concept removal and Covariance prEservation - which eliminates sensitive concepts from representations while exactly preserving their covariance with a target label. SPLINCE achieves this via an oblique projection that 'splices out' the unwanted direction yet protects important label correlations. Theoretically, it is the unique solution that removes linear concept predictability and maintains target covariance with minimal embedding distortion. Empirically, SPLINCE outperforms baselines on benchmarks such as Bias in Bios and Winobias, removing protected attributes while minimally damaging main-task information.