mlsanityData to 7 Oct 2026

Linear Adversarial Concept Erasure

arXiv 2201.12091 ↗First posted 28 Jan 2022cs.LG cs.CLcited by 98

Not on the map. Why?

Abstract (arXiv)

Modern neural models trained on textual data rely on pre-trained representations that emerge without direct supervision. As these representations are increasingly being used in real-world applications, the inability to control their content becomes an increasingly important problem. We formulate the problem of identifying and erasing a linear subspace that corresponds to a given concept, in order to prevent linear predictors from recovering the concept. We model this problem as a constrained, linear maximin game, and show that existing solutions are generally not optimal for this task. We derive a closed-form solution for certain objectives, and propose a convex relaxation, method, that works well for others. When evaluated in the context of binary gender removal, the method recovers a low-dimensional subspace whose removal mitigates bias by intrinsic and extrinsic evaluation. We show that the method is highly expressive, effectively mitigating bias in deep nonlinear classifiers while maintaining tractability and interpretability.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Builds on 35 of its 74 references that are in the map. First posted on arXiv 28 Jan 2022.

Builds on

35 of 74 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  3. 2021Convex optimizationcited by 2,479
  4. 2018Deep contextualized word representationscited by 2,285
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  6. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  7. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  8. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  9. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  10. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  11. 2018Mitigating Unwanted Biases with Adversarial Learningcited by 663
  12. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
23 more
  1. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 346
  2. 2021Efficient and Modular Implicit Differentiationcited by 199
  3. 2015Censoring Representations with an Adversarycited by 191
  4. 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 170
  5. 2018Adversarial Removal of Demographic Attributes from Text Datacited by 166
  6. 2019Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methodscited by 148
  7. 2020Intrinsic Bias Metrics Do Not Correlate with Application Biascited by 148
  8. 2017Controllable Invariance through Adversarial Feature Learningcited by 127
  9. 2021Debiasing Pre-trained Contextualised Embeddingscited by 107
  10. 2020The Complexity of Constrained Min-Max Optimizationcited by 81
  11. 2021Accelerated Algorithms for Smooth Convex-Concave Minimax Problems with O(1/k2) Rate on Squared Gradient Normcited by 71
  12. 2019Attenuating Bias in Word Vectorscited by 64
  13. 2021Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Predictioncited by 58
  14. 2016Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classificationcited by 48
  15. 2015From Facial Parts Responses to Face Detection: A Deep Learning Approachcited by 42
  16. 2020OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddingscited by 40
  17. 2021The Low-Dimensional Linear Geometry of Contextualized Word Representationscited by 39
  18. 2020Improved Algorithms for Convex-Concave Minimax Optimizationcited by 34
  19. 2020It's not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERTcited by 30
  20. 2020Exploring the Linear Subspace Hypothesis in Gender Bias Mitigationcited by 25
  21. 2019On the Global Optima of Kernelized Adversarial Representation Learningcited by 14
  22. 2021Obstructing Classification via Projectioncited by 5
  23. 2021Effects of Pre- and Post-Processing on type-based Embeddings in Lexical Semantic Change Detectioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 20; 2024: 13; 2025: 19; 2026: 31.

98 papers.

Sort:

By area:

  1. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  2. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  3. 2023LEACE: Perfect linear concept erasure in closed formcited by 254
  4. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  5. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  6. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  7. 2023A Survey on Fairness in Large Language Modelscited by 91
  8. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  9. 2022Probing for the Usage of Grammatical Numbercited by 57
  10. 2022Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Modelscited by 56
  11. 2022Probing Classifiers are Unreliable for Concept Removal and Detectioncited by 40
  12. 2022Gender Artifacts in Visual Datasetscited by 38
38 more
  1. 2024Machine Unlearning Fails to Remove Data Poisoning Attackscited by 35
  2. 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 34
  3. 2022Gold Doesn't Always Glitter: Spectral Removal of Linear and Nonlinear Guarded Attribute Informationcited by 33
  4. 2024Immunization against harmful fine-tuning attackscited by 33
  5. 2022MABEL: Attenuating Gender Bias using Textual Entailment Datacited by 31
  6. 2023Parameter-efficient Modularised Bias Mitigation via AdapterFusioncited by 25
  7. 2023Shielded Representations: Protecting Sensitive Attributes Through Iterative Gradient-Based Projectioncited by 25
  8. 2023Debiasing Algorithm through Model Adaptationcited by 24
  9. 2023Efficient fair PCA for fair representation learningcited by 23
  10. 2023A Geometric Notion of Causal Probingcited by 23
  11. 2022BLIND: Bias Removal With No Demographicscited by 22
  12. 2024Controllable Context Sensitivity and the Knob Behind Itcited by 22
  13. 2023Prompt Tuning Pushes Farther, Contrastive Learning Pulls Closer: A Two-Stage Approach to Mitigate Social Biasescited by 21
  14. 2024Decomposing and Editing Predictions by Modeling Model Computationcited by 19
  15. 2024Attribute-to-Delete: Machine Unlearning via Datamodel Matchingcited by 18
  16. 2022Better Hit the Nail on the Head than Beat around the Bush: Removing Protected Attributes with a Single Projectioncited by 17
  17. 2025Improved Representation Steering for Language Modelscited by 17
  18. 2025Analyzing Finetuning Representation Shift for Multimodal LLMs Steeringcited by 15
  19. 2022Visual Comparison of Language Model Adaptationcited by 14
  20. 2023A Review of the Role of Causality in Developing Trustworthy AI Systemscited by 14
  21. 2024Towards a theory of model distillationcited by 14
  22. 2023Uncovering Intermediate Variables in Transformers using Circuit Probingcited by 12
  23. 2025Ascent Fails to Forgetcited by 12
  24. 2025Emergence of Linear Truth Encodings in Language Modelscited by 12
  25. 2022Analyzing Gender Representation in Multilingual Modelscited by 10
  26. 2023Robust Concept Erasure via Kernelized Rate-Distortion Maximizationcited by 10
  27. 2022Fair NLP Models with Differentially Private Text Encoderscited by 9
  28. 2025The Blessing and Curse of Dimensionality in Safety Alignmentcited by 9
  29. 2026The Information Geometry of Softmax: Probing and Steeringcited by 9
  30. 2025NS-Net: Decoupling CLIP Semantic Information through NULL-Space for Generalizable AI-Generated Image Detectioncited by 7
  31. 2023Erasure of Unaligned Attributes from Neural Representationscited by 6
  32. 2024Concept-ROT: Poisoning Concepts in Large Language Models with Model Editingcited by 6
  33. 2023Controlling Learned Effects to Reduce Spurious Correlations in Text Classifierscited by 5
  34. 2023Removing Spurious Concepts from Neural Network Representations via Joint Subspace Estimationcited by 5
  35. 2023Fair Streaming Principal Component Analysis: Statistical and Algorithmic Viewpointcited by 5
  36. 2025Fundamental Limits of Perfect Concept Erasurecited by 5
  37. 2023Competence-Based Analysis of Language Modelscited by 4
  38. 2024On the Encoding of Gender in Transformer-based ASR Representationscited by 4

50 most cited of 98 All 98 (JSON)