mlsanityData to 7 Oct 2026

Erasure of Unaligned Attributes from Neural Representations

arXiv 2302.02997 ↗First posted 6 Feb 2023cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

We present the Assignment-Maximization Spectral Attribute removaL (AMSAL) algorithm, which erases information from neural representations when the information to be erased is implicit rather than directly being aligned to each input example. Our algorithm works by alternating between two steps. In one, it finds an assignment of the input representations to the information to be erased, and in the other, it creates projections of both the input representations and the information to be erased into a joint latent space. We test our algorithm on an extensive array of datasets, including a Twitter dataset with multiple guarded attributes, the BiasBios dataset and the BiasBench benchmark. The last benchmark includes four datasets with various types of protected attributes. Our results demonstrate that bias can often be removed in our setup. We also discuss the limitations of our approach when there is a strong entanglement between the main task and the information to be erased.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 6 Feb 2023.

Builds on

31 of 42 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  4. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  5. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  6. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  7. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  8. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
  9. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  10. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 346
  11. 2016FastText.zip: Compressing text classification modelscited by 331
  12. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
19 more
  1. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 194
  2. 2015Censoring Representations with an Adversarycited by 191
  3. 2018Learning Gender-Neutral Word Embeddingscited by 169
  4. 2018Adversarial Removal of Demographic Attributes from Text Datacited by 166
  5. 2017Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasmcited by 117
  6. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  7. 2022Linear Adversarial Concept Erasurecited by 98
  8. 2021An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Modelscited by 96
  9. 2018Towards Robust and Privacy-preserving Text Representationscited by 75
  10. 2018Privacy-preserving Neural Representations of Textcited by 54
  11. 2020OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddingscited by 40
  12. 2021Contrastive Learning for Fair Representationscited by 34
  13. 2022Gold Doesn't Always Glitter: Spectral Removal of Linear and Nonlinear Guarded Attribute Informationcited by 33
  14. 2022Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translationcited by 23
  15. 2019Measuring Social Bias in Knowledge Graph Embeddingscited by 17
  16. 2022Learning Disentangled Textual Representations via Statistical Measures of Similaritycited by 14
  17. 2022Conditional Supervised Contrastive Learning for Fair Text Classificationcited by 11
  18. 2022Easy Adaptation to Mitigate Gender Bias in Multilingual Text Classificationcited by 5
  19. 2022fairlib: A Unified Framework for Assessing and Improving Classification Fairnesscited by 4

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2; 2025: 1.

6 papers.

  1. 2023LEACE: Perfect linear concept erasure in closed formcited by 254
  2. 2024Spectral Editing of Activations for Large Language Model Alignmentcited by 54
  3. 2023BERT is not The Count: Learning to Match Mathematical Statements with Proofscited by 9
  4. 2025FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Modelscited by 6
  5. 2023How To Build Competitive Multi-gender Speech Translation Models For Controlling Speaker Gender Translationcited by 2
  6. 2024Surgical Feature-Space Decomposition of LLMs: Why, When and How?cited by 2