mlsanityData to 7 Oct 2026

CFL: Causally Fair Language Models Through Token-level Attribute Controlled Generation

arXiv 2306.00374 ↗First posted 1 Jun 2023cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

We propose a method to control the attributes of Language Models (LMs) for the text generation task using Causal Average Treatment Effect (ATE) scores and counterfactual augmentation. We explore this method, in the context of LM detoxification, and propose the Causally Fair Language (CFL) architecture for detoxifying pre-trained LMs in a plug-and-play manner. Our architecture is based on a Structural Causal Model (SCM) that is mathematically transparent and computationally efficient as compared with many existing detoxification techniques. We also propose several new metrics that aim to better understand the behaviour of LMs in the context of toxic text generation. Further, we achieve state of the art performance for toxic degeneration, which are computed using RTP (RTP) benchmark. Our experiments show that CFL achieves such a detoxification without much impact on the model perplexity. We also show that CFL mitigates the unintended bias problem through experiments on the BOLD dataset.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 37 of its 57 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

37 of 57 references.

References · 37

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2020Extracting Training Data from Large Language Modelscited by 2,144
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  7. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  8. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  9. 2022Red Teaming Language Models with Language Modelscited by 983
  10. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  11. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  12. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
25 more
  1. 2019Green AIcited by 620
  2. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  3. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  4. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  5. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  6. 2019Measurement and Fairnesscited by 299
  7. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 215
  8. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 208
  9. 2019SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)cited by 206
  10. 2020HateBERT: Retraining BERT for Abusive Language Detection in Englishcited by 206
  11. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  12. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  13. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 181
  14. 2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 177
  15. 2021Challenges in Detoxifying Language Modelscited by 169
  16. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 147
  17. 2022Causal Machine Learning: A Survey and Open Problemscited by 107
  18. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 106
  19. 2020The Radicalization Risks of GPT-3 and Advanced Neural Language Modelscited by 76
  20. 2016Controlling Output Length in Neural Encoder-Decoderscited by 75
  21. 2021Towards generalisable hate speech detection: a review on obstacles and solutionscited by 66
  22. 2020Exploring Controllable Text Generation Techniquescited by 55
  23. 2022Informativeness and Invariance: Two Perspectives on Spurious Correlations in Natural Languagecited by 21
  24. 2022Reward Modeling for Mitigating Toxicity in Transformer-based Language Modelscited by 17
  25. 2018A Stable and Effective Learning Strategy for Trainable Greedy Decodingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 1; 2026: 2.

6 papers.

  1. 2024Large Language Models and Causal Inference in Collaboration: A Surveycited by 26
  2. 2026CausalDetox: Causal Head Selection and Intervention for Language Model Detoxificationcited by 3
  3. 2025Causality Is Key to Understand and Balance Multiple Goals in Trustworthy ML and Foundation Modelscited by 2
  4. 2023Causal ATE Mitigates Unintended Bias in Controlled Text Generationcited by 1
  5. 2026Trustworthy AI Suffers from Invariance Conflicts and Causality is The Solutioncited by 1
  6. 2024Position Debiasing Fine-Tuning for Causal Perception in Long-Term Dialoguecited by 0