mlsanityData to 14 Oct 2026

A Trip Towards Fairness: Bias and De-Biasing in Large Language Models

arXiv 2305.13862 ↗First posted 23 May 2023cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

Cheap-to-Build Very Large-Language Models (CtB-LLMs) with affordable training are emerging as the next big revolution in natural language processing and understanding. These CtB-LLMs are democratizing access to trainable Very Large-Language Models (VLLMs) and, thus, may represent the building blocks of many NLP systems solving downstream tasks. Hence, a little or a large bias in CtB-LLMs may cause huge harm. In this paper, we performed a large investigation of the bias of three families of CtB-LLMs, and we showed that debiasing techniques are effective and usable. Indeed, according to current tests, the LLaMA and the OPT families have an important bias in gender, race, religion, and profession. In contrast to the analysis for other LLMs, we discovered that bias depends not on the number of parameters but on the perplexity. Finally, the debiasing of OPT using LoRA reduces bias up to 4.12 points in the normalized stereotype score.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 36 of its 62 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

36 of 62 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  11. 2018Deep contextualized word representationscited by 2,290
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
24 more
  1. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  2. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  3. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  4. 2018Neural Network Acceptability Judgmentscited by 958
  5. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  6. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  7. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  8. 2017Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networkscited by 486
  9. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  10. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 296
  11. 2016Recommendations as Treatments: Debiasing Learning and Evaluationcited by 248
  12. 2019Gender Bias in Contextualized Word Embeddingscited by 203
  13. 2018Learning Gender-Neutral Word Embeddingscited by 169
  14. 2019Fair Regression: Quantitative Definitions and Reduction-based Algorithmscited by 168
  15. 2019Mitigate Bias in Face Recognition using Skewness-Aware Reinforcement Learningcited by 145
  16. 2021Debiasing Pre-trained Contextualised Embeddingscited by 107
  17. 2022Perturbation Augmentation for Fairer NLPcited by 52
  18. 2019Natural Language Understanding with the Quora Question Pairs Datasetcited by 51
  19. 2021Sample Selection for Fair and Robust Trainingcited by 48
  20. 2022Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Biascited by 33
  21. 2023Parameter-efficient Modularised Bias Mitigation via AdapterFusioncited by 25
  22. 2022Gender Biases and Where to Find Them: Exploring Gender Bias in Pre-Trained Transformer-based Language Models Using Movement Pruningcited by 21
  23. 2022The Dark Side of the Language: Pre-trained Transformers in the DarkNetcited by 10
  24. 2023Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languagescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 18; 2025: 5; 2026: 3.

32 papers.

Sort:

By area:

  1. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  2. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 84
  3. 2024Safeguarding Large Language Models: A Surveycited by 78
  4. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  5. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 66
  6. 2024Building Guardrails for Large Language Modelscited by 65
  7. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  8. 2024Spectral Editing of Activations for Large Language Model Alignmentcited by 56
  9. 2024Conformity in Large Language Modelscited by 34
  10. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 26
  11. 2024Investigating Implicit Bias in Large Language Models: A Large-Scale Study of Over 50 LLMscited by 25
  12. 2023Debiasing Algorithm through Model Adaptationcited by 24
20 more
  1. 2023Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Modelcited by 21
  2. 2023Indian-BhED: A Dataset for Measuring India-Centric Biases in Large Language Modelscited by 18
  3. 2025Understanding and Mitigating Gender Bias in LLMs via Interpretable Neuron Editingcited by 14
  4. 2024Social Bias in Large Language Models For Bangla: An Empirical Study on Gender and Religious Biascited by 13
  5. 2023Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societiescited by 11
  6. 2024Potential and Challenges of Model Editing for Social Debiasingcited by 10
  7. 2024'Since Lawyers are Males..': Examining Implicit Gender Bias in Hindi Language Generation by LLMscited by 8
  8. 2025Unsupervised Concept Vector Extraction for Bias Control in LLMscited by 7
  9. 2024The Lou Dataset -- Exploring the Impact of Gender-Fair Language in German Text Classificationcited by 5
  10. 2024Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigationcited by 5
  11. 2024Interpreting Bias in Large Language Models: A Feature-Based Approachcited by 4
  12. 2024LIDAO: Towards Limited Interventions for Debiasing (Large) Language Modelscited by 3
  13. 2024Runtime Monitoring and Enforcement of Conditional Fairness in Generative AIscited by 2
  14. 2024Evaluating Gender Bias in Large Language Modelscited by 2
  15. 2026Intersectional Sycophancy: How Perceived User Demographics Shape False Validation in Large Language Modelscited by 2
  16. 2026DeFrame: Debiasing Large Language Models Against Framing Effectscited by 1
  17. 2024Reducing Large Language Model Bias with Emphasis on 'Restricted Industries': Automated Dataset Augmentation and Prejudice Quantificationcited by 0
  18. 2025A Survey on Stereotype Detection in Natural Language Processingcited by 0
  19. 2025FairLangProc: A Python package for fairness in NLPcited by 0
  20. 2026Bias Dynamics in BabyLMs: Towards a Compute-Efficient Sandbox for Democratising Pre-Training Debiasingcited by 0