Detecting Bias in Large Language Models: Fine-tuned KcBERT
Not on the map. Why?
Abstract (arXiv)
The rapid advancement of large language models (LLMs) has enabled natural language processing capabilities similar to those of humans, and LLMs are being widely utilized across various societal domains such as education and healthcare. While the versatility of these models has increased, they have the potential to generate subjective and normative language, leading to discriminatory treatment or outcomes among social groups, especially due to online offensive language. In this paper, we define such harm as societal bias and assess ethnic, gender, and racial biases in a model fine-tuned with Korean comments using Bidirectional Encoder Representations from Transformers (KcBERT) and KOLD data through template-based Masked Language Modeling (MLM). To quantitatively evaluate biases, we employ LPBS and CBS metrics. Compared to KcBERT, the fine-tuned model shows a reduction in ethnic bias but demonstrates significant changes in gender and racial biases. Based on these results, we propose two methods to mitigate societal bias. Firstly, a data balancing approach during the pre-training phase adjusts the uniformity of data by aligning the distribution of the occurrences of specific words and converting surrounding harmful words into non-harmful words. Secondly, during the in-training phase, we apply Debiasing Regularization by adjusting dropout and regularization, confirming a decrease in training loss. Our contribution lies in demonstrating that societal bias exists in Korean language models due to language-dependent characteristics.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 22 of its 33 references that are in the map. First posted on arXiv 16 Mar 2024.
Builds on
22 of 33 references.
References · 22
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
- 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
- 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
- 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
- 2018Mitigating Unwanted Biases with Adversarial Learningcited by 663
- 2023Bias and Fairness in Large Language Models: A Surveycited by 626
- 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
- 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
- 2019On Measuring Social Biases in Sentence Encoderscited by 397
- 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
- 2019Measuring Bias in Contextualized Word Representationscited by 275
- 2020Measuring and Reducing Gendered Correlations in Pre-trained Modelscited by 222
10 more
- 2018Gender Bias in Neural Natural Language Processingcited by 210
- 2019On Measuring and Mitigating Biased Inferences of Word Embeddingscited by 112
- 2021Mitigating Language-Dependent Ethnic Bias in BERTcited by 87
- 2022Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Listscited by 54
- 2022Unified Detoxifying and Debiasing in Language Generation via Inference-time Adaptive Optimizationcited by 32
- 2023On the Independence of Association Bias and Empirical Fairness in Language Modelscited by 29
- 2022Deep Learning on a Healthy Data Diet: Finding Important Examples for Fairnesscited by 21
- 2023Gender Bias in BERT -- Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Taskcited by 21
- 2022KOLD: Korean Offensive Language Datasetcited by 14
- 2020Neutralizing Gender Bias in Word Embedding with Latent Disentanglement and Counterfactual Generationcited by 10
Built on by
Citing papers per year: 2026: 2.
2 papers.