# Exploring Bengali Religious Dialect Biases in Large Language Models with Evaluation Perspectives

arXiv: https://arxiv.org/abs/2407.18376 · HTML: https://mlsanity.com/paper/2407.18376/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 25 Jul 2024.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- [Reducing Sentiment Bias in Language Models via Counterfactual Evaluation](https://mlsanity.com/paper/1911.03064/) (2019)
- [Re-imagining Algorithmic Fairness in India and Beyond](https://mlsanity.com/paper/2101.09995/) (2021)
- [Mitigating Language-Dependent Ethnic Bias in BERT](https://mlsanity.com/paper/2109.05704/) (2021)
- [Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Bias](https://mlsanity.com/paper/2010.14534/) (2020)
- [EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria](https://mlsanity.com/paper/2309.13633/) (2023)
- [Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory](https://mlsanity.com/paper/2305.14889/) (2023)
- [The Unequal Opportunities of Large Language Models: Revealing Demographic Bias through Job Recommendations](https://mlsanity.com/paper/2308.02053/) (2023)
- [Fairness On The Ground: Applying Algorithmic Fairness Approaches to Production Systems](https://mlsanity.com/paper/2103.06172/) (2021)
- [Discovering and Categorising Language Biases in Reddit](https://mlsanity.com/paper/2008.02754/) (2020)
- [BanglaAutoKG: Automatic Bangla Knowledge Graph Construction with Semantic Neural Graph Filtering](https://mlsanity.com/paper/2404.03528/) (2024)

## Built on by (most cited first, 4 of 4)

- [Cross-Lingual Sentiment Misalignment: Auditing Multilingual Language Models for Inversion Risk, Dialectal Representation, and Affective Stability](https://mlsanity.com/paper/2602.17469/) (2026)
- [Exploring Possibilities of AI-Powered Legal Assistance in Bangladesh through Large Language Modeling](https://mlsanity.com/paper/2410.17210/) (2024)
- [BengaliMoralBench: A Benchmark for Auditing Moral Reasoning in Large Language Models within Bengali Language and Culture](https://mlsanity.com/paper/2511.03180/) (2025)
- [Mind the Gap: Pitfalls of LLM Alignment with Asian Public Opinion](https://mlsanity.com/paper/2603.06264/) (2026)

## Abstract (arXiv)

While Large Language Models (LLM) have created a massive technological impact in the past decade, allowing for human-enabled applications, they can produce output that contains stereotypes and biases, especially when using low-resource languages. This can be of great ethical concern when dealing with sensitive topics such as religion. As a means toward making LLMS more fair, we explore bias from a religious perspective in Bengali, focusing specifically on two main religious dialects: Hindu and Muslim-majority dialects. Here, we perform different experiments and audit showing the comparative analysis of different sentences using three commonly used LLMs: ChatGPT, Gemini, and Microsoft Copilot, pertaining to the Hindu and Muslim dialects of specific words and showcasing which ones catch the social biases and which do not. Furthermore, we analyze our findings and relate them to potential reasons and evaluation perspectives, considering their global impact with over 300 million speakers worldwide. With this work, we hope to establish the rigor for creating more fairness in LLMs, as these are widely used as creative writing agents.