Omissive Bias in Religious Representation: Benchmarking LLM Answers to Everyday Ethical Decision-making
Not on the map. Why?
Abstract (arXiv)
As large language models become a default source of guidance on personal, moral, and existential questions, it matters whether they draw on the religious frameworks that have historically shaped such reasoning, or systematically omit them. In this paper, we ask a deliberately narrow question: when posed an everyday ethical question for which religious perspectives may be valuable, do LLMs invoke religion at all? In contrast to benchmarks that look for the presence of political leanings or social bias, we look for the absence of religious representation as a dimension of value alignment and bias in LLMs. We term this ``omissive bias.'' To measure omissive bias, we contribute the AllFaith Religious Representation Benchmark: 150 ethically and personally salient questions, sourced from in-the-wild chat transcripts and faith-community contributors, paired with an LLM-as-judge rubric that gives full credit for any mention of a religion, a religious practice, or a religious leader. The questions are not themselves about religion--they are open-ended questions about grief, forgiveness, relationships, purpose, and honesty, where religion is one valuable perspective among several. We also run a human-subjects survey to compare LLM behavior against human expectations. Evaluating 27 models, we find that LLMs consistently underrepresent religion relative to human expectations. The omission is asymmetric: models invoke religion more readily for abstract existential questions (meaning, death, truth) than for the practical personal situations--grief, marriage, family conflict, addiction--where many people most rely on it. It is not our purpose to adjudicate which values LLMs should hold. We argue, more modestly, that current LLM responses overlook critical opportunities to reflect religious frameworks that many people draw on when navigating personal and ethical challenges.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 38 of its 49 references that are in the map. First posted on arXiv 23 May 2026.
Builds on
38 of 49 references.
References · 38
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
- 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
- 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
- 2023Whose Opinions Do Language Models Reflect?cited by 759
- 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
- 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
- 2023Bias and Fairness in Large Language Models: A Surveycited by 626
- 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
- 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
26 more
- 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
- 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
- 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
- 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 203
- 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
- 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
- 2024Investigating Cultural Alignment of Large Language Modelscited by 140
- 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
- 2021Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologiescited by 128
- 2021On Measures of Biases and Harms in NLPcited by 81
- 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
- 2024Large Language Models Reflect the Ideology of their Creatorscited by 56
- 2025MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomescited by 22
- 2023What does ChatGPT return about human values? Exploring value bias in ChatGPT using a descriptive value theorycited by 19
- 2023Indian-BhED: A Dataset for Measuring India-Centric Biases in Large Language Modelscited by 17
- 2025An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliabilitycited by 17
- 2023Geographical Erasure in Language Generationcited by 15
- 2025How Deep Is Representational Bias in LLMs? The Cases of Caste and Religioncited by 14
- 2024Divine LLaMAs: Bias, Stereotypes, Stigmatization, and Emotion Representation of Religion in Large Language Modelscited by 12
- 2025SATA-BENCH: Select All That Apply Benchmark for Multiple Choice Questionscited by 9
- 2023Muslim-Violence Bias Persists in Debiased GPT Modelscited by 4
- 2024Laissez-Faire Harms: Algorithmic Biases in Generative Language Modelscited by 4
- 2025SortBench: Benchmarking LLMs based on their ability to sort listscited by 4
- 2025Characterizing Selective Refusal Bias in Large Language Modelscited by 4
- 2026Navigating the Shift: A Comparative Analysis of Web Search and Generative AI Response Generationcited by 2
- 2023Western, Religious or Spiritual: An Evaluation of Moral Justification in Large Language Modelscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.