mlsanityData to 7 Oct 2026

SectEval: Evaluating the Latent Sectarian Preferences of Large Language Models

arXiv 2603.12768 ↗First posted 13 Mar 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

As Large Language Models (LLMs) becomes a popular source for religious knowledge, it is important to know if it treats different groups fairly. This study is the first to measure how LLMs handle the differences between the two main sects of Islam: Sunni and Shia. We present a test called SectEval, available in both English and Hindi, consisting of 88 questions, to check the bias-ness of 15 top LLM models, both proprietary and open-weights. Our results show a major inconsistency based on language. In English, many powerful models DeepSeek-v3 and GPT-4o often favored Shia answers. However, when asked the exact same questions in Hindi, these models switched to favoring Sunni answers. This means a user could get completely different religious advice just by changing languages. We also looked at how models react to location. Advanced models Claude-3.5 changed their answers to match the user's country-giving Shia answers to a user from Iran and Sunni answers to a user from Saudi Arabia. In contrast, smaller models (especially in Hindi) ignored the user's location and stuck to a Sunni viewpoint. These findings show that AI is not neutral; its religious ``truth'' changes depending on the language you speak and the country you claim to be from. The data set is available at https://github.com/secteval/SectEval/

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 13 of its 24 references that are in the map. First posted on arXiv 13 Mar 2026.

Builds on

13 of 24 references.

References · 13

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  4. 2024Large Language Models: A Surveycited by 506
  5. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  6. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 203
  7. 2025PalmX 2025: The First Shared Task on Benchmarking LLMs on Arabic and Islamic Culturecited by 20
  8. 2025DECASTE: Unveiling Caste Stereotypes in Large Language Models through Multi-Dimensional Bias Analysiscited by 12
  9. 2025Sacred or Synthetic? Evaluating LLM Reliability and Abstention for Religious Questionscited by 10
  10. 2025Assessing Large Language Models on Islamic Legal Reasoning: Evidence from Inheritance Law Evaluationcited by 10
  11. 2025FairI Tales: Evaluation of Fairness in Indian Contexts with a Focus on Bias and Stereotypescited by 9
  12. 2024A Benchmark Dataset with Larger Context for Non-Factoid Question Answering over Islamic Textcited by 4
1 more
  1. 2025Is Lying Only Sinful in Islam? Exploring Religious Bias in Multilingual Large Language Models Across Major Religionscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Mind the Gap: Pitfalls of LLM Alignment with Asian Public Opinion
  2. 2025Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Models
  3. 2025Surfacing Subtle Stereotypes: A Multilingual, Debate-Oriented Evaluation of Modern LLMs
  4. 2024Improving LLM Reliability with RAG in Religious Question-Answering: MufassirQAS
  5. 2026From RAG to Agentic RAG for Faithful Islamic Question Answering
  6. 2026Islamic Large Language Models: From Knowledge Acquisition to Trustworthy and Hallucination-Resistant AI
  7. 2025DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Models
  8. 2025Beyond MCQ: An Open-Ended Arabic Cultural QA Benchmark with Dialect Variants