# QASiNa: Religious Domain Question Answering using Sirah Nabawiyah

arXiv: https://arxiv.org/abs/2310.08102 · HTML: https://mlsanity.com/paper/2310.08102/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 40 references that are in the map.
- First posted on arXiv 12 Oct 2023.

## Builds on (most cited first, 7 of 7)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Evaluating Theory of Mind in Question Answering](https://mlsanity.com/paper/1808.09352/) (2018)
- [IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP](https://mlsanity.com/paper/2011.00677/) (2020)
- [Sequence-to-Sequence Learning for Indonesian Automatic Question Generator](https://mlsanity.com/paper/2009.13889/) (2020)
- [IDK-MRC: Unanswerable Questions for Indonesian Machine Reading Comprehension](https://mlsanity.com/paper/2210.13778/) (2022)
- [Finding Answers from the Word of God: Domain Adaptation for Neural Networks in Biblical Question Answering](https://mlsanity.com/paper/1810.12118/) (2018)

## Built on by (most cited first, 15 of 15)

- [PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models](https://mlsanity.com/paper/2402.07867/) (2024)
- [SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages](https://mlsanity.com/paper/2406.10118/) (2024)
- [Practical Poisoning Attacks against Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.03957/) (2025)
- [One Shot Dominance: Knowledge Poisoning Attack on Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2505.11548/) (2025)
- [GraphRAG under Fire](https://mlsanity.com/paper/2501.14050/) (2025)
- [Sacred or Synthetic? Evaluating LLM Reliability and Abstention for Religious Questions](https://mlsanity.com/paper/2508.08287/) (2025)
- [Overcoming Data Scarcity in Generative Language Modelling for Low-Resource Languages: A Systematic Review](https://mlsanity.com/paper/2505.04531/) (2025)
- [UniC-RAG: Universal Knowledge Corruption Attacks to Retrieval-Augmented Generation](https://mlsanity.com/paper/2508.18652/) (2025)
- [EmoRAG: Evaluating RAG Robustness to Symbolic Perturbations](https://mlsanity.com/paper/2512.01335/) (2025)
- [Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Models](https://mlsanity.com/paper/2503.16581/) (2025)
- [SEA-BED: How Do Embedding Models Represent Southeast Asian Languages?](https://mlsanity.com/paper/2508.12243/) (2025)
- [Bridging Language Gaps with Adaptive RAG: Improving Indonesian Language Question Answering](https://mlsanity.com/paper/2510.21068/) (2025)
- [RefineRAG: Word-Level Poisoning Attacks via Retriever-Guided Text Refinement](https://mlsanity.com/paper/2604.07403/) (2026)
- [Hadith computational science in the age of large language models: a critical narrative review](https://mlsanity.com/paper/2608.20364/) (2026)
- [Hop-Decayed Influence: New Vulnerabilities of Structural Auxiliary Indexing in GraphRAG Pipelines with LLM](https://mlsanity.com/paper/2610.02373/) (2026)

## Abstract (arXiv)

Nowadays, Question Answering (QA) tasks receive significant research focus, particularly with the development of Large Language Model (LLM) such as Chat GPT [1]. LLM can be applied to various domains, but it contradicts the principles of information transmission when applied to the Islamic domain. In Islam we strictly regulates the sources of information and who can give interpretations or tafseer for that sources [2]. The approach used by LLM to generate answers based on its own interpretation is similar to the concept of tafseer, LLM is neither an Islamic expert nor a human which is not permitted in Islam. Indonesia is the country with the largest Islamic believer population in the world [3]. With the high influence of LLM, we need to make evaluation of LLM in religious domain. Currently, there is only few religious QA dataset available and none of them using Sirah Nabawiyah especially in Indonesian Language. In this paper, we propose the Question Answering Sirah Nabawiyah (QASiNa) dataset, a novel dataset compiled from Sirah Nabawiyah literatures in Indonesian language. We demonstrate our dataset by using mBERT [4], XLM-R [5], and IndoBERT [6] which fine-tuned with Indonesian translation of SQuAD v2.0 [7]. XLM-R model returned the best performance on QASiNa with EM of 61.20, F1-Score of 75.94, and Substring Match of 70.00. We compare XLM-R performance with Chat GPT-3.5 and GPT-4 [1]. Both Chat GPT version returned lower EM and F1-Score with higher Substring Match, the gap of EM and Substring Match get wider in GPT-4. The experiment indicate that Chat GPT tends to give excessive interpretations as evidenced by its higher Substring Match scores compared to EM and F1-Score, even after providing instruction and context. This concludes Chat GPT is unsuitable for question answering task in religious domain especially for Islamic religion.