mlsanityData to 14 Oct 2026

Multi-stage Training of Bilingual Islamic LLM for Neural Passage Retrieval

arXiv 2501.10175 ↗First posted 17 Jan 2025cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

This study examines the use of Natural Language Processing (NLP) technology within the Islamic domain, focusing on developing an Islamic neural retrieval model. By leveraging the robust XLM-R model, the research employs a language reduction technique to create a lightweight bilingual large language model (LLM). Our approach for domain adaptation addresses the unique challenges faced in the Islamic domain, where substantial in-domain corpora exist only in Arabic while limited in other languages, including English. The work utilizes a multi-stage training process for retrieval models, incorporating large retrieval datasets, such as MS MARCO, and smaller, in-domain datasets to improve retrieval performance. Additionally, we have curated an in-domain retrieval dataset in English by employing data augmentation techniques and involving a reliable Islamic source. This approach enhances the domain-specific dataset for retrieval, leading to further performance gains. The findings suggest that combining domain adaptation and a multi-stage training method for the bilingual Islamic neural retrieval model enables it to outperform monolingual models on downstream retrieval tasks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 38 of its 50 references that are in the map. First posted on arXiv 17 Jan 2025.

Builds on

38 of 50 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  10. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  11. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  12. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
26 more
  1. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  2. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
  3. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  4. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  5. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  6. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 493
  7. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 413
  8. 2017Efficient Natural Language Response Suggestion for Smart Replycited by 229
  9. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 176
  10. 2018QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehensioncited by 164
  11. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 153
  12. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
  13. 2020Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Taskscited by 138
  14. 2019Learning Dense Representations for Entity Retrievalcited by 122
  15. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 119
  16. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  17. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  18. 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116
  19. 2018A Survey on Methods and Theories of Quantized Neural Networkscited by 75
  20. 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67
  21. 2021Efficient Domain Adaptation of Language Models via Adaptive Tokenizationcited by 34
  22. 2020Load What You Need: Smaller Versions of Multilingual BERTcited by 32
  23. 2020Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QAcited by 23
  24. 2023Leveraging Domain Adaptation and Data Augmentation to Improve Qur'anic IR in English and Arabiccited by 4
  25. 2022SAMP: A Model Inference Toolkit of Post-Training Quantization for Text Processing via Self-Adaptive Mixed-Precisioncited by 3
  26. 2024Building an Efficient Multilingual Non-Profit IR System for the Islamic Domain Leveraging Multiprocessing Design in Rustcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025FRAME: Boosting LLMs with A Four-Quadrant Multi-Stage Pretraining Strategycited by 4
  2. 2025Efficient and Versatile Model for Multilingual Information Retrieval of Islamic Text: Development and Deployment in Real-World Scenarioscited by 1
  3. 2025MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learningcited by 0
  4. 2026What Makes a Good Fiqh Retriever? Answer Retrieval for Arabic Islamic Jurisprudencecited by 0