# IndicRAGSuite: Large-Scale Datasets and a Benchmark for Indian Language RAG Systems

arXiv: https://arxiv.org/abs/2506.01615 · HTML: https://mlsanity.com/paper/2506.01615/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 16 of its 30 references that are in the map.
- First posted on arXiv 2 Jun 2025.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 16 of 16)

- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2010.08191/) (2020)
- [Text and Code Embeddings by Contrastive Pre-Training](https://mlsanity.com/paper/2201.10005/) (2022)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval](https://mlsanity.com/paper/2108.08787/) (2021)
- [mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset](https://mlsanity.com/paper/2108.13897/) (2021)
- [IndicIRSuite: Multilingual Dataset and Neural Information Models for Indian Languages](https://mlsanity.com/paper/2312.09508/) (2023)

## Built on by (most cited first, 2 of 2)

- [BhashaSutra: A Task-Centric Unified Survey of Indian NLP Datasets, Corpora, and Resources](https://mlsanity.com/paper/2604.18423/) (2026)
- [Task-Conditional Flow Matching for Balanced Multilingual Text Embedding Adaptation](https://mlsanity.com/paper/2608.05785/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) systems enable language models to access relevant information and generate accurate, well-grounded, and contextually informed responses. However, for Indian languages, the development of high-quality RAG systems is hindered by the lack of two critical resources: (1) evaluation benchmarks for retrieval and generation tasks, and (2) large-scale training datasets for multilingual retrieval. Most existing benchmarks and datasets are centered around English or high-resource languages, making it difficult to extend RAG capabilities to the diverse linguistic landscape of India. To address the lack of evaluation benchmarks, we create IndicMSMarco, a multilingual benchmark for evaluating retrieval quality and response generation in 13 Indian languages, created via manual translation of 1000 diverse queries from MS MARCO-dev set. To address the need for training data, we build a large-scale dataset of (question, answer, relevant passage) tuples derived from the Wikipedias of 19 Indian languages using state-of-the-art LLMs. Additionally, we include translated versions of the original MS MARCO dataset to further enrich the training data and ensure alignment with real-world information-seeking tasks. Resources are available here: https://huggingface.co/collections/ai4bharat/indicragsuite-683e7273cb2337208c8c0fcb