Benchmarking and Building Zero-Shot Hindi Retrieval Model with Hindi-BEIR and NLLB-E5
Not on the map. Why?
Abstract (arXiv)
Given the large number of Hindi speakers worldwide, there is a pressing need for robust and efficient information retrieval systems for Hindi. Despite ongoing research, comprehensive benchmarks for evaluating retrieval models in Hindi are lacking. To address this gap, we introduce the Hindi-BEIR benchmark, comprising 15 datasets across seven distinct tasks. We evaluate state-of-the-art multilingual retrieval models on the Hindi-BEIR benchmark, identifying task and domain-specific challenges that impact Hindi retrieval performance. Building on the insights from these results, we introduce NLLB-E5, a multilingual retrieval model that leverages a zero-shot approach to support Hindi without the need for Hindi training data. We believe our contributions, which include the release of the Hindi-BEIR benchmark and the NLLB-E5 model, will prove to be a valuable resource for researchers and promote advancements in multilingual retrieval models.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 24 of its 41 references that are in the map. First posted on arXiv 9 Sep 2024.
Builds on
24 of 41 references.
References · 24
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
- 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
- 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
- 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
- 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
- 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
- 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
- 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
- 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 149
12 more
- 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 118
- 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116
- 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
- 2024IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languagescited by 52
- 2022Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languagescited by 47
- 2024The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model designcited by 20
- 2024MedSumm: A Multimodal Approach to Summarizing Code-Mixed Hindi-English Clinical Queriescited by 19
- 2023BEIR-PL: Zero Shot Information Retrieval Benchmark for the Polish Languagecited by 15
- 2023MILDSum: A Novel Benchmark Dataset for Multilingual Summarization of Indian Legal Case Judgmentscited by 10
- 2024Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languagescited by 9
- 2023IndicIRSuite: Multilingual Dataset and Neural Information Models for Indian Languagescited by 5
- 2023Breaking Language Barriers: A Question Answering Dataset for Hindi and Marathicited by 4
Built on by
Citing papers per year: 2025: 2.
2 papers.