mlsanityData to 7 Oct 2026

mFollowIR: a Multilingual Benchmark for Instruction Following in Retrieval

arXiv 2501.19264 ↗First posted 31 Jan 2025cs.IR cs.CL cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Retrieval systems generally focus on web-style queries that are short and underspecified. However, advances in language models have facilitated the nascent rise of retrieval models that can understand more complex queries with diverse intents. However, these efforts have focused exclusively on English; therefore, we do not yet understand how they work across languages. We introduce mFollowIR, a multilingual benchmark for measuring instruction-following ability in retrieval models. mFollowIR builds upon the TREC NeuCLIR narratives (or instructions) that span three diverse languages (Russian, Chinese, Persian) giving both query and instruction to the retrieval models. We make small changes to the narratives and isolate how well retrieval models can follow these nuanced changes. We present results for both multilingual (XX-XX) and cross-lingual (En-XX) performance. We see strong cross-lingual performance with English-based retrievers that trained using instructions, but find a notable drop in performance in the multilingual setting, indicating that more work is needed in developing data for instruction-based multilingual retrievers.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 44 of its 65 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

44 of 65 references.

References · 44

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2023Mistral 7Bcited by 3,065
  5. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  7. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  8. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  9. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  11. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  12. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
32 more
  1. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  2. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  3. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  4. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  5. 2023Improving Text Embeddings with Large Language Modelscited by 295
  6. 2020Overview of the TREC 2019 deep learning trackcited by 282
  7. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  8. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  9. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  10. 2024Generative Representational Instruction Tuningcited by 227
  11. 2023Large language models can accurately predict searcher preferencescited by 129
  12. 2020XOR QA: Cross-lingual Open-Retrieval Question Answeringcited by 121
  13. 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116
  14. 2022Task-aware Retrieval with Instructionscited by 107
  15. 2024NV-Retriever: Improving text embedding models with effective hard-negative miningcited by 83
  16. 2020Learning from Task Descriptionscited by 78
  17. 2024Making Text Embedders Few-Shot Learnerscited by 78
  18. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 67
  19. 2018Pytrec_eval: An Extremely Fast Python Interface to trec_evalcited by 46
  20. 2024Following Length Constraints in Instructionscited by 45
  21. 2020Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrievalcited by 44
  22. 2024Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Modelscited by 39
  23. 2022Transfer Learning Approaches for Building Cross-Language Dense Retrieval Modelscited by 36
  24. 2024INSTRUCTIR: A Benchmark for Instruction Following of Information Retrieval Modelscited by 32
  25. 2023NevIR: Negation in Neural Information Retrievalcited by 28
  26. 2024MAIR: A Massive Benchmark for Evaluating Instructed Retrievalcited by 23
  27. 2024Overview of the TREC 2023 NeuCLIR Trackcited by 17
  28. 2024Beyond Relevance: Evaluate and Improve Retrievers on Perspective Awarenesscited by 12
  29. 2023Overview of the TREC 2022 NeuCLIR Trackcited by 11
  30. 2024Open-World Evaluation for Retrieving Diverse Perspectivescited by 11
  31. 2023NeuralMind-UNICAMP at 2022 TREC NeuCLIR: Large Boring Rerankers for Cross-lingual Retrievalcited by 7
  32. 2024HLTCOE at TREC 2023 NeuCLIR Trackcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025On the Theoretical Limitations of Embedding-Based Retrievalcited by 98
  2. 2025Seq vs Seq: An Open Suite of Paired Encoders and Decoderscited by 34
  3. 2025IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrievalcited by 16
  4. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  5. 2025Evaluating Large Language Models for Cross-Lingual Retrievalcited by 5
  6. 2025The Medium Is Not the Message: Deconfounding Document Embeddings via Linear Concept Erasurecited by 1
  7. 2026ℝ2k is Theoretically Large Enough for Embedding-based Top-k Retrievalcited by 0