mlsanityData to 14 Oct 2026

FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions

arXiv 2403.15246 ↗First posted 22 Mar 2024cs.IR cs.CL cs.LGcited by 69

Not on the map. Why?

Abstract (arXiv)

Modern Language Models (LMs) are capable of following long and complex instructions that enable a large and diverse set of user requests. While Information Retrieval (IR) models use these LMs as the backbone of their architectures, virtually none of them allow users to provide detailed instructions alongside queries, thus limiting their ability to satisfy complex information needs. In this work, we study the use of instructions in IR systems. First, we introduce our dataset FollowIR, which contains a rigorous instruction evaluation benchmark as well as a training set for helping IR models learn to better follow real-world instructions. FollowIR repurposes detailed instructions -- also known as narratives -- developed for professional assessors to evaluate retrieval systems. In particular, we build our benchmark from three collections curated for shared tasks at the Text REtrieval Conference (TREC). These collections contains hundreds to thousands of labeled documents per query, making them suitable for our exploration. Through this process, we can measure how well IR models follow instructions, through a new pairwise evaluation framework. Our results indicate that existing retrieval models fail to correctly use instructions, using them for basic keywords and struggling to understand long-form information. However, we show that it is possible for IR models to learn to follow complex instructions: our new FollowIR-7B model has significant improvements after fine-tuning on our training set.

Key facts. Cited by 69 papers in the map (data through 14 Oct 2026). Builds on 54 of its 72 references that are in the map. First posted on arXiv 22 Mar 2024.

Builds on

54 of 72 references.

References · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
38 more
  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  5. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  6. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  8. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  9. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  10. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  11. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  12. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  13. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  14. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  15. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  16. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  17. 2019Passage Re-ranking with BERTcited by 645
  18. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  19. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  20. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  21. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  22. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 376
  23. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 333
  24. 2023Improving Text Embeddings with Large Language Modelscited by 303
  25. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  26. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 272
  27. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 248
  28. 2024Generative Representational Instruction Tuningcited by 233
  29. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  30. 2019Multi-Stage Document Ranking with BERTcited by 201
  31. 2023RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!cited by 155
  32. 2019Deeper Text Understanding for IR with Contextual Neural Language Modelingcited by 113
  33. 2022Task-aware Retrieval with Instructionscited by 107
  34. 2023AnglE-optimized Text Embeddingscited by 99
  35. 2020Learning from Task Descriptionscited by 78
  36. 2024INSTRUCTIR: A Benchmark for Instruction Following of Information Retrieval Modelscited by 33
  37. 2023NevIR: Negation in Neural Information Retrievalcited by 29
  38. 2023When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasetscited by 26

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 32; 2026: 24.

69 papers.

Sort:

By area:

  1. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  2. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 158
  3. 2025On the Theoretical Limitations of Embedding-Based Retrievalcited by 104
  4. 2024Making Text Embedders Few-Shot Learnerscited by 79
  5. 2025Rank1: Test-Time Compute for Reranking in Information Retrievalcited by 70
  6. 2025Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Modelscited by 47
  7. 2024Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Modelscited by 40
  8. 2025Seq vs Seq: An Open Suite of Paired Encoders and Decoderscited by 38
  9. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  10. 2025Rank-K: Test-Time Reasoning for Listwise Rerankingcited by 29
  11. 2025Diffusion vs. Autoregressive Language Models: A Text Embedding Perspectivecited by 23
  12. 2024ChatRetriever: Adapting Large Language Models for Generalized and Robust Conversational Dense Retrievalcited by 19
38 more
  1. 2025DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrieverscited by 19
  2. 2024CLERC: A Dataset for Legal Case Retrieval and Retrieval-Augmented Analysis Generationcited by 18
  3. 2025Guiding Retrieval using LLM-based Listwise Rankerscited by 16
  4. 2025IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrievalcited by 16
  5. 2025FLAWS: A Benchmark for Error Identification and Localization in Scientific Paperscited by 14
  6. 2024A Survey of Controllable Learning: Methods and Applications in Information Retrievalcited by 12
  7. 2024RouterRetriever: Routing over a Mixture of Expert Embedding Modelscited by 12
  8. 2025MIR: Methodology Inspiration Retrieval for Scientific Research Problemscited by 11
  9. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  10. 2025Let LLMs Speak Embedding Languages: Generative Text Embeddings via Iterative Contrastive Refinementcited by 10
  11. 2025Towards Better Instruction Following Retrieval Modelscited by 9
  12. 2026LaSER: Internalizing Explicit Reasoning into Latent Space for Dense Retrievalcited by 9
  13. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  14. 2025mFollowIR: a Multilingual Benchmark for Instruction Following in Retrievalcited by 7
  15. 2025Hypencoder: Hypernetworks for Information Retrievalcited by 7
  16. 2025Don't Reinvent the Wheel: Efficient Instruction-Following Text Embedding based on Guided Space Transformationcited by 7
  17. 2025CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrievalcited by 6
  18. 2025ERank: Fusing Supervised Fine-Tuning and Reinforcement Learning for Effective and Efficient Text Rerankingcited by 6
  19. 2025Benchmarking Information Retrieval Models on Complex Retrieval Taskscited by 6
  20. 2025ATEB: Evaluating and Improving Advanced NLP Tasks for Text Embedding Modelscited by 5
  21. 2025ImpRAG: Retrieval-Augmented Generation with Implicit Queriescited by 5
  22. 2025Beyond Sequential Reranking: Reranker-Guided Search Improves Reasoning Intensive Retrievalcited by 5
  23. 2025Evaluating Large Language Models for Cross-Lingual Retrievalcited by 5
  24. 2025When Models Can't Follow: Testing Instruction Adherence Across 256 LLMscited by 5
  25. 2026BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMscited by 5
  26. 2024Instruction Embedding: Latent Representations of Instructions Towards Task Identificationcited by 4
  27. 2025Exploiting Instruction-Following Retrievers for Malicious Information Retrievalcited by 4
  28. 2024Disentangling Questions from Query Generation for Task-Adaptive Retrievalcited by 3
  29. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  30. 2024RARe: Retrieval Augmented Retrieval with In-Context Examplescited by 2
  31. 2025A Vision for Geo-Temporal Deep Research Systems: Towards Comprehensive, Transparent, and Reproducible Geo-Temporal Information Synthesiscited by 2
  32. 2025SEA-BED: How Do Embedding Models Represent Southeast Asian Languages?cited by 2
  33. 2026Align Then Adapt: Label-Efficient Adapter Learning for Asymmetric Dense Retrievalcited by 2
  34. 2024RE-AdaptIR: Improving Information Retrieval through Reverse Engineered Adaptationcited by 1
  35. 2024Learning to Explore and Select for Coverage-Conditioned Retrieval-Augmented Generationcited by 1
  36. 2025LIR: The First Workshop on Late Interaction and Multi Vector Retrieval @ ECIR 2026cited by 1
  37. 2026ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrievalcited by 1
  38. 2026SEIF: Self-Evolving Reinforcement Learning for Instruction Followingcited by 1

50 most cited of 69 All 69 (JSON)