# MAIR: A Massive Benchmark for Evaluating Instructed Retrieval

arXiv: https://arxiv.org/abs/2410.10127 · HTML: https://mlsanity.com/paper/2410.10127/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2024.

## Built on by (most cited first, 23 of 23)

- [Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models](https://mlsanity.com/paper/2503.01763/) (2025)
- [τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge](https://mlsanity.com/paper/2603.04370/) (2026)
- [LLMs are Also Effective Embedding Models: An In-depth Overview](https://mlsanity.com/paper/2412.12591/) (2024)
- [LLM4Ranking: An Easy-to-use Framework of Utilizing Large Language Models for Document Reranking](https://mlsanity.com/paper/2504.07439/) (2025)
- [CoRanking: Collaborative Ranking with Small and Large Ranking Agents](https://mlsanity.com/paper/2503.23427/) (2025)
- [Towards Better Instruction Following Retrieval Models](https://mlsanity.com/paper/2505.21439/) (2025)
- [mFollowIR: a Multilingual Benchmark for Instruction Following in Retrieval](https://mlsanity.com/paper/2501.19264/) (2025)
- [XtraGPT: Context-Aware and Controllable Academic Paper Revision via Human-AI Collaboration](https://mlsanity.com/paper/2505.11336/) (2025)
- [Don't Reinvent the Wheel: Efficient Instruction-Following Text Embedding based on Guided Space Transformation](https://mlsanity.com/paper/2505.24754/) (2025)
- [MIRB: Mathematical Information Retrieval Benchmark](https://mlsanity.com/paper/2505.15585/) (2025)
- [Benchmarking Information Retrieval Models on Complex Retrieval Tasks](https://mlsanity.com/paper/2509.07253/) (2025)
- [mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphs](https://mlsanity.com/paper/2505.11180/) (2025)
- [Think Before You Retrieve: Learning Test-Time Adaptive Search with Small Language Models](https://mlsanity.com/paper/2511.07581/) (2025)
- [DisastIR: A Comprehensive Information Retrieval Benchmark for Disaster Management](https://mlsanity.com/paper/2505.15856/) (2025)
- [Making Large Language Models Efficient Dense Retrievers](https://mlsanity.com/paper/2512.20612/) (2025)
- [OrdRankBen: A Novel Ranking Benchmark for Ordinal Relevance in NLP](https://mlsanity.com/paper/2503.00674/) (2025)
- [Align Then Adapt: Label-Efficient Adapter Learning for Asymmetric Dense Retrieval](https://mlsanity.com/paper/2604.03403/) (2026)
- [From HNSW to Information-Theoretic Binarization: Rethinking the Architecture of Scalable Vector Search](https://mlsanity.com/paper/2601.11557/) (2025)
- [Continual Pretraining on Encrypted Synthetic Data for Privacy-Preserving LLMs](https://mlsanity.com/paper/2601.05635/) (2026)
- [FollowTable: A Benchmark for Instruction-Following Table Retrieval](https://mlsanity.com/paper/2605.00400/) (2026)
- [Reproducing Complex Set-Compositional Information Retrieval](https://mlsanity.com/paper/2605.03824/) (2026)
- [Task-Adaptive Embedding Refinement via Test-time LLM Guidance](https://mlsanity.com/paper/2605.12487/) (2026)
- [How Calibration Content Shapes Attention-Based Reranking](https://mlsanity.com/paper/2609.17764/) (2026)

## Abstract (arXiv)

Recent information retrieval (IR) models are pre-trained and instruction-tuned on massive datasets and tasks, enabling them to perform well on a wide range of tasks and potentially generalize to unseen tasks with instructions. However, existing IR benchmarks focus on a limited scope of tasks, making them insufficient for evaluating the latest IR models. In this paper, we propose MAIR (Massive Instructed Retrieval Benchmark), a heterogeneous IR benchmark that includes 126 distinct IR tasks across 6 domains, collected from existing datasets. We benchmark state-of-the-art instruction-tuned text embedding models and re-ranking models. Our experiments reveal that instruction-tuned models generally achieve superior performance compared to non-instruction-tuned models on MAIR. Additionally, our results suggest that current instruction-tuned text embedding models and re-ranking models still lack effectiveness in specific long-tail tasks. MAIR is publicly available at https://github.com/sunnweiwei/Mair.