mlsanityData to 7 Oct 2026

BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval

arXiv 2407.12883 ↗First posted 16 Jul 2024cs.CL cs.AI cs.IRcited by 153

Not on the map. Why?

Cited by 53 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing retrieval benchmarks primarily consist of information-seeking queries (e.g., aggregated questions from search engines) where keyword or semantic-based retrieval is usually sufficient. However, many complex real-world queries require in-depth reasoning to identify relevant documents that go beyond surface form matching. For example, finding documentation for a coding question requires understanding the logic and syntax of the functions involved. To better benchmark retrieval on such challenging queries, we introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding. These queries are drawn from naturally occurring and carefully curated human data. Extensive evaluation reveals that even state-of-the-art retrieval models perform poorly on BRIGHT. The leading model on the MTEB leaderboard (Muennighoff et al., 2023) SFR-Embedding-Mistral (Meng et al., 2024), which achieves a score of 59.0 nDCG@10,1 produces a score of nDCG@10 of 18.3 on BRIGHT. We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points. Moreover, incorporating retrieved documents from the top-performing retriever boosts question-answering performance. We believe that BRIGHT paves the way for future research on retrieval systems in more realistic and challenging settings.

Key facts. Cited by 153 papers in the map (data through 7 Oct 2026). Builds on 57 of its 77 references that are in the map. First posted on arXiv 16 Jul 2024.

Builds on

57 of 77 references.

References · 57

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
38 more
  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  3. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  4. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  5. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  7. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  9. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  10. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  11. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  12. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  13. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  14. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  15. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  16. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  17. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  18. 2019ELI5: Long Form Question Answeringcited by 567
  19. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  20. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  21. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  22. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  23. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  24. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  25. 2023Improving Text Embeddings with Large Language Modelscited by 295
  26. 2020Overview of the TREC 2019 deep learning trackcited by 282
  27. 2024Generative Representational Instruction Tuningcited by 227
  28. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  29. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 195
  30. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  31. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 189
  32. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  33. 2022Task-aware Retrieval with Instructionscited by 107
  34. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 102
  35. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 81
  36. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  37. 2023MathPile: A Billion-Token-Scale Pretraining Corpus for Mathcited by 67
  38. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 67

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 79; 2026: 64.

153 papers.

Sort:

By area:

  1. 2025s1: Simple test-time scalingcited by 1,311
  2. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 183
  3. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  4. 2025On the Theoretical Limitations of Embedding-Based Retrievalcited by 98
  5. 2024Making Text Embedders Few-Shot Learnerscited by 78
  6. 2025ReasonIR: Training Retrievers for Reasoning Taskscited by 76
  7. 2025Rank1: Test-Time Compute for Reranking in Information Retrievalcited by 68
  8. 2025Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learningcited by 58
  9. 2024Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Modelscited by 57
  10. 2026OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoningcited by 37
  11. 2026τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledgecited by 35
  12. 2024JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Rerankingcited by 34
38 more
  1. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  2. 2025RAG-RL: Advancing Retrieval-Augmented Generation via RL and Curriculum Learningcited by 30
  3. 2025Characterizing Deep Research: A Benchmark and Formal Definitioncited by 29
  4. 2025Rank-K: Test-Time Reasoning for Listwise Rerankingcited by 28
  5. 2025RaDeR: Reasoning-aware Dense Retrieval Modelscited by 27
  6. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  7. 2025HopRAG: Multi-Hop Reasoning for Logic-Aware Retrieval-Augmented Generationcited by 24
  8. 2025REARANK: Reasoning Re-ranking Agent via Reinforcement Learningcited by 24
  9. 2024Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Taskscited by 22
  10. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 21
  11. 2024MRAG: A Modular Retrieval Framework for Time-Sensitive Question Answeringcited by 17
  12. 2025IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrievalcited by 16
  13. 2025EnrichIndex: Using LLMs to Enrich Retrieval Indices Offlinecited by 15
  14. 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
  15. 2025ReasonEmbed: Enhanced Text Embeddings for Reasoning-Intensive Document Retrievalcited by 15
  16. 2025Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Modelscited by 14
  17. 2025PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluationcited by 14
  18. 2025Training a Utility-based Retriever Through Shared Context Attribution for Retrieval-Augmented Language Modelscited by 13
  19. 2025LLM4Ranking: An Easy-to-use Framework of Utilizing Large Language Models for Document Rerankingcited by 12
  20. 2025IMPersona: Evaluating Individual Level LM Impersonationcited by 11
  21. 2025MIEB: Massive Image Embedding Benchmarkcited by 11
  22. 2025TongSearch-QR: Reinforced Query Reasoning for Retrievalcited by 11
  23. 2025CoRanking: Collaborative Ranking with Small and Large Ranking Agentscited by 10
  24. 2025MIR: Methodology Inspiration Retrieval for Scientific Research Problemscited by 10
  25. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  26. 2024ReasoningRank: Teaching Student Models to Rank through Reasoning-Based Knowledge Distillationcited by 9
  27. 2025Towards Better Instruction Following Retrieval Modelscited by 9
  28. 2025Controlled Retrieval-augmented Context Evaluation for Long-form RAGcited by 9
  29. 2025Large Reasoning Embedding Models: Towards Next-Generation Dense Retrieval Paradigmcited by 9
  30. 2024Drowning in Documents: Consequences of Scaling Reranker Inferencecited by 8
  31. 2025AcuRank: Uncertainty-Aware Adaptive Computation for Listwise Rerankingcited by 8
  32. 2025Modeling Uncertainty Trends for Timely Retrieval in Dynamic RAGcited by 8
  33. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  34. 2025ScIRGen: Synthesize Realistic and Large-Scale RAG Dataset for Scientific Researchcited by 7
  35. 2025Retro*: Optimizing LLMs for Reasoning-Intensive Document Retrievalcited by 7
  36. 2025MR2-Bench: Going Beyond Matching to Reasoning in Multimodal Retrievalcited by 7
  37. 2026SAGE: Benchmarking and Improving Retrieval for Deep Research Agentscited by 7
  38. 2026Revisiting Text Ranking in Deep Researchcited by 7

50 most cited of 153 All 153 (JSON)