mlsanityData to 7 Oct 2026

BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives

arXiv 2402.14151 ↗First posted 21 Feb 2024cs.IR cs.AI cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

We present the Benchmark of Information Retrieval (IR) tasks with Complex Objectives (BIRCO). BIRCO evaluates the ability of IR systems to retrieve documents given multi-faceted user objectives. The benchmark's complexity and compact size make it suitable for evaluating large language model (LLM)-based information retrieval systems. We present a modular framework for investigating factors that may influence LLM performance on retrieval tasks, and identify a simple baseline model which matches or outperforms existing approaches and more complex alternatives. No approach achieves satisfactory performance on all benchmark tasks, suggesting that stronger models and new retrieval protocols are necessary to address complex user needs.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 36 of its 58 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

36 of 58 references.

References · 36

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  11. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  12. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
24 more
  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  3. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  4. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  5. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  6. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  7. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  8. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  9. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  10. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  11. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  12. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  13. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 195
  14. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  15. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
  16. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  17. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 124
  18. 2022Task-aware Retrieval with Instructionscited by 107
  19. 2023Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labelscited by 86
  20. 2023Decomposing Complex Queries for Tip-of-the-tongue Retrievalcited by 24
  21. 2023RankCSE: Unsupervised Sentence Representations Learning via Learning to Rankcited by 23
  22. 2022RELIC: Retrieving Evidence for Literary Claimscited by 18
  23. 2023DORIS-MAE: Scientific Document Retrieval using Multi-level Aspect-based Queriescited by 11
  24. 2023I3: Intent-Introspective Retrieval Conditioned on Instructionscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 5.

15 papers.

Sort:

By area:

  1. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  2. 2025ReasonIR: Training Retrievers for Reasoning Taskscited by 76
  3. 2025A Reasoning-Focused Legal Retrieval Benchmarkcited by 37
  4. 2026τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledgecited by 35
  5. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  6. 2024Drowning in Documents: Consequences of Scaling Reranker Inferencecited by 8
  7. 2025Benchmarking Information Retrieval Models on Complex Retrieval Taskscited by 6
  8. 2025Balancing the Blend: An Experimental Analysis of Trade-offs in Hybrid Searchcited by 4
  9. 2024Prompts as Auto-Optimized Training Hyperparameters: Training Best-in-Class IR Models from Scratch with 10 Gold Labelscited by 3
  10. 2024SparseCL: Sparse Contrastive Learning for Contradiction Retrievalcited by 2
  11. 2026Revisiting RAG Retrievers: An Information Theoretic Benchmarkcited by 2
  12. 2026CT Open: An Open-Access, Uncontaminated, Live Platform for the Open Challenge of Clinical Trial Outcome Predictioncited by 2
3 more
  1. 2025ImpliRet: Benchmarking the Implicit Fact Retrieval Challengecited by 1
  2. 2026HAKARI-Bench: A Lightweight Benchmark for Comparing Retrieval Architectures and Efficiency Settings under Unified Conditionscited by 0
  3. 2026Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answeringcited by 0