mlsanityData to 7 Oct 2026

AIR-Bench: Automated Heterogeneous Information Retrieval Benchmark

arXiv 2412.13102 ↗First posted 17 Dec 2024cs.IR cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Evaluation plays a crucial role in the advancement of information retrieval (IR) models. However, current benchmarks, which are based on predefined domains and human-labeled data, face limitations in addressing evaluation needs for emerging domains both cost-effectively and efficiently. To address this challenge, we propose the Automated Heterogeneous Information Retrieval Benchmark (AIR-Bench). AIR-Bench is distinguished by three key features: 1) Automated. The testing data in AIR-Bench is automatically generated by large language models (LLMs) without human intervention. 2) Heterogeneous. The testing data in AIR-Bench is generated with respect to diverse tasks, domains and languages. 3) Dynamic. The domains and languages covered by AIR-Bench are constantly augmented to provide an increasingly comprehensive evaluation benchmark for community developers. We develop a reliable and robust data generation pipeline to automatically create diverse and high-quality evaluation datasets based on real-world corpora. Our findings demonstrate that the generated testing data in AIR-Bench aligns well with human-labeled testing data, making AIR-Bench a dependable benchmark for evaluating IR models. The resources in AIR-Bench are publicly available at https://github.com/AIR-Bench/AIR-Bench.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 35 of its 59 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

35 of 59 references.

References · 35

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  6. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  7. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  8. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  9. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  10. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  11. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  12. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
23 more
  1. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  2. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  3. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  4. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  5. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  6. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  7. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  8. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 138
  9. 2023Large language models can accurately predict searcher preferencescited by 129
  10. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  11. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 118
  12. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  13. 2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 103
  14. 2022SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrievalcited by 90
  15. 2023Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labelscited by 86
  16. 2024Making Text Embedders Few-Shot Learnerscited by 78
  17. 2023InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrievalcited by 73
  18. 2023MultiLegalPile: A 689GB Multilingual Legal Corpuscited by 59
  19. 2023LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Developmentcited by 54
  20. 2021A Statutory Article Retrieval Dataset in Frenchcited by 40
  21. 2023Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrievalcited by 24
  22. 2024Leveraging LLMs for Unsupervised Dense Retriever Rankingcited by 19
  23. 2024Little Giants: Synthesizing High-Quality Embedding Data at Scalecited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 5.

12 papers.

  1. 2025KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Modelcited by 35
  2. 2025Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMscited by 17
  3. 2025Differentiable Reward Optimization for LLM based TTS systemcited by 15
  4. 2026LMEB: Long-horizon Memory Embedding Benchmarkcited by 6
  5. 2025Balancing the Blend: An Experimental Analysis of Trade-offs in Hybrid Searchcited by 4
  6. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  7. 2025Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPscited by 2
  8. 2026PosIR: Position-Aware Heterogeneous Information Retrieval Benchmarkcited by 2
  9. 2025FinCPRG: A Bidirectional Generation Pipeline for Hierarchical Queries and Rich Relevance in Financial Chinese Passage Retrievalcited by 0
  10. 2026FLARE: Task-agnostic embedding model evaluation through a normalization processcited by 0
  11. 2026D-NOVA: In-Storage Retrieval Accelerator via Dual-Bound 3D NAND-Optimized Similarity Search with Vector Adaptationcited by 0
  12. 2026Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systemscited by 0