mlsanityData to 14 Oct 2026

IRSC: A Zero-shot Evaluation Benchmark for Information Retrieval through Semantic Comprehension in Retrieval-Augmented Generation Scenarios

arXiv 2409.15763 ↗First posted 24 Sep 2024cs.IR cs.AIcited by 3

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

In Retrieval-Augmented Generation (RAG) tasks using Large Language Models (LLMs), the quality of retrieved information is critical to the final output. This paper introduces the IRSC benchmark for evaluating the performance of embedding models in multilingual RAG tasks. The benchmark encompasses five retrieval tasks: query retrieval, title retrieval, part-of-paragraph retrieval, keyword retrieval, and summary retrieval. Our research addresses the current lack of comprehensive testing and effective comparison methods for embedding models in RAG scenarios. We introduced new metrics: the Similarity of Semantic Comprehension Index (SSCI) and the Retrieval Capability Contest Index (RCCI), and evaluated models such as Snowflake-Arctic, BGE, GTE, and M3E. Our contributions include: 1) the IRSC benchmark, 2) the SSCI and RCCI metrics, and 3) insights into the cross-lingual limitations of embedding models. The IRSC benchmark aims to enhance the understanding and development of accurate retrieval systems in RAG tasks. All code and datasets are available at: https://github.com/Jasaxion/IRSC_Benchmark

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 18 of its 23 references that are in the map. First posted on arXiv 24 Sep 2024.

Builds on

18 of 23 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Evaluation of Retrieval-Augmented Generation: A Survey13 May

Other references · 17

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  3. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  5. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  6. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  7. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  8. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  9. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  10. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  11. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  12. 2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 153
5 more
  1. 2019On the Use of ArXiv as a Datasetcited by 103
  2. 2024Is Cosine-Similarity of Embeddings Really About Similarity?cited by 89
  3. 2024Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Modelscited by 61
  4. 2022Problems with Cosine as a Measure of Embedding Similarity for High Frequency Wordscited by 49
  5. 2022Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languagescited by 47

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2026τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledgecited by 39
  2. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  3. 2025LexSemBridge: Fine-Grained Dense Representation Enhancement through Token-Aware Embedding Augmentationcited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2025ELITE: Embedding-Less retrieval with Iterative Text Exploration
  2. 2025Enhancing Retrieval-Augmented Generation: A Study of Best Practices
  3. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  4. 2024Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval Augmented Generation Systems
  5. 2026Revisiting RAG Retrievers: An Information Theoretic Benchmark
  6. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  7. 2025MIRAGE: A Metric-Intensive Benchmark for Retrieval-Augmented Generation Evaluation
  8. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Models