mlsanityData to 14 Oct 2026

AMAQA: A Metadata-based QA Dataset for RAG Systems

arXiv 2505.13557 ↗First posted 19 May 2025cs.IR cs.AIcited by 2

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Retrieval-augmented generation (RAG) systems are widely used in question-answering (QA) tasks, but current benchmarks lack metadata integration, limiting their evaluation in scenarios requiring both textual data and external information. To address this, we present AMAQA, a new open-access QA dataset designed to evaluate tasks combining text and metadata. The integration of metadata is especially important in fields that require rapid analysis of large volumes of data, such as cybersecurity and intelligence, where timely access to relevant information is critical. AMAQA includes about 1.1 million English messages collected from 26 public Telegram groups, enriched with metadata such as timestamps and chat names. It also contains 20,000 hotel reviews with metadata. In addition, the dataset provides 2,600 high-quality QA pairs built across both domains, Telegram messages and hotel reviews, making AMAQA a valuable resource for advancing research on metadata-driven QA and RAG systems. Both Telegram messages and Hotel reviews are enriched with emotional tones or toxicity indicators. To the best of our knowledge, AMAQA is the first single-hop QA benchmark to incorporate metadata. We conduct extensive tests on the benchmark, setting a new reference point for future research. We show that leveraging metadata boosts accuracy from 0.5 to 0.86 for GPT-4o and from 0.27 to 0.76 for open source LLMs, highlighting the value of structured context. We conducted experiments on our benchmark to assess the performance of known techniques designed to enhance RAG, highlighting the importance of properly managing metadata throughout the entire RAG pipeline.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 28 of its 44 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

28 of 44 references.

References · 28

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  9. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  10. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  11. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
  12. 2018CoQA: A Conversational Question Answering Challengecited by 683
16 more
  1. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  2. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 331
  3. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 281
  4. 2024Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoderscited by 224
  5. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 202
  6. 2022Re2G: Retrieve, Rerank, Generatecited by 102
  7. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpuscited by 57
  8. 2024Large Language Models Offer an Alternative to the Traditional Approach of Topic Modellingcited by 35
  9. 2022Natural Language Inference Prompts for Zero-shot Emotion Classification in Text across Corporacited by 13
  10. 2020The Pushshift Telegram Datasetcited by 12
  11. 2018Emotion Detection in Text: a Reviewcited by 10
  12. 2023Text2Topic: Multi-Label Text Classification System for Efficient Topic Detection in User Generated Content with Zero-Shot Capabilitiescited by 6
  13. 2023TGDataset: Collecting and Exploring the Largest Telegram Channels Datasetcited by 4
  14. 2018You are your Metadata: Identification and Obfuscation of Social Media Users using Metadata Informationcited by 3
  15. 2023Cybersecurity Threat Hunting and Vulnerability Analysis Using a Neo4j Graph Database of Open Source Intelligencecited by 2
  16. 2024Zero-shot prompt-based classification: topic labeling in times of foundation models in German Tweetscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SRAG: RAG with Structured Data Improves Vector Retrievalcited by 0
  2. 2026ThreatCore: A Benchmark for Explicit and Implicit Threat Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025WixQA: A Multi-Dataset Benchmark for Enterprise Retrieval-Augmented Generation
  2. 2025Evaluation of retrieval-based QA on QUEST-LOFT
  3. 2025THELMA: Task Based Holistic Evaluation of Large Language Model Applications-RAG Question Answering
  4. 2026Augmenting Question Answering with A Hybrid RAG Approach
  5. 2026LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake
  6. 2024RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering
  7. 2025T2-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation
  8. 2024ChatQA: Surpassing GPT-4 on Conversational QA and RAG