# AMAQA: A Metadata-based QA Dataset for RAG Systems

arXiv: https://arxiv.org/abs/2505.13557 · HTML: https://mlsanity.com/paper/2505.13557/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 19 May 2025.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 28 of 28)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://mlsanity.com/paper/2309.07597/) (2023)
- [Benchmarking Retrieval-Augmented Generation for Medicine](https://mlsanity.com/paper/2402.13178/) (2024)
- [The Power of Noise: Redefining Retrieval for RAG Systems](https://mlsanity.com/paper/2401.14887/) (2024)
- [Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders](https://mlsanity.com/paper/2403.03952/) (2024)
- [RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs](https://mlsanity.com/paper/2407.02485/) (2024)
- [Re2G: Retrieve, Rerank, Generate](https://mlsanity.com/paper/2207.06300/) (2022)
- [VideoRAG: Retrieval-Augmented Generation over Video Corpus](https://mlsanity.com/paper/2501.05874/) (2025)
- [Large Language Models Offer an Alternative to the Traditional Approach of Topic Modelling](https://mlsanity.com/paper/2403.16248/) (2024)
- [Natural Language Inference Prompts for Zero-shot Emotion Classification in Text across Corpora](https://mlsanity.com/paper/2209.06701/) (2022)
- [The Pushshift Telegram Dataset](https://mlsanity.com/paper/2001.08438/) (2020)
- [Emotion Detection in Text: a Review](https://mlsanity.com/paper/1806.00674/) (2018)
- [Text2Topic: Multi-Label Text Classification System for Efficient Topic Detection in User Generated Content with Zero-Shot Capabilities](https://mlsanity.com/paper/2310.14817/) (2023)
- [TGDataset: Collecting and Exploring the Largest Telegram Channels Dataset](https://mlsanity.com/paper/2303.05345/) (2023)
- [You are your Metadata: Identification and Obfuscation of Social Media Users using Metadata Information](https://mlsanity.com/paper/1803.10133/) (2018)
- [Cybersecurity Threat Hunting and Vulnerability Analysis Using a Neo4j Graph Database of Open Source Intelligence](https://mlsanity.com/paper/2301.12013/) (2023)
- [Zero-shot prompt-based classification: topic labeling in times of foundation models in German Tweets](https://mlsanity.com/paper/2406.18239/) (2024)

## Built on by (most cited first, 2 of 2)

- [SRAG: RAG with Structured Data Improves Vector Retrieval](https://mlsanity.com/paper/2603.26670/) (2026)
- [ThreatCore: A Benchmark for Explicit and Implicit Threat Detection](https://mlsanity.com/paper/2605.10563/) (2026)

## Abstract (arXiv)

Retrieval-augmented generation (RAG) systems are widely used in question-answering (QA) tasks, but current benchmarks lack metadata integration, limiting their evaluation in scenarios requiring both textual data and external information. To address this, we present AMAQA, a new open-access QA dataset designed to evaluate tasks combining text and metadata. The integration of metadata is especially important in fields that require rapid analysis of large volumes of data, such as cybersecurity and intelligence, where timely access to relevant information is critical. AMAQA includes about 1.1 million English messages collected from 26 public Telegram groups, enriched with metadata such as timestamps and chat names. It also contains 20,000 hotel reviews with metadata. In addition, the dataset provides 2,600 high-quality QA pairs built across both domains, Telegram messages and hotel reviews, making AMAQA a valuable resource for advancing research on metadata-driven QA and RAG systems. Both Telegram messages and Hotel reviews are enriched with emotional tones or toxicity indicators. To the best of our knowledge, AMAQA is the first single-hop QA benchmark to incorporate metadata. We conduct extensive tests on the benchmark, setting a new reference point for future research. We show that leveraging metadata boosts accuracy from 0.5 to 0.86 for GPT-4o and from 0.27 to 0.76 for open source LLMs, highlighting the value of structured context. We conducted experiments on our benchmark to assess the performance of known techniques designed to enhance RAG, highlighting the importance of properly managing metadata throughout the entire RAG pipeline.