# UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis

arXiv: https://arxiv.org/abs/2406.15187 · HTML: https://mlsanity.com/paper/2406.15187/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 50 of its 64 references that are in the map.
- First posted on arXiv 21 Jun 2024.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 30 of 50)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [Active Retrieval Augmented Generation](https://mlsanity.com/paper/2305.06983/) (2023)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [InternLM2 Technical Report](https://mlsanity.com/paper/2403.17297/) (2024)
- [Benchmarking Large Language Models in Retrieval-Augmented Generation](https://mlsanity.com/paper/2309.01431/) (2023)
- [HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data](https://mlsanity.com/paper/2004.07347/) (2020)
- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)

## Built on by (most cited first, 30 of 40)

- [Retrieval-Augmented Generation for Natural Language Processing: A Survey](https://mlsanity.com/paper/2407.13193/) (2024)
- [MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understanding](https://mlsanity.com/paper/2503.13964/) (2025)
- [Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs](https://mlsanity.com/paper/2507.09477/) (2025)
- [VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.10704/) (2024)
- [SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinement](https://mlsanity.com/paper/2506.14035/) (2025)
- [OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation](https://mlsanity.com/paper/2412.02592/) (2024)
- [Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding](https://mlsanity.com/paper/2511.22850/) (2025)
- [MoLoRAG: Bootstrapping Document Understanding via Multi-modal Logic-aware Retrieval](https://mlsanity.com/paper/2509.07666/) (2025)
- [Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning](https://mlsanity.com/paper/2506.08354/) (2025)
- [EnronQA: Towards Personalized RAG over Private Documents](https://mlsanity.com/paper/2505.00263/) (2025)
- [Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding](https://mlsanity.com/paper/2510.15253/) (2025)
- [Structured Attention Matters to Multimodal LLMs in Document Understanding](https://mlsanity.com/paper/2506.21600/) (2025)
- [Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasets](https://mlsanity.com/paper/2504.20119/) (2025)
- [LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understanding](https://mlsanity.com/paper/2510.07233/) (2025)
- [Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?](https://mlsanity.com/paper/2508.03644/) (2025)
- [Document Intelligence in the Era of Large Language Models: A Survey](https://mlsanity.com/paper/2510.13366/) (2025)
- [CTIConnect: A Benchmark for Retrieval-Augmented LLMs over Heterogeneous Cyber Threat Intelligence](https://mlsanity.com/paper/2510.11974/) (2025)
- [A Scalable Data-Driven Framework for Systematic Analysis of SEC 10-K Filings Using Large Language Models](https://mlsanity.com/paper/2409.17581/) (2024)
- [T2-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.12071/) (2025)
- [ScaleDoc: Scaling LLM-based Predicates over Large Document Collections](https://mlsanity.com/paper/2509.12610/) (2025)
- [M3DR: Towards Universal Multilingual Multimodal Document Retrieval](https://mlsanity.com/paper/2512.03514/) (2025)
- [OkraLong: A Flexible Retrieval-Augmented Framework for Long-Text Query Processing](https://mlsanity.com/paper/2503.02603/) (2025)
- [Document Retrieval Augmented Fine-Tuning (DRAFT) for safety-critical software assessments](https://mlsanity.com/paper/2505.01307/) (2025)
- [Multimodal RAG for Unstructured Data:Leveraging Modality-Aware Knowledge Graphs with Hybrid Retrieval](https://mlsanity.com/paper/2510.14592/) (2025)
- [ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documents](https://mlsanity.com/paper/2510.25668/) (2025)
- [pdfQA: Diverse, Challenging, and Realistic Question Answering over PDFs](https://mlsanity.com/paper/2601.02285/) (2026)
- [Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QA](https://mlsanity.com/paper/2604.22239/) (2026)
- [Hierarchical Evidence-Driven Reasoning for Long Document Understanding](https://mlsanity.com/paper/2607.04625/) (2026)
- [CReSt: A Comprehensive Benchmark for Retrieval-Augmented Generation with Complex Reasoning over Structured Documents](https://mlsanity.com/paper/2505.17503/) (2025)
- [CMRAG: Co-modality-based visual document retrieval and question answering](https://mlsanity.com/paper/2509.02123/) (2025)

## Abstract (arXiv)

The use of Retrieval-Augmented Generation (RAG) has improved Large Language Models (LLMs) in collaborating with external data, yet significant challenges exist in real-world scenarios. In areas such as academic literature and finance question answering, data are often found in raw text and tables in HTML or PDF formats, which can be lengthy and highly unstructured. In this paper, we introduce a benchmark suite, namely Unstructured Document Analysis (UDA), that involves 2,965 real-world documents and 29,590 expert-annotated Q&A pairs. We revisit popular LLM- and RAG-based solutions for document analysis and evaluate the design choices and answer qualities across multiple document domains and diverse query types. Our evaluation yields interesting findings and highlights the importance of data parsing and retrieval. We hope our benchmark can shed light and better serve real-world document analysis applications. The benchmark suite and code can be found at https://github.com/qinchuanhui/UDA-Benchmark.