mlsanityData to 14 Oct 2026

UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis

arXiv 2406.15187 ↗First posted 21 Jun 2024cs.AI cs.IRcited by 40

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

The use of Retrieval-Augmented Generation (RAG) has improved Large Language Models (LLMs) in collaborating with external data, yet significant challenges exist in real-world scenarios. In areas such as academic literature and finance question answering, data are often found in raw text and tables in HTML or PDF formats, which can be lengthy and highly unstructured. In this paper, we introduce a benchmark suite, namely Unstructured Document Analysis (UDA), that involves 2,965 real-world documents and 29,590 expert-annotated Q&A pairs. We revisit popular LLM- and RAG-based solutions for document analysis and evaluate the design choices and answer qualities across multiple document domains and diverse query types. Our evaluation yields interesting findings and highlights the importance of data parsing and retrieval. We hope our benchmark can shed light and better serve real-world document analysis applications. The benchmark suite and code can be found at https://github.com/qinchuanhui/UDA-Benchmark.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 50 of its 64 references that are in the map. First posted on arXiv 21 Jun 2024.

Builds on

50 of 64 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models30 Jan
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec

Other references · 48

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2023Mistral 7Bcited by 3,164
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  12. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
36 more
  1. 2024Mixtral of Expertscited by 1,823
  2. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  3. 2021YOLOX: Exceeding YOLO Series in 2021cited by 1,071
  4. 2024Yi: Open Foundation Models by 01.AIcited by 719
  5. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  6. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  7. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 562
  8. 2023Active Retrieval Augmented Generationcited by 516
  9. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
  10. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  11. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 476
  12. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  13. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 408
  14. 2024InternLM2 Technical Reportcited by 397
  15. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  16. 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 264
  17. 2023Challenges and Applications of Large Language Modelscited by 254
  18. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
  19. 2021FeTaQA: Free-form Table Question Answeringcited by 175
  20. 2020GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsingcited by 171
  21. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  22. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 151
  23. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 147
  24. 2021PubTables-1M: Towards comprehensive table extraction from unstructured documentscited by 126
  25. 2023Retrieval meets Long Context Large Language Modelscited by 115
  26. 2023Learning to Filter Context for Retrieval-Augmented Generationcited by 83
  27. 2022Towards Complex Document Understanding By Discrete Reasoningcited by 82
  28. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 79
  29. 2023Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-Incited by 71
  30. 2023Calibrating LLM-Based Evaluatorcited by 53
  31. 2023PDFVQA: A New Dataset for Real-World VQA on PDF Documentscited by 28
  32. 2024OpenTab: Advancing Large Language Models as Open-domain Table Reasonerscited by 25
  33. 2023Structure-Aware Language Model Pretraining Improves Dense Retrieval on Structured Datacited by 23
  34. 2024TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Datacited by 19
  35. 2023Towards reducing hallucination in extracting information from financial reports using Large Language Modelscited by 11
  36. 2023Dual-Feedback Knowledge Retrieval for Task-Oriented Dialogue Systemscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 23; 2026: 13.

40 papers.

Sort:

By area:

  1. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 71
  2. 2025MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understandingcited by 67
  3. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 42
  4. 2024VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generationcited by 41
  5. 2025SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinementcited by 28
  6. 2024OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generationcited by 26
  7. 2025Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understandingcited by 18
  8. 2025MoLoRAG: Bootstrapping Document Understanding via Multi-modal Logic-aware Retrievalcited by 17
  9. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  10. 2025EnronQA: Towards Personalized RAG over Private Documentscited by 9
  11. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  12. 2025Structured Attention Matters to Multimodal LLMs in Document Understandingcited by 8
28 more
  1. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  2. 2025LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understandingcited by 7
  3. 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?cited by 5
  4. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  5. 2025CTIConnect: A Benchmark for Retrieval-Augmented LLMs over Heterogeneous Cyber Threat Intelligencecited by 4
  6. 2024A Scalable Data-Driven Framework for Systematic Analysis of SEC 10-K Filings Using Large Language Modelscited by 3
  7. 2025T2-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generationcited by 3
  8. 2025ScaleDoc: Scaling LLM-based Predicates over Large Document Collectionscited by 3
  9. 2025M3DR: Towards Universal Multilingual Multimodal Document Retrievalcited by 3
  10. 2025OkraLong: A Flexible Retrieval-Augmented Framework for Long-Text Query Processingcited by 2
  11. 2025Document Retrieval Augmented Fine-Tuning (DRAFT) for safety-critical software assessmentscited by 2
  12. 2025Multimodal RAG for Unstructured Data:Leveraging Modality-Aware Knowledge Graphs with Hybrid Retrievalcited by 2
  13. 2025ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documentscited by 2
  14. 2026pdfQA: Diverse, Challenging, and Realistic Question Answering over PDFscited by 2
  15. 2026Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QAcited by 2
  16. 2026Hierarchical Evidence-Driven Reasoning for Long Document Understandingcited by 2
  17. 2025CReSt: A Comprehensive Benchmark for Retrieval-Augmented Generation with Complex Reasoning over Structured Documentscited by 1
  18. 2025CMRAG: Co-modality-based visual document retrieval and question answeringcited by 1
  19. 2026ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Managementcited by 1
  20. 2026G2-Reader: Dual Evolving Graphs for Multimodal Document QAcited by 1
  21. 2026FLOWREADER: Min-Cost Flow Optimization for Multi-Modal Long Document Q&Acited by 1
  22. 2026MAB-DQA: Addressing Query Aspect Importance in Document Question Answering with Multi-Armed Banditscited by 0
  23. 2026You Snooze, You Lose: Automatic Safety Alignment Restoration through Neural Weight Translationcited by 0
  24. 2026LFRAG: Layout-oriented Fine-grained Retrieval-Augmented Generation on Multimodal Document Understandingcited by 0
  25. 2026DeCoRAG: Cognitive Decoupling and Semantic-Aware Cropping for Complex Document Understandingcited by 0
  26. 2026DocMemo: Dynamic Evidence Discovery via Probabilistic Memory-Guided Retrieval for Multi-Modal Document Understandingcited by 0
  27. 2026D2-ScaleAgent: Dual-Dimensional Scaling for Long Document Understandingcited by 0
  28. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphs
  2. 2024Retrieval-Augmented Generation for Domain-Specific Question Answering: A Case Study on Pittsburgh and CMU
  3. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  4. 2025REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark
  5. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
  6. 2025Towards Global Retrieval Augmented Generation: A Benchmark for Corpus-Level Reasoning
  7. 2026ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios
  8. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions