mlsanityData to 7 Oct 2026

OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation

arXiv 2412.02592 ↗First posted 3 Dec 2024cs.CVcited by 24

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Retrieval-augmented Generation (RAG) enhances Large Language Models (LLMs) by integrating external knowledge to reduce hallucinations and incorporate up-to-date information without retraining. As an essential part of RAG, external knowledge bases are commonly built by extracting structured data from unstructured PDF documents using Optical Character Recognition (OCR). However, given the imperfect prediction of OCR and the inherent non-uniform representation of structured data, knowledge bases inevitably contain various OCR noises. In this paper, we introduce OHRBench, the first benchmark for understanding the cascading impact of OCR on RAG systems. OHRBench includes 8,561 carefully selected unstructured document images from seven real-world RAG application domains, along with 8,498 Q&A pairs derived from multimodal elements in documents, challenging existing OCR solutions used for RAG. To better understand OCR's impact on RAG systems, we identify two primary types of OCR noise: Semantic Noise and Formatting Noise and apply perturbation to generate a set of structured data with varying degrees of each OCR noise. Using OHRBench, we first conduct a comprehensive evaluation of current OCR solutions and reveal that none is competent for constructing high-quality knowledge bases for RAG systems. We then systematically evaluate the impact of these two noise types and demonstrate the trend relationship between the degree of OCR noise and RAG performance. Our OHRBench, including PDF documents, Q&As, and the ground truth structured data are released at: https://github.com/opendatalab/OHR-Bench

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 37 of its 59 references that are in the map. First posted on arXiv 3 Dec 2024.

Builds on

37 of 59 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding7 Nov
  2. 2024Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction28 Oct
  3. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents14 Oct

Other references · 34

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  7. 2024Qwen2 Technical Reportcited by 2,076
  8. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  10. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  11. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  12. 2023In-Context Retrieval-Augmented Language Modelscited by 694
22 more
  1. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  2. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
  3. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  4. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 255
  5. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 253
  6. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 240
  7. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  8. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  9. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  10. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 174
  11. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 132
  12. 2024Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Trainingcited by 88
  13. 2024NV-Retriever: Improving text embedding models with effective hard-negative miningcited by 83
  14. 2024Typos that Broke the RAG's Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbationscited by 64
  15. 2024Focus Anywhere for Fine-grained Multi-page Document Understandingcited by 62
  16. 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 53
  17. 2024LLM as Dataset Analyst: Subpopulation Structure Discovery with Large Language Modelcited by 43
  18. 2024UniMERNet: A Universal Network for Real-World Mathematical Expression Recognitioncited by 41
  19. 2024UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysiscited by 40
  20. 2024DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systemscited by 37
  21. 2024Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generationcited by 19
  22. 2024READoc: A Unified Benchmark for Realistic Document Structured Extractioncited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 12.

24 papers.

Sort:

By area:

  1. 2025MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understandingcited by 66
  2. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  3. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  4. 2026MinerU2.5-Pro: Pushing the Limits of Data-Centric Document Parsing at Scalecited by 26
  5. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  6. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  7. 2025AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMscited by 19
  8. 2025HiChunk: Evaluating and Enhancing Retrieval-Augmented Generation with Hierarchical Chunkingcited by 9
  9. 2025DOCR-Inspector: Fine-Grained and Automated Evaluation of Document Parsing with VLMcited by 6
  10. 2026CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embeddingcited by 6
  11. 2025IPCV: Information-Preserving Compression for MLLM Visual Encoderscited by 5
  12. 2026Confundo: Learning to Generate Robust Poison for Practical RAG Systemscited by 5
12 more
  1. 2025TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognitioncited by 4
  2. 2025Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluationcited by 3
  3. 2026Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representationscited by 3
  4. 2025OmniDocLayout: Towards Diverse Document Layout Generation via Coarse-to-Fine LLM Learningcited by 2
  5. 2026Efficient Document Parsing via Parallel Token Predictioncited by 2
  6. 2026LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinementcited by 1
  7. 2026Benchmarking Complex Multimodal Document Processing Pipelines: A Unified Evaluation Framework for Enterprise AIcited by 0
  8. 2026Cascading Hallucination in Agentic RAG: The CHARM Framework for Detection and Mitigationcited by 0
  9. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0
  10. 2026CMDR: Contextual Multimodal Document Retrievalcited by 0
  11. 2026VLD-RAG: Agentic Vision-Language Retrieval-Augmented Generation for Long, Visually-Rich Multi-Page Documentscited by 0
  12. 2026Evaluating Modern RAG: Textual, Multimodal, Dense, and Late Interaction Pipelinescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Lost in OCR Translation? Vision-Based Approaches to Robust Document Retrieval
  2. 2026When Good OCR Is Not Enough: Benchmarking OCR Robustness for Retrieval-Augmented Generation
  3. 2025Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?
  4. 2026Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG
  5. 2025SCAN: Semantic Document Layout Analysis for Textual and Visual Retrieval-Augmented Generation
  6. 2025RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance
  7. 2025Re-ranking the Context for Multimodal Retrieval Augmented Generation
  8. 2025Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Survey