# Lost in OCR Translation? Vision-Based Approaches to Robust Document Retrieval

arXiv: https://arxiv.org/abs/2505.05666 · HTML: https://mlsanity.com/paper/2505.05666/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 11 of its 20 references that are in the map.
- First posted on arXiv 8 May 2025.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 11 of 11)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021)
- [Unifying Vision, Text, and Layout for Universal Document Processing](https://mlsanity.com/paper/2212.02623/) (2022)
- [Enhancing Code Translation in Language Models with Few-Shot Learning via Retrieval-Augmented Generation](https://mlsanity.com/paper/2407.19619/) (2024)
- [Decompose and Compare Consistency: Measuring VLMs' Answer Reliability via Task-Decomposition Consistency Comparison](https://mlsanity.com/paper/2407.07840/) (2024)
- [HEAL: Hierarchical Embedding Alignment Loss for Improved Retrieval and Representation Learning](https://mlsanity.com/paper/2412.04661/) (2024)
- [Evaluating Robustness of LLMs in Question Answering on Multilingual Noisy OCR Data](https://mlsanity.com/paper/2502.16781/) (2025)

## Built on by (most cited first, 11 of 11)

- [DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruning](https://mlsanity.com/paper/2509.23883/) (2025)
- [Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval](https://mlsanity.com/paper/2602.19961/) (2026)
- [Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrieval](https://mlsanity.com/paper/2604.10167/) (2026)
- [CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embedding](https://mlsanity.com/paper/2601.21262/) (2026)
- [Structure-Aware Text Recognition for Ancient Greek Critical Editions](https://mlsanity.com/paper/2603.02803/) (2026)
- [Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representations](https://mlsanity.com/paper/2603.01666/) (2026)
- [BLUEPRINT Rebuilding a Legacy: Multimodal Retrieval for Complex Engineering Drawings and Documents](https://mlsanity.com/paper/2602.13345/) (2026)
- [Index Light, Reason Deep: Deferred Visual Ingestion for Visual-Dense Document Question Answering](https://mlsanity.com/paper/2602.14162/) (2026)
- [RobustVisRAG: Causality-Aware Vision-Based Retrieval-Augmented Generation under Visual Degradations](https://mlsanity.com/paper/2602.22013/) (2026)
- [LazySloth: Bounded LLM-based Lazy Tree Search for Fast Long Video Comprehension](https://mlsanity.com/paper/2609.37426/) (2026)
- [Cascaded Retrieval with Cross-Modal Query Refinement for Efficient Visual Document Search](https://mlsanity.com/paper/2610.20367/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) has become a popular technique for enhancing the reliability and utility of Large Language Models (LLMs) by grounding responses in external documents. Traditional RAG systems rely on Optical Character Recognition (OCR) to first process scanned documents into text. However, even state-of-the-art OCRs can introduce errors, especially in degraded or complex documents. Recent vision-language approaches, such as ColPali, propose direct visual embedding of documents, eliminating the need for OCR. This study presents a systematic comparison between a vision-based RAG system (ColPali) and more traditional OCR-based pipelines utilizing Llama 3.2 (90B) and Nougat OCR across varying document qualities. Beyond conventional retrieval accuracy metrics, we introduce a semantic answer evaluation benchmark to assess end-to-end question-answering performance. Our findings indicate that while vision-based RAG performs well on documents it has been fine-tuned on, OCR-based RAG is better able to generalize to unseen documents of varying quality. We highlight the key trade-offs between computational efficiency and semantic accuracy, offering practical guidance for RAG practitioners in selecting between OCR-dependent and vision-based document retrieval systems in production environments.