# Evaluating Robustness of LLMs in Question Answering on Multilingual Noisy OCR Data

arXiv: https://arxiv.org/abs/2502.16781 · HTML: https://mlsanity.com/paper/2502.16781/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 41 references that are in the map.
- First posted on arXiv 24 Feb 2025.

## Builds on (most cited first, 12 of 12)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [On the Impact of Various Types of Noise on Neural Machine Translation](https://mlsanity.com/paper/1805.12282/) (2018)
- [FQuAD: French Question Answering Dataset](https://mlsanity.com/paper/2002.06071/) (2020)
- [Named Entity Recognition and Classification on Historical Documents: A Survey](https://mlsanity.com/paper/2109.11406/) (2021)
- [GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrieval](https://mlsanity.com/paper/2104.12741/) (2021)
- [Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages](https://mlsanity.com/paper/2404.11553/) (2024)
- [Are Large Language Models Really Robust to Word-Level Perturbations?](https://mlsanity.com/paper/2309.11166/) (2023)
- [Ground Truth for training OCR engines on historical documents in German Fraktur and Early Modern Latin](https://mlsanity.com/paper/1809.05501/) (2018)
- [ChroniclingAmericaQA: A Large-scale Question Answering Dataset based on Historical American Newspaper Pages](https://mlsanity.com/paper/2403.17859/) (2024)

## Built on by (most cited first, 2 of 2)

- [Lost in OCR Translation? Vision-Based Approaches to Robust Document Retrieval](https://mlsanity.com/paper/2505.05666/) (2025)
- [Towards Reliable and Interpretable Document Question Answering via VLMs](https://mlsanity.com/paper/2509.10129/) (2025)

## Abstract (arXiv)

Optical Character Recognition (OCR) plays a crucial role in digitizing historical and multilingual documents, yet OCR errors - imperfect extraction of text, including character insertion, deletion, and substitution can significantly impact downstream tasks like question-answering (QA). In this work, we conduct a comprehensive analysis of how OCR-induced noise affects the performance of Multilingual QA Systems. To support this analysis, we introduce a multilingual QA dataset MultiOCR-QA, comprising 50K question-answer pairs across three languages, English, French, and German. The dataset is curated from OCR-ed historical documents, which include different levels and types of OCR noise. We then evaluate how different state-of-the-art Large Language Models (LLMs) perform under different error conditions, focusing on three major OCR error types. Our findings show that QA systems are highly prone to OCR-induced errors and perform poorly on noisy OCR text. By comparing model performance on clean versus noisy texts, we provide insights into the limitations of current approaches and emphasize the need for more noise-resilient QA systems in historical digitization contexts.