mlsanityData to 14 Oct 2026

Entropy Heat-Mapping: Localizing GPT-Based OCR Errors with Sliding-Window Shannon Analysis

arXiv 2505.00746 ↗First posted 30 Apr 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Vision-language models such as OpenAI GPT-4o can transcribe mathematical documents directly from images, yet their token-level confidence signals are seldom used to pinpoint local recognition mistakes. We present an entropy-heat-mapping proof-of-concept that turns per-token Shannon entropy into a visual ''uncertainty landscape''. By scanning the entropy sequence with a fixed-length sliding window, we obtain hotspots that are likely to contain OCR errors such as missing symbols, mismatched braces, or garbled prose. Using a small, curated set of scanned research pages rendered at several resolutions, we compare the highlighted hotspots with the actual transcription errors produced by GPT-4o. Our analysis shows that the vast majority of true errors are indeed concentrated inside the high-entropy regions. This study demonstrates--in a minimally engineered setting--that sliding-window entropy can serve as a practical, lightweight aid for post-editing GPT-based OCR. All code and annotation guidelines are released to encourage replication and further research.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 37 references that are in the map. First posted on arXiv 30 Apr 2025.

Builds on

22 of 37 references.

References · 22

  1. 2024GPT-4o System Cardcited by 4,588
  2. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  3. 2021Probabilistic Embeddings for Cross-Modal Retrievalcited by 151
  4. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 87
  5. 2021Energy-Based Open-World Uncertainty Modeling for Confidence Calibrationcited by 46
  6. 2022MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Modelcited by 41
  7. 2023Improved Probabilistic Image-Text Representationscited by 40
  8. 2024ExACT: Language-guided Conceptual Reasoning and Uncertainty Estimation for Event-based Action Recognition and Morecited by 33
  9. 2024Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answeringcited by 32
  10. 2022BayesCap: Bayesian Identity Cap for Calibrated Uncertainty in Frozen Neural Networkscited by 23
  11. 2023Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrievalcited by 23
  12. 2024Just rephrase it! Uncertainty estimation in closed-source language models via multiple rephrased queriescited by 15
10 more
  1. 2024Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decodingcited by 15
  2. 2022Seq-UPS: Sequential Uncertainty-aware Pseudo-label Selection for Semi-Supervised Text Recognitioncited by 14
  3. 2025Early evidence of how LLMs outperform traditional systems on OCR/HTR tasks for historical recordscited by 10
  4. 2023ProbVLM: Probabilistic Adapter for Frozen Vision-Language Modelscited by 7
  5. 2024Notes on Applicability of GPT-4 to Document Understandingcited by 7
  6. 2024UBench: Benchmarking Uncertainty in Large Language Models with Multiple Choice Questionscited by 7
  7. 2022Uncertainty-Aware Image Captioningcited by 6
  8. 2024Post-OCR Text Correction for Bulgarian Historical Documentscited by 3
  9. 2024MathSpeech: Leveraging Small LMs for Accurate Conversion in Mathematical Speech-to-Formulacited by 3
  10. 2024Scrambled text: training Language Models to correct OCR errors using synthetic datacited by 2

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Language Model Guided Reinforcement Learning in Quantitative Tradingcited by 3
  2. 2025Layout-Aware OCR for Black Digital Archives with Unsupervised Evaluationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Consensus Entropy: Harnessing Multi-VLM Agreement for Self-Verifying and Self-Improving OCR
  2. 2026OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios
  3. 2026How Robust is OCR-Reasoning? Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbations
  4. 2026Visual Merit or Linguistic Crutch? A Close Look at DeepSeek-OCR
  5. 2026Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions
  6. 2025DeepSeek-OCR: Contexts Optical Compression
  7. 2026GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scripts
  8. 2026From Retrieval to Recognition:How Vision--Language Models Become OCR Specialists