# Entropy Heat-Mapping: Localizing GPT-Based OCR Errors with Sliding-Window Shannon Analysis

arXiv: https://arxiv.org/abs/2505.00746 · HTML: https://mlsanity.com/paper/2505.00746/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 22 of its 37 references that are in the map.
- First posted on arXiv 30 Apr 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 22 of 22)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Probabilistic Embeddings for Cross-Modal Retrieval](https://mlsanity.com/paper/2101.05068/) (2021)
- [Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?](https://mlsanity.com/paper/2405.16908/) (2024)
- [Energy-Based Open-World Uncertainty Modeling for Confidence Calibration](https://mlsanity.com/paper/2107.12628/) (2021)
- [MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model](https://mlsanity.com/paper/2210.05335/) (2022)
- [Improved Probabilistic Image-Text Representations](https://mlsanity.com/paper/2305.18171/) (2023)
- [ExACT: Language-guided Conceptual Reasoning and Uncertainty Estimation for Event-based Action Recognition and More](https://mlsanity.com/paper/2403.12534/) (2024)
- [Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answering](https://mlsanity.com/paper/2404.10193/) (2024)
- [BayesCap: Bayesian Identity Cap for Calibrated Uncertainty in Frozen Neural Networks](https://mlsanity.com/paper/2207.06873/) (2022)
- [Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval](https://mlsanity.com/paper/2309.17093/) (2023)
- [Just rephrase it! Uncertainty estimation in closed-source language models via multiple rephrased queries](https://mlsanity.com/paper/2405.13907/) (2024)
- [Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding](https://mlsanity.com/paper/2412.06474/) (2024)
- [Seq-UPS: Sequential Uncertainty-aware Pseudo-label Selection for Semi-Supervised Text Recognition](https://mlsanity.com/paper/2209.00641/) (2022)
- [Early evidence of how LLMs outperform traditional systems on OCR/HTR tasks for historical records](https://mlsanity.com/paper/2501.11623/) (2025)
- [ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models](https://mlsanity.com/paper/2307.00398/) (2023)
- [Notes on Applicability of GPT-4 to Document Understanding](https://mlsanity.com/paper/2405.18433/) (2024)
- [UBench: Benchmarking Uncertainty in Large Language Models with Multiple Choice Questions](https://mlsanity.com/paper/2406.12784/) (2024)
- [Uncertainty-Aware Image Captioning](https://mlsanity.com/paper/2211.16769/) (2022)
- [Post-OCR Text Correction for Bulgarian Historical Documents](https://mlsanity.com/paper/2409.00527/) (2024)
- [MathSpeech: Leveraging Small LMs for Accurate Conversion in Mathematical Speech-to-Formula](https://mlsanity.com/paper/2412.15655/) (2024)
- [Scrambled text: training Language Models to correct OCR errors using synthetic data](https://mlsanity.com/paper/2409.19735/) (2024)

## Built on by (most cited first, 2 of 2)

- [Language Model Guided Reinforcement Learning in Quantitative Trading](https://mlsanity.com/paper/2508.02366/) (2025)
- [Layout-Aware OCR for Black Digital Archives with Unsupervised Evaluation](https://mlsanity.com/paper/2509.13236/) (2025)

## Abstract (arXiv)

Vision-language models such as OpenAI GPT-4o can transcribe mathematical documents directly from images, yet their token-level confidence signals are seldom used to pinpoint local recognition mistakes. We present an entropy-heat-mapping proof-of-concept that turns per-token Shannon entropy into a visual ''uncertainty landscape''. By scanning the entropy sequence with a fixed-length sliding window, we obtain hotspots that are likely to contain OCR errors such as missing symbols, mismatched braces, or garbled prose. Using a small, curated set of scanned research pages rendered at several resolutions, we compare the highlighted hotspots with the actual transcription errors produced by GPT-4o. Our analysis shows that the vast majority of true errors are indeed concentrated inside the high-entropy regions. This study demonstrates--in a minimally engineered setting--that sliding-window entropy can serve as a practical, lightweight aid for post-editing GPT-based OCR. All code and annotation guidelines are released to encourage replication and further research.