mlsanityData to 14 Oct 2026

Consensus Entropy: Harnessing Multi-VLM Agreement for Self-Verifying and Self-Improving OCR

arXiv 2504.11101 ↗First posted 15 Apr 2025cs.CV cs.AI cs.MMcited by 12

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

also in 1 more
Abstract (arXiv)

Optical Character Recognition (OCR) is fundamental to Vision-Language Models (VLMs) and high-quality data generation for LLM training. Yet, despite progress in average OCR accuracy, state-of-the-art VLMs still struggle with detecting sample-level errors and lack effective unsupervised quality control. We introduce Consensus Entropy (CE), a training-free, model-agnostic metric that estimates output reliability by measuring inter-model agreement entropy. The core insight is that correct predictions converge in output space, while errors diverge. Based on CE, we develop CE-OCR, a lightweight multi-model framework that verifies outputs by ensemble agreement, selects the best outputs, and further improves efficiency through adaptive routing. Experiments demonstrate that CE is robust for quality verification, improving F1 scores by 42.1% over VLM-as-Judge. CE-OCR achieves consistent OCR gains, outperforming self-consistency and single-model baselines at the same cost. Notably, CE requires no training or supervision, enabling plug-and-play integration. Code: https://github.com/Aslan-yulong/consensus-entropy.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 63 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

45 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models25 Feb
  2. 2024CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy3 Dec
  3. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models13 May

Other references · 42

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  9. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  10. 2024A Survey on LLM-as-a-Judgecited by 1,423
  11. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  12. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
30 more
  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  2. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  3. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  4. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  5. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
  6. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  7. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  8. 2019PubLayNet: largest dataset ever for document layout analysiscited by 351
  9. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 267
  10. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 245
  11. 2024More Agents Is All You Needcited by 180
  12. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 180
  13. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
  14. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 95
  15. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  16. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  17. 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimationcited by 73
  18. 2024Focus Anywhere for Fine-grained Multi-page Document Understandingcited by 62
  19. 2024Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluationcited by 57
  20. 2023Calibrating LLM-Based Evaluatorcited by 53
  21. 2023Getting MoRE out of Mixture of Language Model Reasoning Expertscited by 44
  22. 2023Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generatorscited by 43
  23. 2024LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversitycited by 42
  24. 2024Smoothie: Label Free Language Model Routingcited by 31
  25. 2024OpenDataLab: Empowering General Artificial Intelligence with Open Datasetscited by 29
  26. 2021VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groupscited by 27
  27. 2024Uncertainty Quantification via Stable Distribution Propagationcited by 10
  28. 2025Enhancing Uncertainty Modeling with Semantic Graph for Hallucination Detectioncited by 8
  29. 2025Collective Reasoning Among LLMs: A Framework for Answer Validation Without Ground Truthcited by 7
  30. 2024TextCenGen: Attention-Guided Text-Centric Background Adaptation for Text-to-Image Generationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 7.

12 papers.

  1. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  2. 2026SEAD: Self-Evolving Agent for Multi-Turn Service Dialoguecited by 7
  3. 2025NCV: A Node-Wise Consistency Verification Approach for Low-Cost Structured Error Localization in LLM Reasoningcited by 1
  4. 2025High-Entropy Tokens as Multimodal Failure Points in Vision-Language Modelscited by 1
  5. 2025Improving MLLM Historical Record Extraction with Test-Time Imagecited by 0
  6. 2025OCR-Quality: A Human-Annotated Dataset for OCR Quality Assessmentcited by 0
  7. 2026SCoOP: Semantic Consistent Opinion Pooling for Uncertainty Quantification in Multiple Vision-Language Model Systemscited by 0
  8. 2026From Diagnosis to Correction: Benchmarking and Improving Real-World Table Parsingcited by 0
  9. 2026Uncertainty-Aware Decision Making in Multimodal Large Language Modelscited by 0
  10. 2026CUSP: Decomposable Collective Uncertainty for Multi-Agent Multimodal Reasoningcited by 0
  11. 2026PrismAlign: Prior-Steered Multi-View VLM Alignment for Hallucination-Robust Table OCRcited by 0
  12. 2026Collapse, Not Complexity: Failure-Conditioned Decomposition Repair for End-to-End Document Parsingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Benchmarking Vision-Language Models on Optical Character Recognition in Dynamic Video Environments
  2. 2026How Robust is OCR-Reasoning? Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbations
  3. 2026PolyOCR-Venus: Unified OCR Foundation Models for Text-Centric Visual Intelligence
  4. 2026Geometric Risk Control for Vision-Language Model OCR
  5. 2025DianJin-OCR-R1: Enhancing OCR Capabilities via a Reasoning-and-Tool Interleaved Vision-Language Model
  6. 2026Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models
  7. 2026OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios
  8. 2026SP-DocReader: Difference-Aware Self-Play for Precise Document OCR