mlsanityData to 7 Oct 2026

Benchmarking Vision-Language Models on Optical Character Recognition in Dynamic Video Environments

arXiv 2502.06445 ↗First posted 10 Feb 2025cs.CVcited by 10

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

This paper introduces an open-source benchmark for evaluating Vision-Language Models (VLMs) on Optical Character Recognition (OCR) tasks in dynamic video environments. We present a curated dataset containing 1,477 manually annotated frames spanning diverse domains, including code editors, news broadcasts, YouTube videos, and advertisements. Three state of the art VLMs - Claude-3, Gemini-1.5, and GPT-4o are benchmarked against traditional OCR systems such as EasyOCR and RapidOCR. Evaluation metrics include Word Error Rate (WER), Character Error Rate (CER), and Accuracy. Our results highlight the strengths and limitations of VLMs in video-based OCR tasks, demonstrating their potential to outperform conventional OCR models in many scenarios. However, challenges such as hallucinations, content security policies, and sensitivity to occluded or stylized text remain. The dataset and benchmarking framework are publicly available to foster further research.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 3 of its 13 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

3 of 13 references.

References · 3

  1. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  2. 2024VHELM: A Holistic Evaluation of Vision Language Modelscited by 53
  3. 2025Video Summarisation with Incident and Context Information using Generative AIcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 3.

10 papers.

  1. 2025MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarioscited by 18
  2. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  3. 2025ReVision: A Dataset and Baseline VLM for Privacy-Preserving Task-Oriented Visual Instruction Rewritingcited by 6
  4. 2026Structure-Aware Text Recognition for Ancient Greek Critical Editionscited by 5
  5. 2025Video-based Vehicle Surveillance in the Wild: License Plate, Make, and Model Recognition with Self Reflective Vision-Language Modelscited by 2
  6. 2025E-ARMOR: Edge case Assessment and Review of Multilingual Optical Character Recognitioncited by 1
  7. 2025Vision Language Models Are Not (Yet) Spelling Correctorscited by 1
  8. 2026If you're waiting for a sign... that might not be it! Mitigating Trust Boundary Confusion from Visual Injections on Vision-Language Agentic Systemscited by 1
  9. 2025Exploring Light-Weight Object Recognition for Real-Time Document Detectioncited by 0
  10. 2026ReViCo: Unveiling the Limitations of VLMs in Visual Text Understanding via Error Correctioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Do Current Video LLMs Have Strong OCR Abilities? A Preliminary Study
  2. 2026PP-OCRv5: A Specialized 5M-Parameter Model Rivaling Billion-Parameter Vision-Language Models on OCR Tasks
  3. 2026GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scripts
  4. 2025PsOCR: Benchmarking Large Multimodal Models for Optical Character Recognition in Low-resource Pashto Language
  5. 2024TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Models
  6. 2026Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models
  7. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
  8. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models