# Benchmarking Vision-Language Models on Optical Character Recognition in Dynamic Video Environments

arXiv: https://arxiv.org/abs/2502.06445 · HTML: https://mlsanity.com/paper/2502.06445/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 3 of its 13 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 3 of 3)

- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [VHELM: A Holistic Evaluation of Vision Language Models](https://mlsanity.com/paper/2410.07112/) (2024)
- [Video Summarisation with Incident and Context Information using Generative AI](https://mlsanity.com/paper/2501.04764/) (2025)

## Built on by (most cited first, 10 of 10)

- [MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios](https://mlsanity.com/paper/2505.21333/) (2025)
- [Object Detection with Multimodal Large Vision-Language Models: An In-depth Review](https://mlsanity.com/paper/2508.19294/) (2025)
- [ReVision: A Dataset and Baseline VLM for Privacy-Preserving Task-Oriented Visual Instruction Rewriting](https://mlsanity.com/paper/2502.14780/) (2025)
- [Structure-Aware Text Recognition for Ancient Greek Critical Editions](https://mlsanity.com/paper/2603.02803/) (2026)
- [Video-based Vehicle Surveillance in the Wild: License Plate, Make, and Model Recognition with Self Reflective Vision-Language Models](https://mlsanity.com/paper/2508.01387/) (2025)
- [E-ARMOR: Edge case Assessment and Review of Multilingual Optical Character Recognition](https://mlsanity.com/paper/2509.03615/) (2025)
- [Vision Language Models Are Not (Yet) Spelling Correctors](https://mlsanity.com/paper/2509.17418/) (2025)
- [If you're waiting for a sign... that might not be it! Mitigating Trust Boundary Confusion from Visual Injections on Vision-Language Agentic Systems](https://mlsanity.com/paper/2604.19844/) (2026)
- [Exploring Light-Weight Object Recognition for Real-Time Document Detection](https://mlsanity.com/paper/2509.06246/) (2025)
- [ReViCo: Unveiling the Limitations of VLMs in Visual Text Understanding via Error Correction](https://mlsanity.com/paper/2608.27154/) (2026)

## Abstract (arXiv)

This paper introduces an open-source benchmark for evaluating Vision-Language Models (VLMs) on Optical Character Recognition (OCR) tasks in dynamic video environments. We present a curated dataset containing 1,477 manually annotated frames spanning diverse domains, including code editors, news broadcasts, YouTube videos, and advertisements. Three state of the art VLMs - Claude-3, Gemini-1.5, and GPT-4o are benchmarked against traditional OCR systems such as EasyOCR and RapidOCR. Evaluation metrics include Word Error Rate (WER), Character Error Rate (CER), and Accuracy. Our results highlight the strengths and limitations of VLMs in video-based OCR tasks, demonstrating their potential to outperform conventional OCR models in many scenarios. However, challenges such as hallucinations, content security policies, and sensitivity to occluded or stylized text remain. The dataset and benchmarking framework are publicly available to foster further research.