mlsanityData to 7 Oct 2026

Do Current Video LLMs Have Strong OCR Abilities? A Preliminary Study

arXiv 2412.20613 ↗First posted 29 Dec 2024cs.CVcited by 7

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

With the rise of multimodal large language models, accurately extracting and understanding textual information from video content, referred to as video based optical character recognition (Video OCR), has become a crucial capability. This paper introduces a novel benchmark designed to evaluate the video OCR performance of multi-modal models in videos. Comprising 1,028 videos and 2,961 question-answer pairs, this benchmark proposes several key challenges through 6 distinct subtasks: (1) Recognition of text content itself and its basic visual attributes, (2)Semantic and Spatial Comprehension of OCR objects in videos (3) Dynamic Motion detection and Temporal Localization. We developed this benchmark using a semi-automated approach that integrates the OCR ability of image LLMs with manual refinement, balancing efficiency, cost, and data quality. Our resource aims to help advance research in video LLMs and underscores the need for improving OCR ability for video LLMs. The benchmark will be released on https://github.com/YuHuiGao/FG-Bench.git.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 33 references that are in the map. First posted on arXiv 29 Dec 2024.

Builds on

22 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document7 Mar
  2. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding4 Jul

Other references · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  6. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  7. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  8. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  10. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  11. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  12. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
8 more
  1. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  2. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  3. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokenscited by 116
  4. 2020On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answeringcited by 78
  5. 2021A Bilingual, OpenWorld Video Text Dataset and End-to-end Video Text Spotter with Transformercited by 26
  6. 2020RoadText-1K: Text Detection & Recognition Dataset for Driving Videoscited by 25
  7. 2022Watching the News: Towards VideoQA Models that can Readcited by 20
  8. 2024Exploring the Capabilities of Large Multimodal Models on Dense Textcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarioscited by 18
  2. 2025Dense Video Understanding with Gated Residual Tokenizationcited by 4
  3. 2026Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editionscited by 4
  4. 2025Tinker Tales: Interactive Storytelling Framework for Early Childhood Narrative Development and AI Literacycited by 1
  5. 2026HomeSafe-Bench: Evaluating Vision-Language Models on Unsafe Action Detection for Embodied Agents in Household Scenarioscited by 1
  6. 2026ViTexQA: A Multi-Frame Temporal Perception Dataset for Video Text Question Answeringcited by 1
  7. 2026RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Context-Independent OCR with Multimodal LLMs: Effects of Image Resolution and Visual Complexity
  2. 2025Benchmarking Vision-Language Models on Optical Character Recognition in Dynamic Video Environments
  3. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning
  4. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models
  5. 2025Reasoning-OCR: Can Large Multimodal Models Solve Complex Logical Reasoning Problems from OCR Cues?
  6. 2023What Large Language Models Bring to Text-rich VQA?
  7. 2025E-ARMOR: Edge case Assessment and Review of Multilingual Optical Character Recognition
  8. 2024TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Models