mlsanityData to 14 Oct 2026

PsOCR: Benchmarking Large Multimodal Models for Optical Character Recognition in Low-resource Pashto Language

arXiv 2505.10055 ↗First posted 15 May 2025cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

This paper evaluates the performance of Large Multimodal Models (LMMs) on Optical Character Recognition (OCR) in the low-resource Pashto language. Natural Language Processing (NLP) in Pashto faces several challenges due to the cursive nature of its script and a scarcity of structured datasets. To address this, we developed a synthetic Pashto OCR dataset, PsOCR, consisting of one million images annotated with bounding boxes at word, line, and document levels, suitable for training and evaluating models based on different architectures, including Convolutional Neural Networks (CNNs) and Transformers. PsOCR covers variations across 1,000 unique font families, colors, image sizes, and layouts. A benchmark subset of 10K images was selected to evaluate the performance of several LMMs, including seven open-source models: DeepSeek's Janus, InternVL, MiniCPM, Florence, and Qwen (3B and 7B), and four closed-source models: GPT-4o, Gemini, Claude, and Grok. Experimental results demonstrate that Gemini achieves the best performance among all models, whereas among open-source models, Qwen-7B stands out. This work provides an insightful assessment of the capabilities and limitations of current LMMs for OCR tasks in Pashto and establishes a foundation for further research not only in Pashto OCR but also for other similar scripts such as Arabic, Persian, and Urdu. PsOCR is available at https://github.com/zirak-ai/PashtoOCR.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 69 references that are in the map. First posted on arXiv 15 May 2025.

Builds on

32 of 69 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Evaluating Menu OCR and Translation: A Benchmark for Aligning Human and Automated Evaluations in Large Vision-Language Models16 Apr
  2. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning31 Dec
  3. 2024CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy3 Dec

Other references · 29

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2019Towards VQA Models That Can Readcited by 1,718
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  8. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  9. 2021InfographicVQAcited by 437
  10. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 410
  11. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  12. 2016Synthetic Data for Text Localisation in Natural Imagescited by 396
17 more
  1. 2019Scene Text Visual Question Answeringcited by 331
  2. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  3. 2014Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognitioncited by 214
  4. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 171
  5. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  6. 2020On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answeringcited by 78
  7. 2024Focus Anywhere for Fine-grained Multi-page Document Understandingcited by 62
  8. 2020Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Textcited by 46
  9. 2025From Prompts to Templates: A Systematic Prompt Template Analysis for Real-world LLMappscited by 21
  10. 2024CAMEL-Bench: A Comprehensive Arabic LMM Benchmarkcited by 17
  11. 2025KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and Document Understandingcited by 15
  12. 2024Muffin or Chihuahua? Challenging Multimodal Large Language Models with Multipanel VQAcited by 9
  13. 2024Advancing Post-OCR Correction: A Comparative Study of Synthetic Datacited by 8
  14. 2024MMR: Evaluating Reading Ability of Large Multimodal Modelscited by 6
  15. 2022OCR Improves Machine Translation for Low-Resource Languagescited by 5
  16. 2022OCR Synthetic Benchmark Dataset for Indic Languagescited by 4
  17. 2024RoundTripOCR: A Data Generation Technique for Enhancing Post-OCR Error Correction in Low-Resource Devanagari Languagescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Benchmarking Vision-Language Models on Optical Character Recognition in Dynamic Video Environments
  2. 2025E-ARMOR: Edge case Assessment and Review of Multilingual Optical Character Recognition
  3. 2026GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scripts
  4. 2026KazakhOCR: A Synthetic Benchmark for Evaluating Multimodal Models in Low-Resource Kazakh Script OCR
  5. 2025MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios
  6. 2026Designing Production-Scale OCR for India: Multilingual and Domain-Specific Systems
  7. 2026What Can Low Resource Languages Learn From Each Other?
  8. 2025Context-Independent OCR with Multimodal LLMs: Effects of Image Resolution and Visual Complexity