# Qalam : A Multimodal LLM for Arabic Optical Character and Handwriting Recognition

arXiv: https://arxiv.org/abs/2407.13559 · HTML: https://mlsanity.com/paper/2407.13559/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 74 references that are in the map.
- First posted on arXiv 18 Jul 2024.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding](https://mlsanity.com/paper/2306.17107/) (2023)
- [TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document](https://mlsanity.com/paper/2403.04473/) (2024)
- [Recursive Recurrent Nets with Attention Modeling for OCR in the Wild](https://mlsanity.com/paper/1603.03101/) (2016)
- [Pay Attention to What You Read: Non-recurrent Handwritten Text-Line Recognition](https://mlsanity.com/paper/2005.13044/) (2020)
- [Evaluating Sequence-to-Sequence Models for Handwritten Text Recognition](https://mlsanity.com/paper/1903.07377/) (2019)
- [End-to-end Handwritten Paragraph Text Recognition Using a Vertical Attention Network](https://mlsanity.com/paper/2012.03868/) (2020)
- [Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation](https://mlsanity.com/paper/2310.16809/) (2023)
- [Scan, Attend and Read: End-to-End Handwritten Paragraph Recognition with MDLSTM Attention](https://mlsanity.com/paper/1604.03286/) (2016)
- [Dropout improves Recurrent Neural Networks for Handwriting Recognition](https://mlsanity.com/paper/1312.4569/) (2013)
- [Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks](https://mlsanity.com/paper/2403.01031/) (2024)
- [Accurate, Data-Efficient, Unconstrained Text Recognition with Convolutional Neural Networks](https://mlsanity.com/paper/1812.11894/) (2018)
- [Recurrence-free unconstrained handwritten text recognition using gated fully convolutional network](https://mlsanity.com/paper/2012.04961/) (2020)
- [Rescoring Sequence-to-Sequence Models for Text Line Recognition with CTC-Prefixes](https://mlsanity.com/paper/2110.05909/) (2021)
- [AlexU-Word: A New Dataset for Isolated-Word Closed-Vocabulary Offline Arabic Handwriting Recognition](https://mlsanity.com/paper/1411.4670/) (2014)

## Built on by (most cited first, 7 of 7)

- [QARI-OCR: High-Fidelity Arabic Text Recognition through Multimodal Large Language Model Adaptation](https://mlsanity.com/paper/2506.02295/) (2025)
- [Large Multimodal Models for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2502.05568/) (2025)
- [Arabic Multimodal Machine Learning: Datasets, Applications, Approaches, and Challenges](https://mlsanity.com/paper/2508.12227/) (2025)
- [Invizo: Arabic Handwritten Document Optical Character Recognition Solution](https://mlsanity.com/paper/2502.05277/) (2025)
- [A-SEA3L-QA: A Fully Automated Self-Evolving, Adversarial Workflow for Arabic Long-Context Question-Answer Generation](https://mlsanity.com/paper/2509.02864/) (2025)
- [LLM-Driven AutoML for Cross-Lingual Handwritten OCR: Closed-Loop Neural Architecture Search with GPT-5, GPT-4o, and Claude Sonnet 4](https://mlsanity.com/paper/2607.15509/) (2026)
- [When Do VLMs Help Arabic Manuscript OCR? A Cross-Dataset Study](https://mlsanity.com/paper/2608.22366/) (2026)

## Abstract (arXiv)

Arabic Optical Character Recognition (OCR) and Handwriting Recognition (HWR) pose unique challenges due to the cursive and context-sensitive nature of the Arabic script. This study introduces Qalam, a novel foundation model designed for Arabic OCR and HWR, built on a SwinV2 encoder and RoBERTa decoder architecture. Our model significantly outperforms existing methods, achieving a Word Error Rate (WER) of just 0.80% in HWR tasks and 1.18% in OCR tasks. We train Qalam on a diverse dataset, including over 4.5 million images from Arabic manuscripts and a synthetic dataset comprising 60k image-text pairs. Notably, Qalam demonstrates exceptional handling of Arabic diacritics, a critical feature in Arabic scripts. Furthermore, it shows a remarkable ability to process high-resolution inputs, addressing a common limitation in current OCR systems. These advancements underscore Qalam's potential as a leading solution for Arabic script recognition, offering a significant leap in accuracy and efficiency.