mlsanityData to 7 Oct 2026

Transferring General Multimodal Pretrained Models to Text Recognition

arXiv 2212.09297 ↗First posted 19 Dec 2022cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

This paper proposes a new method, OFA-OCR, to transfer multimodal pretrained models to text recognition. Specifically, we recast text recognition as image captioning and directly transfer a unified vision-language pretrained model to the end task. Without pretraining on large-scale annotated or synthetic text recognition data, OFA-OCR outperforms the baselines and achieves state-of-the-art performance in the Chinese text recognition benchmark. Additionally, we construct an OCR pipeline with OFA-OCR, and we demonstrate that it can achieve competitive performance with the product-level API. The code (https://github.com/OFA-Sys/OFA) and demo (https://modelscope.cn/studios/damo/ofa_ocr_pipeline/summary) are publicly available.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 28 of its 43 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

28 of 43 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  11. 2016Modeling Context in Referring Expressionscited by 1,054
  12. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
16 more
  1. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  2. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 519
  3. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  4. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  5. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 278
  6. 2021Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognitioncited by 194
  7. 2019ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT)cited by 147
  8. 2020Towards Accurate Scene Text Recognition with Semantic Reasoning Networkscited by 131
  9. 2018Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognitioncited by 130
  10. 2019ICDAR 2019 Robust Reading Challenge on Reading Chinese Text on Signboardcited by 127
  11. 2019ICDAR 2019 Competition on Large-scale Street View Text with Partial Labeling -- RRC-LSVTcited by 121
  12. 2017ICDAR2017 Competition on Reading Chinese Text in the Wild (RCTW-17)cited by 103
  13. 2021M6: A Chinese Multimodal Pretrainercited by 91
  14. 2021Vision Transformer for Fast and Efficient Scene Text Recognitioncited by 81
  15. 2021Benchmarking Chinese Text Recognition: Datasets, Baselines, and an Empirical Studycited by 52
  16. 2022MaskOCR: Text Recognition with Masked Encoder-Decoder Pretrainingcited by 34

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Can Visual Language Models Replace OCR-Based Visual Question Answering Pipelines in Production? A Case Study in Retailcited by 4
  2. 2025SELECT: Detecting Label Errors in Real-world Scene Text Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models
  2. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning
  3. 2026Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models
  4. 2024TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Models
  5. 2026LightOnOCR: A 1B End-to-End Multilingual Vision-Language Model for State-of-the-Art OCR
  6. 2026PP-OCRv5: A Specialized 5M-Parameter Model Rivaling Billion-Parameter Vision-Language Models on OCR Tasks
  7. 2026Xiaomi-OCR-0 Technical Report
  8. 2025Context-Independent OCR with Multimodal LLMs: Effects of Image Resolution and Visual Complexity