mlsanityData to 7 Oct 2026

CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model

arXiv 2305.14014 ↗First posted 23 May 2023cs.CVcited by 30

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Pre-trained vision-language models~(VLMs) are the de-facto foundation models for various downstream tasks. However, scene text recognition methods still prefer backbones pre-trained on a single modality, namely, the visual modality, despite the potential of VLMs to serve as powerful scene text readers. For example, CLIP can robustly identify regular (horizontal) and irregular (rotated, curved, blurred, or occluded) text in images. With such merits, we transform CLIP into a scene text reader and introduce CLIP4STR, a simple yet effective STR method built upon image and text encoders of CLIP. It has two encoder-decoder branches: a visual branch and a cross-modal branch. The visual branch provides an initial prediction based on the visual feature, and the cross-modal branch refines this prediction by addressing the discrepancy between the visual feature and text semantics. To fully leverage the capabilities of both branches, we design a dual predict-and-refine decoding scheme for inference. We scale CLIP4STR in terms of the model size, pre-training data, and training data, achieving state-of-the-art performance on 13 STR benchmarks. Additionally, a comprehensive empirical study is provided to enhance the understanding of the adaptation of CLIP to STR. Our method establishes a simple yet strong baseline for future STR research with VLMs.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 80 of its 120 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

80 of 120 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network22 Aug

Other references · 79

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  9. 2016Layer Normalizationcited by 5,579
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  8. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2017Mixed Precision Trainingcited by 1,135
  12. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  13. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  14. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  15. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  16. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  17. 2022Language-driven Semantic Segmentationcited by 711
  18. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  19. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  20. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  21. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  22. 2021ImageNet-21K Pretraining for the Massescited by 607
  23. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 519
  24. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  25. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  26. 2016Synthetic Data for Text Localisation in Natural Imagescited by 396
  27. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  28. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  29. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  30. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 278
  31. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  32. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  33. 2014Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognitioncited by 214
  34. 2021Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognitioncited by 194
  35. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 170
  36. 2022Scene Text Recognition with Permuted Autoregressive Sequence Modelscited by 148
  37. 2019ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT)cited by 147
  38. 2019ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection and Recognition -- RRC-MLT-2019cited by 138

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 9; 2025: 12; 2026: 5.

30 papers.

Sort:

By area:

  1. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  2. 2024Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Renderingcited by 80
  3. 2024SVTRv2: CTC Beats Encoder-Decoder Models in Scene Text Recognitioncited by 23
  4. 2025UniRec-0.1B: Unified Text and Formula Recognition with 0.1B Parameterscited by 12
  5. 2024NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Modelscited by 9
  6. 2023An Empirical Study of Scaling Law for OCRcited by 7
  7. 2024Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrievalcited by 6
  8. 2023Masked and Permuted Implicit Context Learning for Scene Text Recognitioncited by 4
  9. 2025EVA02-AT: Egocentric Video-Language Understanding with Spatial-Temporal Rotary Positional Embeddings and Symmetric Optimizationcited by 3
  10. 2025An Explainable Deep Neural Network with Frequency-Aware Channel and Spatial Refinement for Flood Prediction in Sustainable Citiescited by 2
  11. 2026ZAYA1-VL-8B Technical Reportcited by 2
  12. 2025MSTAR: Box-free Multi-query Scene Text Retrieval with Attention Recyclingcited by 1
18 more
  1. 2025LadderMoE: Ladder-Side Mixture of Experts Adapters for Bronze Inscription Recognitioncited by 1
  2. 20253D Weakly Supervised Semantic Segmentation via Class-Aware and Geometry-Guided Pseudo-Label Refinementcited by 1
  3. 2025Bharat Scene Text: A Novel Comprehensive Dataset and Benchmark for Indian Language Scene Text Understandingcited by 1
  4. 2023Reading Between the Mud: A Challenging Motorcycle Racer Number Datasetcited by 0
  5. 2024Enhancing Small Object Encoding in Deep Neural Networks: Introducing Fast&Focused-Net with Volume-wise Dot Product Layercited by 0
  6. 2024Beyond the Mud: Datasets and Benchmarks for Computer Vision in Off-Road Racingcited by 0
  7. 2024HAAP: Vision-context Hierarchical Attention Autoregressive with Adaptive Permutation for Scene Text Recognitioncited by 0
  8. 2024Classification of Non-native Handwritten Characters Using Convolutional Neural Networkcited by 0
  9. 2024Stratified Domain Adaptation: A Progressive Self-Training Approach for Scene Text Recognitioncited by 0
  10. 2025Accurate Scene Text Recognition with Efficient Model Scaling and Cloze Self-Distillationcited by 0
  11. 2025Inverse Scene Text Removalcited by 0
  12. 2025TEACH: Text Encoding as Curriculum Hints for Scene Text Recognitioncited by 0
  13. 2025Text-VQA Aug: Pipelined Harnessing of Large Multimodal Models for Automated Synthesiscited by 0
  14. 2025Remodeling Semantic Relationships in Vision-Language Fine-Tuningcited by 0
  15. 2026TraRA: Trajectory-level Recognition Aggregation for Video Text Spotting in Urban Surveillancecited by 0
  16. 2026HAT Super-Resolution and a PARSeq+CLIP4STR Voting Ensemble for Extreme In-the-Wild License Plate Recognitioncited by 0
  17. 2026When VLMs Trust Context: Evaluating Scene Text Recognition under Misleading Contextcited by 0
  18. 2026Handwritten Text Recognition Lives in the High-Pixel Variance Subspacecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Decoder Pre-Training with only Text for Scene Text Recognition
  2. 2022Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting
  3. 2023Turning a CLIP Model into a Scene Text Detector
  4. 2025Visual Text Processing: A Comprehensive Review and Unified Evaluation
  5. 2023Text encoders bottleneck compositionality in contrastive vision-language models
  6. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
  7. 2025Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions
  8. 2024CLIPArTT: Adaptation of CLIP to New Domains at Test Time