mlsanityData to 7 Oct 2026

CLIPTER: Looking at the Bigger Picture in Scene Text Recognition

arXiv 2301.07464 ↗First posted 18 Jan 2023cs.CV cs.LGcited by 16

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Reading text in real-world scenarios often requires understanding the context surrounding it, especially when dealing with poor-quality text. However, current scene text recognizers are unaware of the bigger picture as they operate on cropped text images. In this study, we harness the representative capabilities of modern vision-language models, such as CLIP, to provide scene-level information to the crop-based recognizer. We achieve this by fusing a rich representation of the entire image, obtained from the vision-language model, with the recognizer word-level features via a gated cross-attention mechanism. This component gradually shifts to the context-enhanced representation, allowing for stable fine-tuning of a pretrained recognizer. We demonstrate the effectiveness of our model-agnostic framework, CLIPTER (CLIP TExt Recognition), on leading text recognition architectures and achieve state-of-the-art results across multiple benchmarks. Furthermore, our analysis highlights improved robustness to out-of-vocabulary words and enhanced generalization in low-data regimes.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 72 references that are in the map. First posted on arXiv 18 Jan 2023.

Builds on

57 of 72 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
38 more
  1. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  2. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  3. 2016Synthetic Data for Text Localisation in Natural Imagescited by 396
  4. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  5. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  6. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  7. 2014Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognitioncited by 214
  8. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  9. 2021Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognitioncited by 194
  10. 2019What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysiscited by 191
  11. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 170
  12. 2022Scene Text Recognition with Permuted Autoregressive Sequence Modelscited by 148
  13. 2019ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT)cited by 147
  14. 2019ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection and Recognition -- RRC-MLT-2019cited by 138
  15. 2020Towards Accurate Scene Text Recognition with Semantic Reasoning Networkscited by 131
  16. 2019MMTM: Multimodal Transfer Module for CNN Fusioncited by 128
  17. 2019ICDAR 2019 Robust Reading Challenge on Reading Chinese Text on Signboardcited by 127
  18. 2020SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognitioncited by 119
  19. 2020Mask TextSpotter v3: Segmentation Proposal Network for Robust Scene Text Spottingcited by 114
  20. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  21. 2017ICDAR2017 Competition on Reading Chinese Text in the Wild (RCTW-17)cited by 103
  22. 2021From Two to One: A New Scene Text Recognizer with Visual Language Modeling Networkcited by 97
  23. 2022Towards End-to-End Unified Scene Text Detection and Layout Analysiscited by 96
  24. 2021ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spottingcited by 84
  25. 2021Vision Transformer for Fast and Efficient Scene Text Recognitioncited by 81
  26. 2021LaTr: Layout-Aware Transformer for Scene-Text VQAcited by 72
  27. 2020Sequence-to-Sequence Contrastive Learning for Text Recognitioncited by 70
  28. 2022SwinTextSpotter: Scene Text Spotting via Better Synergy between Text Detection and Text Recognitioncited by 70
  29. 2020SCATTER: Selective Context Attentional Scene Text Recognizercited by 69
  30. 2021Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristicscited by 67
  31. 2021What If We Only Use Real Datasets for Scene Text Recognition? Toward Scene Text Recognition With Fewer Labelscited by 63
  32. 2020MANGO: A Mask Attention Guided One-Stage Scene Text Spottercited by 51
  33. 2022Towards Weakly-Supervised Text Spotting using a Multi-Task Transformercited by 41
  34. 2020On Vocabulary Reliance in Scene Text Recognitioncited by 38
  35. 2021Multi-modal Text Recognition Networks: Interactive Enhancements between Visual and Semantic Featurescited by 34
  36. 2022MaskOCR: Text Recognition with Masked Encoder-Decoder Pretrainingcited by 34
  37. 2022GLASS: Global to Local Attention for Scene-Text Spottingcited by 34
  38. 2022Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spottingcited by 27

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 4; 2025: 5; 2026: 3.

16 papers.

Sort:

By area:

  1. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captionscited by 33
  2. 2024Question Aware Vision Transformer for Multimodal Reasoningcited by 32
  3. 2023CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Modelcited by 30
  4. 2024DocVLM: Make Your VLM an Efficient Readercited by 27
  5. 2024GRAM: Global Reasoning for Multi-Page VQAcited by 23
  6. 2023Symmetrical Linguistic Feature Distillation with CLIP for Scene Text Recognitioncited by 14
  7. 2025Towards General Urban Monitoring with Vision-Language Models: A Review, Evaluation, and a Research Agendacited by 2
  8. 2026DODO: Discrete OCR Diffusion Modelscited by 2
  9. 2023Harnessing the Power of Multi-Lingual Datasets for Pre-training: Towards Enhancing Text Spotting Performancecited by 1
  10. 2024TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Modelscited by 1
  11. 2025MSTAR: Box-free Multi-query Scene Text Retrieval with Attention Recyclingcited by 1
  12. 2025Bharat Scene Text: A Novel Comprehensive Dataset and Benchmark for Indian Language Scene Text Understandingcited by 1
4 more
  1. 2025BioD2C: A Dual-level Semantic Consistency Constraint Framework for Biomedical VQAcited by 0
  2. 2025TEACH: Text Encoding as Curriculum Hints for Scene Text Recognitioncited by 0
  3. 2026When VLMs Trust Context: Evaluating Scene Text Recognition under Misleading Contextcited by 0
  4. 2026Handwritten Text Recognition Lives in the High-Pixel Variance Subspacecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Decoder Pre-Training with only Text for Scene Text Recognition
  2. 2023Turning a CLIP Model into a Scene Text Detector
  3. 2025Helping CLIP See Both the Forest and the Trees: A Decomposition and Description Approach
  4. 2024CLIPArTT: Adaptation of CLIP to New Domains at Test Time
  5. 2025DiffCLIP: Differential Attention Meets CLIP
  6. 2026LiteEmbed: Adapting CLIP to Rare Classes
  7. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters
  8. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models