mlsanityData to 7 Oct 2026

From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network

arXiv 2108.09661 ↗First posted 22 Aug 2021cs.CVcited by 97

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

In this paper, we abandon the dominant complex language model and rethink the linguistic learning process in the scene text recognition. Different from previous methods considering the visual and linguistic information in two separate structures, we propose a Visual Language Modeling Network (VisionLAN), which views the visual and linguistic information as a union by directly enduing the vision model with language capability. Specially, we introduce the text recognition of character-wise occluded feature maps in the training stage. Such operation guides the vision model to use not only the visual texture of characters, but also the linguistic information in visual context for recognition when the visual cues are confused (e.g. occlusion, noise, etc.). As the linguistic information is acquired along with visual features without the need of extra language model, VisionLAN significantly improves the speed by 39% and adaptively considers the linguistic information to enhance the visual features for accurate recognition. Furthermore, an Occlusion Scene Text (OST) dataset is proposed to evaluate the performance on the case of missing character-wise visual cues. The state of-the-art results on several benchmarks prove our effectiveness. Code and dataset are available at https://github.com/wangyuxin87/VisionLAN.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 51 references that are in the map. First posted on arXiv 22 Aug 2021.

Builds on

31 of 51 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020End-to-End Object Detection with Transformerscited by 7,384
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  6. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  7. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  8. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 519
  9. 2016Synthetic Data for Text Localisation in Natural Imagescited by 396
  10. 2014Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognitioncited by 214
  11. 2019Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapescited by 171
  12. 2019ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection and Recognition -- RRC-MLT-2019cited by 138
19 more
  1. 2021Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detectioncited by 133
  2. 2020Towards Accurate Scene Text Recognition with Semantic Reasoning Networkscited by 131
  3. 2018Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognitioncited by 130
  4. 2020SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognitioncited by 119
  5. 2017Focusing Attention: Towards Accurate Text Recognition in Natural Imagescited by 110
  6. 2019Decoupled Attention Network for Text Recognitioncited by 102
  7. 2020RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognitioncited by 85
  8. 2018ESIR: End-to-end Scene Text Recognition via Iterative Image Rectificationcited by 79
  9. 2019Fusion of Detected Objects in Text for Visual Question Answeringcited by 74
  10. 2020ContourNet: Taking a Further Step toward Accurate Arbitrary-shaped Scene Text Detectioncited by 69
  11. 2019TextScanner: Reading Characters in Order for Robust Scene Text Recognitioncited by 68
  12. 2018Scene Text Recognition from Two-Dimensional Perspectivecited by 57
  13. 2019Symmetry-constrained Rectification Network for Scene Text Recognitioncited by 45
  14. 2019Aggregation Cross-Entropy for Sequence Recognitioncited by 22
  15. 20192D Attentional Irregular Scene Text Recognizercited by 18
  16. 2015ICDAR 2015 Text Reading in the Wild Competitioncited by 8
  17. 2020Adaptive Text Recognition through Visual Matchingcited by 8
  18. 2020Exploring Font-independent Features for Scene Text Recognitioncited by 8
  19. 2017Scene Text Recognition with Sliding Convolutional Character Modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 15; 2023: 35; 2024: 26; 2025: 10; 2026: 6.

97 papers.

Sort:

By area:

  1. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  2. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 278
  3. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  4. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  5. 2022Scene Text Recognition with Permuted Autoregressive Sequence Modelscited by 148
  6. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  7. 2022SVTR: Scene Text Recognition with a Single Visual Modelcited by 86
  8. 2023UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understandingcited by 74
  9. 2022SwinTextSpotter: Scene Text Spotting via Better Synergy between Text Detection and Text Recognitioncited by 70
  10. 2023Conditional Text Image Generation with Diffusion Modelscited by 57
  11. 2023Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizercited by 53
  12. 2021Benchmarking Chinese Text Recognition: Datasets, Baselines, and an Empirical Studycited by 52
38 more
  1. 2022Reading and Writing: Discriminative and Generative Modeling for Self-Supervised Text Recognitioncited by 45
  2. 2023Revisiting Scene Text Recognition: A Data Perspectivecited by 45
  3. 2024Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extractioncited by 44
  4. 2022ABINet++: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Spottingcited by 40
  5. 2023DTrOCR: Decoder-only Transformer for Optical Character Recognitioncited by 40
  6. 2021CDistNet: Perceiving Multi-Domain Character Distance for Robust Text Recognitioncited by 36
  7. 2022Multi-Granularity Prediction for Scene Text Recognitioncited by 36
  8. 2021Multi-modal Text Recognition Networks: Interactive Enhancements between Visual and Semantic Featurescited by 34
  9. 2022MaskOCR: Text Recognition with Masked Encoder-Decoder Pretrainingcited by 34
  10. 2023CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Modelcited by 30
  11. 2023ESTextSpotter: Towards Better Scene Text Spotting with Explicit Synergy in Transformercited by 29
  12. 2022Self-supervised Character-to-Character Distillation for Text Recognitioncited by 27
  13. 2022Self-supervised Implicit Glyph Attention for Text Recognitioncited by 25
  14. 2022Levenshtein OCRcited by 24
  15. 2024SVTRv2: CTC Beats Encoder-Decoder Models in Scene Text Recognitioncited by 23
  16. 2023Context Perception Parallel Decoder for Scene Text Recognitioncited by 19
  17. 2023Linguistic More: Taking a Further Step toward Efficient and Accurate Scene Text Recognitioncited by 18
  18. 2024Instruction-Guided Scene Text Recognitioncited by 17
  19. 2022Multimodal Semi-Supervised Learning for Text Recognitioncited by 16
  20. 2023CLIPTER: Looking at the Bigger Picture in Scene Text Recognitioncited by 16
  21. 2023Improving Scene Text Image Super-resolution via Dual Prior Modulation Networkcited by 15
  22. 2023Symmetrical Linguistic Feature Distillation with CLIP for Scene Text Recognitioncited by 14
  23. 2023LISTER: Neighbor Decoding for Length-Insensitive Scene Text Recognitioncited by 13
  24. 2023IPAD: Iterative, Parallel, and Diffusion-based Network for Scene Text Recognitioncited by 12
  25. 2022Out-of-Vocabulary Challenge Reportcited by 11
  26. 2024Out of Length Text Recognition with Sub-String Matchingcited by 11
  27. 2024Class-Aware Mask-Guided Feature Refinement for Scene Text Recognitioncited by 10
  28. 2022Pure Transformer with Integrated Experts for Scene Text Recognitioncited by 9
  29. 2023TPS++: Attention-Enhanced Thin-Plate Spline for Scene Text Recognitioncited by 9
  30. 2023TinyLVLM-eHub: Towards Comprehensive and Efficient Evaluation for Large Vision-Language Modelscited by 9
  31. 2023Self-distillation Regularized Connectionist Temporal Classification Loss for Text Recognition: A Simple Yet Effective Approachcited by 8
  32. 2024Self-Supervised Pre-training with Symmetric Superimposition Modeling for Scene Text Recognitioncited by 8
  33. 2023MRN: Multiplexed Routing Network for Incremental Multilingual Text Recognitioncited by 7
  34. 2023Multi-Granularity Prediction with Learnable Fusion for Scene Text Recognitioncited by 7
  35. 2023An Empirical Study of Scaling Law for OCRcited by 7
  36. 2023Relational Contrastive Learning for Scene Text Recognitioncited by 6
  37. 2025Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognitioncited by 6
  38. 2025From Long Videos to Engaging Clips: A Human-Inspired Video Editing Framework with Multimodal Narrative Understandingcited by 6

50 most cited of 97 All 97 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting
  2. 2022Vision-Language Pre-Training for Boosting Scene Text Detectors
  3. 2026All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts
  4. 2025Visual Text Processing: A Comprehensive Review and Unified Evaluation
  5. 2021Primitive Representation Learning for Scene Text Recognition
  6. 2024TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Models
  7. 2024Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models
  8. 2026Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models