mlsanityData to 7 Oct 2026

TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text

arXiv 2105.05486 ↗First posted 12 May 2021cs.CVcited by 170

Not on the map. Why?

Cited by 46 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A crucial component for the scene text based reasoning required for TextVQA and TextCaps datasets involve detecting and recognizing text present in the images using an optical character recognition (OCR) system. The current systems are crippled by the unavailability of ground truth text annotations for these datasets as well as lack of scene text detection and recognition datasets on real images disallowing the progress in the field of OCR and evaluation of scene text based reasoning in isolation from OCR systems. In this work, we propose TextOCR, an arbitrary-shaped scene text detection and recognition with 900k annotated words collected on real images from TextVQA dataset. We show that current state-of-the-art text-recognition (OCR) models fail to perform well on TextOCR and that training on TextOCR helps achieve state-of-the-art performance on multiple other OCR datasets as well. We use a TextOCR trained OCR model to create PixelM4C model which can do scene text based reasoning on an image in an end-to-end fashion, allowing us to revisit several design choices to achieve new state-of-the-art performance on TextVQA dataset.

Key facts. Cited by 170 papers in the map (data through 7 Oct 2026). Builds on 39 of its 71 references that are in the map. First posted on arXiv 12 May 2021.

Builds on

39 of 71 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2019Towards VQA Models That Can Readcited by 1,696
  9. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  10. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 519
  11. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  12. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
27 more
  1. 2016Synthetic Data for Text Localisation in Natural Imagescited by 396
  2. 2016FastText.zip: Compressing text classification modelscited by 331
  3. 2019Scene Text Visual Question Answeringcited by 327
  4. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  5. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  6. 2014Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognitioncited by 214
  7. 2019What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysiscited by 191
  8. 2019Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapescited by 171
  9. 2020ABCNet: Real-time Scene Text Spotting with Adaptive Bezier-Curve Networkcited by 170
  10. 2019ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT)cited by 147
  11. 2019ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection and Recognition -- RRC-MLT-2019cited by 138
  12. 2019ICDAR 2019 Competition on Large-scale Street View Text with Partial Labeling -- RRC-LSVTcited by 121
  13. 2020Mask TextSpotter v3: Segmentation Proposal Network for Robust Scene Text Spottingcited by 114
  14. 2020Captioning Images Taken by People Who Are Blindcited by 108
  15. 2019Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQAcited by 107
  16. 2017ICDAR2017 Competition on Reading Chinese Text in the Wild (RCTW-17)cited by 103
  17. 2017Detecting Curve Text in the Wild: New Dataset and New Solutioncited by 82
  18. 2020On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answeringcited by 78
  19. 2019Rosetta: Large scale system for text detection and recognition in imagescited by 69
  20. 2019All You Need Is Boundary: Toward Arbitrary-Shaped Text Spottingcited by 66
  21. 2019Towards Unconstrained End-to-End Text Spottingcited by 57
  22. 2019LEAF-QA: Locate, Encode & Attend for Figure Question Answeringcited by 53
  23. 2020Spatially Aware Multimodal Transformers for TextVQAcited by 44
  24. 2020Structured Multimodal Attentions for TextVQAcited by 23
  25. 2020Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answeringcited by 19
  26. 2021A Multiplexed Network for End-to-End, Multilingual OCRcited by 18
  27. 2020RUArt: A Novel Text-Centered Solution for Text-Based Visual Question Answeringcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 22; 2023: 25; 2024: 49; 2025: 52; 2026: 17.

170 papers.

Sort:

By area:

  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  2. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  3. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  4. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  5. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  6. 2022Scene Text Recognition with Permuted Autoregressive Sequence Modelscited by 148
  7. 2023PACO: Parts and Attributes of Common Objectscited by 146
  8. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  9. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  10. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  11. 2022Towards End-to-End Unified Scene Text Detection and Layout Analysiscited by 96
  12. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
38 more
  1. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  2. 2022DeepSolo: Let Transformer Decoder with Explicit Points Solo for Text Spottingcited by 74
  3. 2024OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognitioncited by 74
  4. 2021LaTr: Layout-Aware Transformer for Scene-Text VQAcited by 72
  5. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  6. 2025Detect Anything via Next Point Predictioncited by 66
  7. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  8. 2025UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encodingcited by 49
  9. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
  10. 2021TextStyleBrush: Transfer of Text Aesthetics from a Single Examplecited by 48
  11. 2021Open Images V5 Text Annotation and Yet Another Mask Text Spottercited by 46
  12. 2022Reading and Writing: Discriminative and Generative Modeling for Self-Supervised Text Recognitioncited by 45
  13. 2023Revisiting Scene Text Recognition: A Data Perspectivecited by 45
  14. 2023Turning a CLIP Model into a Scene Text Detectorcited by 44
  15. 2023Improving Diffusion Models for Scene Text Editing with Dual Encoderscited by 44
  16. 2023DiffUTE: Universal Text Editing Diffusion Modelcited by 44
  17. 2024SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Foldingcited by 42
  18. 2024PromptFix: You Prompt and We Fix the Photocited by 40
  19. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  20. 2022MaskOCR: Text Recognition with Masked Encoder-Decoder Pretrainingcited by 34
  21. 2022GLASS: Global to Local Attention for Scene-Text Spottingcited by 34
  22. 2024VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learningcited by 34
  23. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  24. 2023CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Modelcited by 30
  25. 2024Hi-SAM: Marrying Segment Anything Model for Hierarchical Text Segmentationcited by 30
  26. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  27. 2022Self-supervised Character-to-Character Distillation for Text Recognitioncited by 27
  28. 2023Towards Unified Scene Text Spotting based on Sequence Generationcited by 27
  29. 2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Modelscited by 25
  30. 2023ICDAR 2023 Competition on Hierarchical Text Detection and Recognitioncited by 23
  31. 2025FaceXBench: Evaluating Multimodal LLMs on Face Understandingcited by 23
  32. 2024Visual Text Generation in the Wildcited by 22
  33. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Modelscited by 22
  34. 2024Revisiting Tampered Scene Text Detection in the Era of Generative AIcited by 21
  35. 2024ROADWork: A Dataset and Benchmark for Learning to Recognize, Observe, Analyze and Drive Through Work Zonescited by 20
  36. 2025Valley2: Exploring Multimodal Models with Scalable Vision-Language Designcited by 20
  37. 2025Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligencecited by 20
  38. 2025Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understandingcited by 20

50 most cited of 170 All 170 (JSON)