mlsanityData to 7 Oct 2026

TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models

arXiv 2109.10282 ↗First posted 21 Sep 2021cs.CL cs.CVcited by 278

Not on the map. Why?

Abstract (arXiv)

Text recognition is a long-standing research problem for document digitalization. Existing approaches are usually built based on CNN for image understanding and RNN for char-level text generation. In addition, another language model is usually needed to improve the overall accuracy as a post-processing step. In this paper, we propose an end-to-end text recognition approach with pre-trained image Transformer and text Transformer models, namely TrOCR, which leverages the Transformer architecture for both image understanding and wordpiece-level text generation. The TrOCR model is simple but effective, and can be pre-trained with large-scale synthetic data and fine-tuned with human-labeled datasets. Experiments show that the TrOCR model outperforms the current state-of-the-art models on the printed, handwritten and scene text recognition tasks. The TrOCR models and code are publicly available at urlhttps://aka.ms/trocr.

Key facts. Cited by 278 papers in the map (data through 7 Oct 2026). Builds on 46 of its 75 references that are in the map. First posted on arXiv 21 Sep 2021.

Builds on

46 of 75 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  9. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  12. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
34 more
  1. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  2. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 519
  3. 2016Synthetic Data for Text Localisation in Natural Imagescited by 396
  4. 2014Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognitioncited by 214
  5. 2019Real-time Scene Text Detection with Differentiable Binarizationcited by 212
  6. 2021Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognitioncited by 194
  7. 2019What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysiscited by 191
  8. 2020Towards Accurate Scene Text Recognition with Semantic Reasoning Networkscited by 131
  9. 2016Robust Scene Text Recognition with Automatic Rectificationcited by 121
  10. 2019Decoupled Attention Network for Text Recognitioncited by 102
  11. 2021From Two to One: A New Scene Text Recognizer with Visual Language Modeling Networkcited by 97
  12. 2020RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognitioncited by 85
  13. 2021Vision Transformer for Fast and Efficient Scene Text Recognitioncited by 81
  14. 2018ESIR: End-to-end Scene Text Recognition via Iterative Image Rectificationcited by 79
  15. 2018NRTR: A No-Recurrence Sequence-to-Sequence Model For Scene Text Recognitioncited by 73
  16. 2020Pay Attention to What You Read: Non-recurrent Handwritten Text-Line Recognitioncited by 71
  17. 2020SCATTER: Selective Context Attentional Scene Text Recognizercited by 69
  18. 2019TextScanner: Reading Characters in Order for Robust Scene Text Recognitioncited by 68
  19. 2021What If We Only Use Real Datasets for Scene Text Recognition? Toward Scene Text Recognition With Fewer Labelscited by 63
  20. 2019On Recognizing Texts of Arbitrary Shapes with 2D Self-Attentioncited by 59
  21. 2019Evaluating Sequence-to-Sequence Models for Handwritten Text Recognitioncited by 58
  22. 2020Handwritten Optical Character Recognition (OCR): A Comprehensive Systematic Literature Review (SLR)cited by 56
  23. 2016Joint Line Segmentation and Transcription for End-to-End Handwritten Paragraph Recognitioncited by 46
  24. 2021Rethinking Text Line Recognition Modelscited by 45
  25. 2013Dropout improves Recurrent Neural Networks for Handwriting Recognitioncited by 39
  26. 2021Primitive Representation Learning for Scene Text Recognitioncited by 28
  27. 2018An Efficient End-to-End Neural Model for Handwritten Text Recognitioncited by 25
  28. 2019A Scalable Handwritten Text Recognition Systemcited by 25
  29. 2021Joint Visual Semantic Reasoning: Multi-Stage Decoder for Text Recognitioncited by 24
  30. 2021Towards the Unseen: Iterative Text Recognition by Distilling from Errorscited by 12
  31. 2016Generating Synthetic Data for Text Recognitioncited by 11
  32. 2019Bidirectional Scene Text Recognition with a Single Decodercited by 7
  33. 2021Representation and Correlation Enhanced Encoder-Decoder Framework for Scene Text Recognitioncited by 6
  34. 2021Revisiting Classification Perspective on Scene Text Recognitioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 15; 2023: 62; 2024: 62; 2025: 64; 2026: 73.

278 papers.

Sort:

By area:

  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  2. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 240
  3. 2023TextDiffuser: Diffusion Models as Text Painterscited by 223
  4. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 174
  5. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  6. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 136
  7. 2023Large Language Models for Robotics: A Surveycited by 132
  8. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95
  9. 2022DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognitioncited by 62
  10. 2025TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Modelcited by 57
  11. 2023Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generationcited by 56
  12. 2023Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluationcited by 46
38 more
  1. 2023DiffUTE: Universal Text Editing Diffusion Modelcited by 44
  2. 2024Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extractioncited by 44
  3. 2023OWL: A Large Language Model for IT Operationscited by 43
  4. 2023DTrOCR: Decoder-only Transformer for Optical Character Recognitioncited by 40
  5. 2024MoAI: Mixture of All Intelligence for Large Language and Vision Modelscited by 36
  6. 2022MaskOCR: Text Recognition with Masked Encoder-Decoder Pretrainingcited by 34
  7. 2023CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Modelcited by 30
  8. 2022Levenshtein OCRcited by 24
  9. 2024HTR-VT: Handwritten Text Recognition with Vision Transformercited by 24
  10. 2026AgentOCR: Reimagining Agent History via Optical Self-Compressioncited by 22
  11. 2026LightOnOCR: A 1B End-to-End Multilingual Vision-Language Model for State-of-the-Art OCRcited by 22
  12. 2024The Manga Whisperer: Automatically Generating Transcriptions for Comicscited by 20
  13. 2023Less is More: Removing Text-regions Improves CLIP Training Efficiency and Robustnesscited by 19
  14. 2021Rescoring Sequence-to-Sequence Models for Text Line Recognition with CTC-Prefixescited by 18
  15. 2022Vision-Language Pre-Training for Boosting Scene Text Detectorscited by 18
  16. 2024VATr++: Choose Your Words Wisely for Handwritten Text Generationcited by 18
  17. 2025HunyuanOCR Technical Reportcited by 18
  18. 2024MolParser: End-to-end Visual Recognition of Molecule Structures in the Wildcited by 17
  19. 2022The LAM Dataset: A Novel Benchmark for Line-Level Handwritten Text Recognitioncited by 16
  20. 2022Boosting Modern and Historical Handwritten Text Recognition with Deformable Convolutionscited by 14
  21. 2023Summaries as Captions: Generating Figure Captions for Scientific Documents with Automated Text Summarizationcited by 14
  22. 2024Layout Agnostic Scene Text Image Synthesis with Diffusion Modelscited by 14
  23. 2023How to Choose Pretrained Handwriting Recognition Models for Single Writer Fine-Tuningcited by 13
  24. 2023LISTER: Neighbor Decoding for Length-Insensitive Scene Text Recognitioncited by 13
  25. 2024End-to-End Full-Page Optical Music Recognition for Pianoform Sheet Musiccited by 13
  26. 2024Muharaf: Manuscripts of Handwritten Arabic Dataset for Cursive Text Recognitioncited by 13
  27. 2025Zero-Shot Styled Text Image Generation, but Make It Autoregressivecited by 13
  28. 2025Handwritten Text Recognition: A Surveycited by 12
  29. 2023American Stories: A Large-Scale Structured Text Dataset of Historical U.S. Newspaperscited by 11
  30. 2025Benchmarking Large Language Models for Handwritten Text Recognitioncited by 11
  31. 2022Transformer-based HTR for Historical Documentscited by 10
  32. 2023Efficient OCR for Building a Diverse Digital Historycited by 10
  33. 2024On the Generalization of Handwritten Text Recognition Modelscited by 10
  34. 2025GlyphMastero: A Glyph Encoder for High-Fidelity Scene Text Editingcited by 10
  35. 2023What Makes ImageNet Look Unlike LAIONcited by 9
  36. 2023A Transformer-based Approach for Arabic Offline Handwritten Text Recognitioncited by 9
  37. 2023Automatic Report Generation for Histopathology images using pre-trained Vision Transformerscited by 9
  38. 2024DANIEL: A fast Document Attention Network for Information Extraction and Labelling of handwritten documentscited by 9

50 most cited of 278 All 278 (JSON)