# Self-Supervised Pre-training with Symmetric Superimposition Modeling for Scene Text Recognition

arXiv: https://arxiv.org/abs/2405.05841 · HTML: https://mlsanity.com/paper/2405.05841/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 37 references that are in the map.
- First posted on arXiv 9 May 2024.

## Builds on (most cited first, 30 of 32)

- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [Context Autoencoder for Self-Supervised Representation Learning](https://mlsanity.com/paper/2202.03026/) (2022)
- [Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition](https://mlsanity.com/paper/1406.2227/) (2014)
- [Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://mlsanity.com/paper/2103.06495/) (2021)
- [Scene Text Recognition with Permuted Autoregressive Sequence Models](https://mlsanity.com/paper/2207.06966/) (2022)
- [ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection and Recognition -- RRC-MLT-2019](https://mlsanity.com/paper/1907.00945/) (2019)
- [Towards Accurate Scene Text Recognition with Semantic Reasoning Networks](https://mlsanity.com/paper/2003.12294/) (2020)
- [Decoupled Attention Network for Text Recognition](https://mlsanity.com/paper/1912.10205/) (2019)
- [From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network](https://mlsanity.com/paper/2108.09661/) (2021)
- [SVTR: Scene Text Recognition with a Single Visual Model](https://mlsanity.com/paper/2205.00159/) (2022)
- [RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition](https://mlsanity.com/paper/2007.07542/) (2020)
- [Siamese Image Modeling for Self-Supervised Vision Representation Learning](https://mlsanity.com/paper/2206.01204/) (2022)
- [NRTR: A No-Recurrence Sequence-to-Sequence Model For Scene Text Recognition](https://mlsanity.com/paper/1806.00926/) (2018)
- [Sequence-to-Sequence Contrastive Learning for Text Recognition](https://mlsanity.com/paper/2012.10873/) (2020)
- [SCATTER: Selective Context Attentional Scene Text Recognizer](https://mlsanity.com/paper/2003.11288/) (2020)
- [Reading and Writing: Discriminative and Generative Modeling for Self-Supervised Text Recognition](https://mlsanity.com/paper/2207.00193/) (2022)
- [Revisiting Scene Text Recognition: A Data Perspective](https://mlsanity.com/paper/2307.08723/) (2023)
- [Multi-Granularity Prediction for Scene Text Recognition](https://mlsanity.com/paper/2209.03592/) (2022)
- [Multi-modal Text Recognition Networks: Interactive Enhancements between Visual and Semantic Features](https://mlsanity.com/paper/2111.15263/) (2021)
- [MaskOCR: Text Recognition with Masked Encoder-Decoder Pretraining](https://mlsanity.com/paper/2206.00311/) (2022)
- [Self-supervised Character-to-Character Distillation for Text Recognition](https://mlsanity.com/paper/2211.00288/) (2022)
- [Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval](https://mlsanity.com/paper/2312.12155/) (2023)
- [SimAN: Exploring Self-Supervised Representation Learning of Scene Text via Similarity-Aware Normalization](https://mlsanity.com/paper/2203.10492/) (2022)
- [Linguistic More: Taking a Further Step toward Efficient and Accurate Scene Text Recognition](https://mlsanity.com/paper/2305.05140/) (2023)
- [Choose What You Need: Disentangled Representation Learning for Scene Text Recognition, Removal and Editing](https://mlsanity.com/paper/2405.04377/) (2024)
- [LISTER: Neighbor Decoding for Length-Insensitive Scene Text Recognition](https://mlsanity.com/paper/2308.12774/) (2023)

## Built on by (most cited first, 8 of 8)

- [Hybrid-Level Instruction Injection for Video Token Compression in Multi-modal Large Language Models](https://mlsanity.com/paper/2503.16036/) (2025)
- [How Control Information Influences Multilingual Text Image Generation and Editing?](https://mlsanity.com/paper/2407.11502/) (2024)
- [Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognition](https://mlsanity.com/paper/2503.18746/) (2025)
- [Self-Supervised Learning for Text Recognition: A Critical Survey](https://mlsanity.com/paper/2407.19889/) (2024)
- [Exploring the Performance Frontier of Compact Unified Image Generation Models](https://mlsanity.com/paper/2608.20334/) (2026)
- [Towards Spoken Language Understanding via Multi-level Multi-grained Contrastive Learning](https://mlsanity.com/paper/2405.20852/) (2024)
- [Masked Self-Supervised Pre-Training for Text Recognition Transformers on Large-Scale Datasets](https://mlsanity.com/paper/2503.22513/) (2025)
- [Masked Next-Scale Prediction for Self-supervised Scene Text Recognition](https://mlsanity.com/paper/2605.14885/) (2026)

## Abstract (arXiv)

In text recognition, self-supervised pre-training emerges as a good solution to reduce dependence on expansive annotated real data. Previous studies primarily focus on local visual representation by leveraging mask image modeling or sequence contrastive learning. However, they omit modeling the linguistic information in text images, which is crucial for recognizing text. To simultaneously capture local character features and linguistic information in visual space, we propose Symmetric Superimposition Modeling (SSM). The objective of SSM is to reconstruct the direction-specific pixel and feature signals from the symmetrically superimposed input. Specifically, we add the original image with its inverted views to create the symmetrically superimposed inputs. At the pixel level, we reconstruct the original and inverted images to capture character shapes and texture-level linguistic context. At the feature level, we reconstruct the feature of the same original image and inverted image with different augmentations to model the semantic-level linguistic context and the local character discrimination. In our design, we disrupt the character shape and linguistic rules. Consequently, the dual-level reconstruction facilitates understanding character shapes and linguistic information from the perspective of visual texture and feature semantics. Experiments on various text recognition benchmarks demonstrate the effectiveness and generality of SSM, with 4.1% average performance gains and 86.6% new state-of-the-art average word accuracy on Union14M benchmarks. The code is available at https://github.com/FaltingsA/SSM.