# Zero-Shot Chinese Character Recognition with Hierarchical Multi-Granularity Image-Text Aligning

arXiv: https://arxiv.org/abs/2505.24837 · HTML: https://mlsanity.com/paper/2505.24837/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Learning Deconvolution Network for Semantic Segmentation](https://mlsanity.com/paper/1505.04366/) (2015)
- [Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillation](https://mlsanity.com/paper/2203.10900/) (2022)
- [Zero-Shot Chinese Character Recognition with Stroke-Level Decomposition](https://mlsanity.com/paper/2106.11613/) (2021)
- [Chinese Text Recognition with A Pre-Trained CLIP-Like Model Through Image-IDS Aligning](https://mlsanity.com/paper/2309.01083/) (2023)
- [DenseRAN for Offline Handwritten Chinese Character Recognition](https://mlsanity.com/paper/1808.04134/) (2018)
- [Online and Offline Handwritten Chinese Character Recognition: A Comprehensive Study and New Benchmark](https://mlsanity.com/paper/1606.05763/) (2016)
- [Multi-Column Deep Neural Networks for Offline Handwritten Chinese Character Classification](https://mlsanity.com/paper/1309.0261/) (2013)
- [DropSample: A New Training Method to Enhance Deep Convolutional Neural Networks for Large-Scale Unconstrained Handwritten Chinese Character Recognition](https://mlsanity.com/paper/1505.05354/) (2015)
- [RZCR: Zero-shot Character Recognition via Radical-based Reasoning](https://mlsanity.com/paper/2207.05842/) (2022)
- [Chinese Character Recognition with Radical-Structured Stroke Trees](https://mlsanity.com/paper/2211.13518/) (2022)
- [Template-Instance Loss for Offline Handwritten Chinese Character Recognition](https://mlsanity.com/paper/1910.05545/) (2019)
- [STAR: Zero-Shot Chinese Character Recognition with Stroke- and Radical-Level Decompositions](https://mlsanity.com/paper/2210.08490/) (2022)
- [Chinese Text in the Wild](https://mlsanity.com/paper/1803.00085/) (2018)
- [Stroke-Based Autoencoders: Self-Supervised Learners for Efficient Zero-Shot Chinese Character Recognition](https://mlsanity.com/paper/2207.08191/) (2022)
- [Orientation-Independent Chinese Text Recognition in Scene Images](https://mlsanity.com/paper/2309.01081/) (2023)

## Built on by (most cited first, 1 of 1)

- [Decompose Radicals, Then Reward: Fine-Grained Inspection for Accurate Chinese Text Rendering](https://mlsanity.com/paper/2609.37569/) (2026)

## Abstract (arXiv)

Chinese Character Recognition (CCR) is a fundamental technology for intelligent document processing. Unlike Latin characters, Chinese characters exhibit unique spatial structures and compositional rules, allowing for the use of fine-grained semantic information in representation. However, existing approaches are usually based on auto-regressive as well as edit distance post-process and typically rely on a single-level character representation. In this paper, we propose a Hierarchical Multi-Granularity Image-Text Aligning (Hi-GITA) framework based on a contrastive paradigm. To leverage the abundant fine-grained semantic information of Chinese characters, we propose multi-granularity encoders on both image and text sides. Specifically, the Image Multi-Granularity Encoder extracts hierarchical image representations from character images, capturing semantic cues from localized strokes to holistic structures. The Text Multi-Granularity Encoder extracts stroke and radical sequence representations at different levels of granularity. To better capture the relationships between strokes and radicals, we introduce Multi-Granularity Fusion Modules on the image and text sides, respectively. Furthermore, to effectively bridge the two modalities, we further introduce a Fine-Grained Decoupled Image-Text Contrastive loss, which aligns image and text representations across multiple granularities. Extensive experiments demonstrate that our proposed Hi-GITA significantly outperforms existing zero-shot CCR methods. For instance, it brings about 20% accuracy improvement in handwritten character and radical zero-shot settings. Code and models will be released soon.