# One Image, No Tokens: A Controlled Study of Glyph-Based Chinese Language Modeling

arXiv: https://arxiv.org/abs/2607.03994 · HTML: https://mlsanity.com/paper/2607.03994/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 4 Jul 2026.

## Builds on (most cited first, 13 of 13)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information](https://mlsanity.com/paper/2106.16038/) (2021)
- [Language Modelling with Pixels](https://mlsanity.com/paper/2207.06991/) (2022)
- [CLIPPO: Image-and-Language Understanding from Pixels Only](https://mlsanity.com/paper/2212.08045/) (2022)
- [Robust Open-Vocabulary Translation from Visual Text Representations](https://mlsanity.com/paper/2104.08211/) (2021)
- [Glyce: Glyph-vectors for Chinese Character Representations](https://mlsanity.com/paper/1901.10125/) (2019)
- [Multilingual Pretraining for Pixel Language Models](https://mlsanity.com/paper/2505.21265/) (2025)
- [The Impact of Visual Information in Chinese Characters: Evaluating Large Models' Ability to Recognize and Utilize Radicals](https://mlsanity.com/paper/2410.09013/) (2024)
- [LogogramNLP: Comparing Visual and Textual Representations of Ancient Logographic Writing Systems for NLP](https://mlsanity.com/paper/2408.04628/) (2024)
- [Hot-Start Chinese Language Modeling:Visual Glyphs Accelerate Sample-Efficient Learning](https://mlsanity.com/paper/2601.09566/) (2026)

## Abstract (arXiv)

Modern language models generally represent text as sequences of discrete token embeddings, an assumption deeply rooted in current practice but rarely questioned. We challenge this representation, especially for Chinese, by replacing index-based token embeddings entirely with a single rasterized image of the character sequence, processed by a vision encoder composed of a shared ResNet and a shallow Vision Transformer. To isolate the role of input representation, we construct a dual-branch controlled framework in which both a Vision-based model and an index-based baseline share an identical decoder backbone, training objective, optimizer, and data curriculum. Any performance difference is therefore attributable to the input modality only. Across all tested decoder backbones, the Vision-based model consistently outperforms the baseline, reaching a peak accuracy of 0.429 versus 0.355 for the index-based baseline---a 21% relative improvement---while converging in about half the number of training epochs. The advantage emerges especially within the first five epochs (under 21% of total data) and persists under moderate character corruption: the corrupted Vision model matches the clean index-based baseline. Ablation studies reveal that the advantage requires both spatially coherent input and a ViT encoder with 2D positional encodings. A cross-script comparison on English shows the advantage does not transfer directly to alphabetic writing systems, suggesting that the uniform visual density and radical structure of Chinese characters are enabling conditions. These findings suggest that transformers are more modality-agnostic than commonly assumed, and that discrete tokenization is not a fundamental requirement for Chinese language modeling.