# Does Visual Rendering Bypass Tokenization? Investigating Script-Tokenizer Misalignment in Pixel-Based Language Models

arXiv: https://arxiv.org/abs/2602.06973 · HTML: https://mlsanity.com/paper/2602.06973/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 12 Jan 2026.

## Builds on (most cited first, 14 of 14)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages](https://mlsanity.com/paper/2305.17179/) (2023)
- [Language Modelling with Pixels](https://mlsanity.com/paper/2207.06991/) (2022)
- [Text Rendering Strategies for Pixel Language Models](https://mlsanity.com/paper/2311.00522/) (2023)
- [Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization](https://mlsanity.com/paper/2508.04796/) (2025)
- [MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages](https://mlsanity.com/paper/2506.19468/) (2025)
- [A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Models](https://mlsanity.com/paper/2210.07111/) (2022)
- [Multilingual Pretraining for Pixel Language Models](https://mlsanity.com/paper/2505.21265/) (2025)
- [Autoregressive Pre-Training on Pixels and Texts](https://mlsanity.com/paper/2404.10710/) (2024)
- [NusaAksara: A Multimodal and Multilingual Benchmark for Preserving Indonesian Indigenous Scripts](https://mlsanity.com/paper/2502.18148/) (2025)

## Abstract (arXiv)

While pixel-based language modeling aims to bypass the sub-word tokenization bottleneck by rendering text as images, recent multimodal variants such as DualGPT reintroduce text tokenizers to improve autoregressive performance. We investigate a fundamental question, does visual rendering truly decouple a model from tokenization constraints? Focusing on four Indonesian low-resource local languages that have their own non-Latin scripts (i.e., Javanese, Balinese, Sundanese, and Lampungnese), we evaluate the impact of script-tokenizer alignment within the DualGPT architecture. Our results show that, despite visual rendering, reintegrating a text tokenizer into the architecture reintroduces the same issue that pixel-based language modeling aims to resolve, which is the tokenizer misalignment problem. Despite having lower OOV and fertility rates, we show that the Llama 2 tokenizer performs significantly worse than a custom tokenizer, with improvements of up to 30.15 chrF++. Our findings serve as a warning for future multimodal variants, as text tokenizers remain a significant barrier to equitable models.