mlsanityData to 14 Oct 2026

Overcoming Vocabulary Constraints with Pixel-level Fallback

arXiv 2504.02122 ↗First posted 2 Apr 2025cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Subword tokenization requires balancing computational efficiency and vocabulary coverage, which often leads to suboptimal performance on languages and scripts not prioritized during training. We propose to augment pretrained language models with a vocabulary-free encoder that generates input embeddings from text rendered as pixels. Through experiments on English-centric language models, we demonstrate that our approach substantially improves machine translation performance and facilitates effective cross-lingual transfer, outperforming tokenizer-based methods. Furthermore, we find that pixel-based representations outperform byte-level approaches and standard vocabulary expansion. Our approach enhances the multilingual capabilities of monolingual language models without extensive retraining and reduces decoding latency via input compression.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 66 of its 86 references that are in the map. First posted on arXiv 2 Apr 2025.

Builds on

66 of 86 references.

References · 66

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  10. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  11. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  12. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
38 more
  1. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  2. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  3. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  4. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  5. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  6. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  7. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  8. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  9. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 302
  10. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 279
  11. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
  12. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 170
  13. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  14. 2020SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddingscited by 156
  15. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
  16. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  17. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 138
  18. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  19. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  20. 2016Discrete Variational Autoencoderscited by 125
  21. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  22. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  23. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  24. 2020Rethinking the Hyperparameters for Fine-tuningcited by 69
  25. 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
  26. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languagescited by 59
  27. 2022Language Modelling with Pixelscited by 55
  28. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 55
  29. 2021Robust Open-Vocabulary Translation from Visual Text Representationscited by 47
  30. 2024JetFormer: An Autoregressive Generative Model of Raw Images and Textcited by 46
  31. 2024Fast Vocabulary Transfer for Language Model Compressioncited by 42
  32. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  33. 2024Zero-Shot Tokenizer Transfercited by 31
  34. 2017Mimicking Word Embeddings using Subword RNNscited by 30
  35. 2021From Machine Translation to Code-Switching: Generating High-Quality Code-Switched Textcited by 29
  36. 2024RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanizationcited by 29
  37. 2022Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languagescited by 28
  38. 2024MaLA-500: Massive Language Adaptation of Large Language Modelscited by 28

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Multilingual Pretraining for Pixel Language Modelscited by 12
  2. 2025See the Text: From Tokenization to Visual Readingcited by 9
  3. 2025zip2zip: Inference-Time Adaptive Tokenization via Online Compressioncited by 5
  4. 2026MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scriptscited by 1
  5. 2025Enhancing Robustness of Autoregressive Language Models against Orthographic Attacks via Pixel-based Approachcited by 0
  6. 2026Pixel-TTS: Image based Text Rendering for Robust Text-to-Speechcited by 0