mlsanityData to 7 Oct 2026

Language Modelling with Pixels

arXiv 2207.06991 ↗First posted 14 Jul 2022cs.CL cs.AI cs.CVcited by 53

Not on the map. Why?

Abstract (arXiv)

Language models are defined over a finite set of inputs, which creates a vocabulary bottleneck when we attempt to scale the number of supported languages. Tackling this bottleneck results in a trade-off between what can be represented in the embedding matrix and computational issues in the output layer. This paper introduces PIXEL, the Pixel-based Encoder of Language, which suffers from neither of these issues. PIXEL is a pretrained language model that renders text as images, making it possible to transfer representations across languages based on orthographic similarity or the co-activation of pixels. PIXEL is trained to reconstruct the pixels of masked patches instead of predicting a distribution over tokens. We pretrain the 86M parameter PIXEL model on the same English data as BERT and evaluate on syntactic and semantic tasks in typologically diverse languages, including various non-Latin scripts. We find that PIXEL substantially outperforms BERT on syntactic and semantic processing tasks on scripts that are not found in the pretraining data, but PIXEL is slightly weaker than BERT when working with Latin scripts. Furthermore, we find that PIXEL is more robust than BERT to orthographic attacks and linguistic code-switching, further confirming the benefits of modelling language with pixels.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 105 of its 145 references that are in the map. First posted on arXiv 14 Jul 2022.

Builds on

105 of 145 references.

References · 105

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2016Layer Normalizationcited by 5,579
  12. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
38 more
  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  11. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  12. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  13. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  14. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  15. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  16. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  17. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  18. 2019How multilingual is Multilingual BERT?cited by 642
  19. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  20. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  21. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  22. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  23. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  24. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  25. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 359
  26. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  27. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  28. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  29. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 269
  30. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  31. 2019Fixing the train-test resolution discrepancycited by 209
  32. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  33. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  34. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  35. 2019Combating Adversarial Misspellings with Robust Word Recognitioncited by 162
  36. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  37. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  38. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131

50 most cited of 105 All 105 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 12; 2024: 10; 2025: 13; 2026: 13.

53 papers.

Sort:

By area:

  1. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  2. 2023Challenges and Applications of Large Language Modelscited by 251
  3. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  4. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  5. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  6. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 53
  7. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  8. 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 27
  9. 2025Vision-centric Token Compression in Large Language Modelcited by 21
  10. 2024Tur[k]ingBench: A Challenge Benchmark for Web Agentscited by 18
  11. 2024Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learningcited by 17
  12. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 17
38 more
  1. 2024LLaVA-Read: Enhancing Reading Ability of Multimodal Language Modelscited by 12
  2. 2022Fact-Saboteurs: A Taxonomy of Evidence Manipulation Attacks against Fact-Verification Systemscited by 11
  3. 2022Incorporating Context into Subword Vocabulariescited by 11
  4. 2022Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?cited by 10
  5. 2023Efficient OCR for Building a Diverse Digital Historycited by 10
  6. 2024From Text to Pixel: Advancing Long-Context Understanding in MLLMscited by 9
  7. 2025See the Text: From Tokenization to Visual Readingcited by 9
  8. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
  9. 2023Byte-Level Grammatical Error Correction Using Synthetic and Curated Corporacited by 7
  10. 2023Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels?cited by 7
  11. 2024Pixel Sentence Representation Learningcited by 7
  12. 2025Overcoming Vocabulary Constraints with Pixel-level Fallbackcited by 6
  13. 2026Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMscited by 6
  14. 2023Tokenization with Factorized Subword Encodingcited by 4
  15. 2023To token or not to token: A Comparative Study of Text Representations for Cross-Lingual Transfercited by 4
  16. 2025Beyond Words: Advancing Long-Text Image Generation via Multimodal Autoregressive Modelscited by 4
  17. 2025Misspellings in Natural Language Processing: A surveycited by 3
  18. 2025PixelWorld: How Far Are We from Perceiving Everything as Pixels?cited by 3
  19. 2025LF2AR: Accounting for Layerwise Dynamics to Improve Multimodal Adaptation of Language Modelscited by 3
  20. 2025UTF-8 Plumbing: Byte-level Tokenizers Unavoidably Enable LLMs to Generate Ill-formed UTF-8cited by 3
  21. 2026Proxy Compression for Language Modelingcited by 3
  22. 2024Chameleon: Images Are What You Need For Multimodal Learning Robust To Missing Modalitiescited by 2
  23. 2024Everything is a Video: Unifying Modalities through Next-Frame Predictioncited by 2
  24. 2026LensVLM: Selective Context Expansion for Compressed Visual Representation of Textcited by 2
  25. 2026Decoupling semantics from vision: A framework for faithful visual-text compression evaluationcited by 2
  26. 2023GlyphDiffusion: Text Generation as Image Generationcited by 1
  27. 2025Coarse-to-Fine Personalized LLM Impressions for Streamlined Radiology Reportscited by 1
  28. 2025Exploring a Unified Vision-Centric Contrastive Alternatives on Multi-Modal Web Documentscited by 1
  29. 2026Hot-Start Chinese Language Modeling:Visual Glyphs Accelerate Sample-Efficient Learningcited by 1
  30. 2026FlowInOne:Unifying Multimodal Generation as Image-in, Image-out Flow Matchingcited by 1
  31. 2026MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scriptscited by 1
  32. 2026Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Modelscited by 1
  33. 2023OneCAD: One Classifier for All image Datasets using multimodal learningcited by 0
  34. 2023Multilingual Text Representationcited by 0
  35. 2024MOoSE: Multi-Orientation Sharing Experts for Open-set Scene Text Recognitioncited by 0
  36. 2025Joint Low-level and High-level Textual Representation Learning with Multiple Masking Strategiescited by 0
  37. 2026Enabling Stroke-Level Structural Analysis of Hieroglyphic Scripts without Language-Specific Priorscited by 0
  38. 2026Does Visual Rendering Bypass Tokenization? Investigating Script-Tokenizer Misalignment in Pixel-Based Language Modelscited by 0

50 most cited of 53 All 53 (JSON)