# Evaluating Pixel Language Models on Non-Standardized Languages

arXiv: https://arxiv.org/abs/2412.09084 · HTML: https://mlsanity.com/paper/2412.09084/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 29 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [Robust Open-Vocabulary Translation from Visual Text Representations](https://mlsanity.com/paper/2104.08211/) (2021)
- [The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation](https://mlsanity.com/paper/2110.03036/) (2021)
- [From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding](https://mlsanity.com/paper/2105.07316/) (2021)
- [SwissDial: Parallel Multidialectal Corpus of Spoken Swiss German](https://mlsanity.com/paper/2103.11401/) (2021)
- [Text Rendering Strategies for Pixel Language Models](https://mlsanity.com/paper/2311.00522/) (2023)
- [Improving Zero-shot Cross-lingual Transfer between Closely Related Languages by injecting Character-level Noise](https://mlsanity.com/paper/2109.06772/) (2021)
- [Does Manipulating Tokenization Aid Cross-Lingual Transfer? A Study on POS Tagging for Non-Standardized Languages](https://mlsanity.com/paper/2304.10158/) (2023)
- [Findings of the VarDial Evaluation Campaign 2023](https://mlsanity.com/paper/2305.20080/) (2023)
- [Fine-Tuning BERT with Character-Level Noise for Zero-Shot Transfer to Dialects and Closely-Related Languages](https://mlsanity.com/paper/2303.17683/) (2023)
- [BERTwich: Extending BERT's Capabilities to Model Dialectal and Noisy Text](https://mlsanity.com/paper/2311.00116/) (2023)

## Built on by (most cited first, 10 of 10)

- [Multilingual Pretraining for Pixel Language Models](https://mlsanity.com/paper/2505.21265/) (2025)
- [Improving Dialectal Slot and Intent Detection with Auxiliary Tasks: A Multi-Dialectal Bavarian Case Study](https://mlsanity.com/paper/2501.03863/) (2025)
- [Add Noise, Tasks, or Layers? MaiNLP at the VarDial 2025 Shared Task on Norwegian Dialectal Slot and Intent Detection](https://mlsanity.com/paper/2501.03870/) (2025)
- [Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialects](https://mlsanity.com/paper/2510.07890/) (2025)
- [Enhancing Robustness of Autoregressive Language Models against Orthographic Attacks via Pixel-based Approach](https://mlsanity.com/paper/2508.21206/) (2025)
- [Hebrew Diacritics Restoration using Visual Representation](https://mlsanity.com/paper/2510.26521/) (2025)
- [Diacritic Restoration for Low-Resource Indigenous Languages: Case Study with Bribri and Cook Islands M=aori](https://mlsanity.com/paper/2512.19630/) (2025)
- [Meenz bleibt Meenz, but Large Language Models Do Not Speak Its Dialect](https://mlsanity.com/paper/2602.16852/) (2026)
- [Saar-Voice: A Multi-Speaker Saarbr"ucken Dialect Speech Corpus](https://mlsanity.com/paper/2604.11803/) (2026)
- [Resource-Lean Lexicon Induction for German Dialects](https://mlsanity.com/paper/2604.23824/) (2026)

## Abstract (arXiv)

We explore the potential of pixel-based models for transfer learning from standard languages to dialects. These models convert text into images that are divided into patches, enabling a continuous vocabulary representation that proves especially useful for out-of-vocabulary words common in dialectal data. Using German as a case study, we compare the performance of pixel-based models to token-based models across various syntactic and semantic tasks. Our results show that pixel-based models outperform token-based models in part-of-speech tagging, dependency parsing and intent detection for zero-shot dialect evaluation by up to 26 percentage points in some scenarios, though not in Standard German. However, pixel-based models fall short in topic classification. These findings emphasize the potential of pixel-based models for handling dialectal data, though further research should be conducted to assess their effectiveness in various linguistic contexts.