# Multilingual Language Processing From Bytes

arXiv: https://arxiv.org/abs/1512.00103 · HTML: https://mlsanity.com/paper/1512.00103/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2015.

## Built on by (most cited first, 26 of 26)

- [FAST: Efficient Action Tokenization for Vision-Language-Action Models](https://mlsanity.com/paper/2501.09747/) (2025)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [A Unified Generative Framework for Various NER Subtasks](https://mlsanity.com/paper/2106.01223/) (2021)
- [Unified Named Entity Recognition as Word-Word Relation Classification](https://mlsanity.com/paper/2112.10070/) (2021)
- [One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia](https://mlsanity.com/paper/2203.13357/) (2022)
- [MrT5: Dynamic Token Merging for Efficient Byte-level Language Models](https://mlsanity.com/paper/2410.20771/) (2024)
- [MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies](https://mlsanity.com/paper/2502.00894/) (2025)
- [Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfer](https://mlsanity.com/paper/2305.14280/) (2023)
- [Training Multilingual Pre-trained Language Model with Byte-level Subwords](https://mlsanity.com/paper/2101.09469/) (2021)
- [Recent Trends in Named Entity Recognition (NER)](https://mlsanity.com/paper/2101.11420/) (2021)
- [TOE: A Grid-Tagging Discontinuous NER Model Enhanced by Embedding Tag/Word Relations and More Fine-Grained Tags](https://mlsanity.com/paper/2211.00684/) (2022)
- [What is the best recipe for character-level encoder-only modelling?](https://mlsanity.com/paper/2305.05461/) (2023)
- [Family of Origin and Family of Choice: Massively Parallel Lexiconized Iterative Pretraining for Severely Low Resource Machine Translation](https://mlsanity.com/paper/2104.05848/) (2021)
- [Active Learning for Massively Parallel Translation of Constrained Text into Low Resource Languages](https://mlsanity.com/paper/2108.07127/) (2021)
- [Bilingual End-to-End ASR with Byte-Level Subwords](https://mlsanity.com/paper/2205.00485/) (2022)
- [A multi-level interpretable sleep stage scoring system by infusing experts' knowledge into a deep network architecture](https://mlsanity.com/paper/2207.04585/) (2022)
- [Towards A Multi-agent System for Online Hate Speech Detection](https://mlsanity.com/paper/2105.01129/) (2021)
- [Language-universal phonetic encoder for low-resource speech recognition](https://mlsanity.com/paper/2305.11576/) (2023)
- [Optimizing Byte-level Representation for End-to-end ASR](https://mlsanity.com/paper/2406.09676/) (2024)
- [Label Alignment and Reassignment with Generalist Large Language Model for Enhanced Cross-Domain Named Entity Recognition](https://mlsanity.com/paper/2407.17344/) (2024)
- [Dutch Named Entity Recognition and De-identification Methods for the Human Resource Domain](https://mlsanity.com/paper/2106.02287/) (2021)
- [byteSteady: Fast Classification Using Byte-Level n-Gram Embeddings](https://mlsanity.com/paper/2106.13302/) (2021)
- [ML Based Lineage in Databases](https://mlsanity.com/paper/2109.06339/) (2021)
- [MoVoC: Morphology-Aware Subword Construction for Geez Script Languages](https://mlsanity.com/paper/2509.08812/) (2025)
- [Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models](https://mlsanity.com/paper/2606.14122/) (2026)

## Abstract (arXiv)

We describe an LSTM-based model which we call Byte-to-Span (BTS) that reads text as bytes and outputs span annotations of the form [start, length, label] where start positions, lengths, and labels are separate entries in our vocabulary. Because we operate directly on unicode bytes rather than language-specific words or characters, we can analyze text in many languages with a single model. Due to the small vocabulary size, these multilingual models are very compact, but produce results similar to or better than the state-of- the-art in Part-of-Speech tagging and Named Entity Recognition that use only the provided training datasets (no external data sources). Our models are learning "from scratch" in that they do not rely on any elements of the standard pipeline in Natural Language Processing (including tokenization), and thus can run in standalone fashion on raw text.