mlsanityData to 7 Oct 2026

CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation

arXiv 2103.06874 ↗First posted 11 Mar 2021cs.CL cs.LGcited by 229

Not on the map. Why?

Abstract (arXiv)

Pipelined NLP systems have largely been superseded by end-to-end neural modeling, yet nearly all commonly-used models still require an explicit tokenization step. While recent tokenization approaches based on data-derived subword lexicons are less brittle than manually engineered tokenizers, these techniques are not equally suited to all languages, and the use of any fixed vocabulary may limit a model's ability to adapt. In this paper, we present CANINE, a neural encoder that operates directly on character sequences, without explicit tokenization or vocabulary, and a pre-training strategy that operates either directly on characters or optionally uses subwords as a soft inductive bias. To use its finer-grained input effectively and efficiently, CANINE combines downsampling, which reduces the input sequence length, with a deep transformer stack, which encodes context. CANINE outperforms a comparable mBERT model by 2.8 F1 on TyDi QA, a challenging multilingual benchmark, despite having 28% fewer model parameters.

Key facts. Cited by 229 papers in the map (data through 7 Oct 2026). Builds on 61 of its 84 references that are in the map. First posted on arXiv 11 Mar 2021.

Builds on

61 of 84 references.

References · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2018Deep contextualized word representationscited by 2,285
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  12. 2020Rethinking Attention with Performerscited by 1,584
38 more
  1. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  2. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  3. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  4. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  5. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  6. 2019How multilingual is Multilingual BERT?cited by 642
  7. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  8. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  9. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  10. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  11. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  12. 2021Random Feature Attentioncited by 350
  13. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  14. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  15. 2019Combating Adversarial Misspellings with Robust Word Recognitioncited by 162
  16. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  17. 2017Learning to Generate Reviews and Discovering Sentimentcited by 147
  18. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  19. 2015Character-Aware Neural Language Modelscited by 129
  20. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  21. 2016Hierarchical Multiscale Recurrent Neural Networkscited by 102
  22. 2020RobBERT: a Dutch RoBERTa-based Language Modelcited by 88
  23. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  24. 2020ConvBERT: Improving BERT with Span-based Dynamic Convolutioncited by 88
  25. 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
  26. 2020Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERTcited by 69
  27. 2016Fully Character-Level Neural Machine Translation without Explicit Segmentationcited by 55
  28. 2017Hash Embeddings for Efficient Word Representationscited by 54
  29. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  30. 2018Neural Cross-Lingual Named Entity Recognition with Minimal Resourcescited by 44
  31. 2020English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Toocited by 39
  32. 2017Mimicking Word Embeddings using Subword RNNscited by 30
  33. 2015Multilingual Language Processing From Bytescited by 26
  34. 2020AMBERT: A Pre-trained Language Model with Multi-Grained Tokenizationcited by 22
  35. 2019Bridging the Gap for Tokenizer-Free Language Modelscited by 21
  36. 2016Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Modelscited by 18
  37. 2016Charagram: Embedding Words and Sentences via Character n-gramscited by 14
  38. 2016Character-Level Language Modeling with Hierarchical Recurrent Neural Networkscited by 13

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 33; 2023: 37; 2024: 49; 2025: 40; 2026: 51.

229 papers.

Sort:

By area:

  1. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  2. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  3. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  4. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
  5. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  6. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  7. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  8. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  9. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  10. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  11. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 99
  12. 2022Character-Aware Models Improve Visual Text Renderingcited by 90
38 more
  1. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  2. 2022Efficient Transformers with Dynamic Token Poolingcited by 75
  3. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  4. 2024Anisotropy Is Inherent to Self-Attention in Transformerscited by 62
  5. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  6. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  7. 2024MambaByte: Token-free Selective State Space Modelcited by 58
  8. 2025SuperBPE: Space Travel for Language Modelscited by 58
  9. 2022Language Modelling with Pixelscited by 53
  10. 2021Robust Open-Vocabulary Translation from Visual Text Representationscited by 47
  11. 2025Over-Tokenized Transformer: Vocabulary is Generally Worth Scalingcited by 42
  12. 2024Segment Any Text: A Universal Approach for Robust, Efficient and Adaptable Sentence Segmentationcited by 41
  13. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  14. 2022What do tokens know about their characters and how do they know it?cited by 36
  15. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 31
  16. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  17. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  18. 2024Zero-Shot Tokenizer Transfercited by 30
  19. 2021You should evaluate your language model on marginal likelihood over tokenisationscited by 29
  20. 2021Why don't people use character-level machine translation?cited by 27
  21. 2022KinyaBERT: a Morphology-aware Kinyarwanda Language Modelcited by 27
  22. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  23. 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 27
  24. 2022ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Modelscited by 25
  25. 2025MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologiescited by 23
  26. 2023Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfercited by 22
  27. 2023Text Rendering Strategies for Pixel Language Modelscited by 22
  28. 2021Specializing Multilingual Language Models: An Empirical Studycited by 21
  29. 2022How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training?cited by 20
  30. 2024The Foundations of Tokenization: Statistical and Computational Concernscited by 20
  31. 2024Egalitarian Language Representation in Language Models: It All Begins with Tokenizerscited by 20
  32. 2024Tokenization Falling Short: On Subword Robustness in Large Language Modelscited by 18
  33. 2025Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenizationcited by 18
  34. 2021Translate & Fill: Improving Zero-Shot Multilingual Semantic Parsing with Synthetic Datacited by 17
  35. 2023Where's the Point? Self-Supervised Multilingual Punctuation-Agnostic Sentence Segmentationcited by 17
  36. 2023Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPTcited by 17
  37. 2022Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Trainingcited by 16
  38. 2024Training LLMs over Neurally Compressed Textcited by 16

50 most cited of 229 All 229 (JSON)