mlsanityData to 7 Oct 2026

Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words

arXiv 2101.00403 ↗First posted 2 Jan 2021cs.CLcited by 63

Not on the map. Why?

Abstract (arXiv)

How does the input segmentation of pretrained language models (PLMs) affect their interpretations of complex words? We present the first study investigating this question, taking BERT as the example PLM and focusing on its semantic representations of English derivatives. We show that PLMs can be interpreted as serial dual-route models, i.e., the meanings of complex words are either stored or else need to be computed from the subwords, which implies that maximally meaningful input tokens should allow for the best generalization on new words. This hypothesis is confirmed by a series of semantic probing tasks on which DelBERT (Derivation leveraging BERT), a model with derivational input segmentation, substantially outperforms BERT with WordPiece segmentation. Our results suggest that the generalization capabilities of PLMs could be further improved if a morphologically-informed vocabulary of input tokens were used.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 37 of its 113 references that are in the map. First posted on arXiv 2 Jan 2021.

Builds on

37 of 113 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  11. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  12. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
25 more
  1. 2019CamemBERT: a Tasty French Language Modelcited by 368
  2. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  3. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  4. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  5. 2019Assessing BERT's Syntactic Abilitiescited by 191
  6. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  7. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
  8. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  9. 2015Character-Aware Neural Language Modelscited by 129
  10. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
  11. 2020Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!cited by 73
  12. 2020CharBERT: Character-aware Pre-trained Language Modelcited by 63
  13. 2018Probabilistic FastText for Multi-Sense Word Embeddingscited by 25
  14. 2015Overcoming Language Variation in Sentiment Analysis with Social Attentioncited by 23
  15. 2020Mind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encodingcited by 22
  16. 2020A Systematic Analysis of Morphological Content in BERT Models for Multiple Languagescited by 20
  17. 2019Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluationcited by 14
  18. 2014Compositional Morphology for Word Representations and Language Modellingcited by 11
  19. 2020Will it Unblend?cited by 11
  20. 2016Morphological Priors for Probabilistic Neural Word Embeddingscited by 6
  21. 2017Joint Semantic Synthesis and Morphological Analysis of the Derived Wordcited by 6
  22. 2017Paradigm Completion for Derivational Morphologycited by 5
  23. 2020Enhancing deep neural networks with morphological informationcited by 5
  24. 2017Context-Aware Prediction of Derivational Word-formscited by 4
  25. 2018Incorporating Subword Information into Matrix Factorization Word Embeddingscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 9; 2023: 13; 2024: 22; 2025: 11; 2026: 7.

63 papers.

Sort:

By area:

  1. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  2. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  3. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  4. 2024Tokenization Is More Than Compressioncited by 87
  5. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  6. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  7. 2025SuperBPE: Space Travel for Language Modelscited by 58
  8. 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 53
  9. 2024Greed is All You Need: An Evaluation of Tokenizer Inference Methodscited by 34
  10. 2024Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenizationcited by 32
  11. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  12. 2024BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Trainingcited by 30
38 more
  1. 2025Evaluating Morphological Alignment of Tokenizers in 70 Languagescited by 22
  2. 2024The Foundations of Tokenization: Statistical and Computational Concernscited by 20
  3. 2022Improving Tokenisation by Alternative Treatment of Spacescited by 19
  4. 2024Evaluating Morphological Compositional Generalization in Large Language Modelscited by 16
  5. 2025Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizationscited by 16
  6. 2024Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5cited by 14
  7. 2023Effects of sub-word segmentation on performance of transformer language modelscited by 12
  8. 2022Incorporating Context into Subword Vocabulariescited by 11
  9. 2024DrBenchmark: A Large Language Understanding Evaluation Benchmark for French Biomedical Domaincited by 11
  10. 2024Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNAcited by 11
  11. 2025Beyond Text Compression: Evaluating Tokenizers Across Scalescited by 11
  12. 2023Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Modelcited by 9
  13. 2023Biomedical Language Models are Robust to Sub-optimal Tokenizationcited by 8
  14. 2024Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challengecited by 8
  15. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
  16. 2022Linguistically inspired roadmap for building biologically reliable protein language modelscited by 7
  17. 2023CompoundPiece: Evaluating and Improving Decompounding Performance of Language Modelscited by 7
  18. 2024Time Machine GPTcited by 7
  19. 2025Tokens with Meaning: A Hybrid Tokenization Approach for Turkishcited by 7
  20. 2023Can Pretrained Language Models Derive Correct Semantics from Corrupt Subwords under Noise?cited by 6
  21. 2023Sentence Embedding Models for Ancient Greek Using Multilingual Knowledge Distillationcited by 6
  22. 2023BERTwich: Extending BERT's Capabilities to Model Dialectal and Noisy Textcited by 6
  23. 2024The Impact of Word Splitting on the Semantic Content of Contextualized Word Representationscited by 6
  24. 2024HeSum: a Novel Dataset for Abstractive Text Summarization in Hebrewcited by 6
  25. 2023Explicit Morphological Knowledge Improves Pre-training of Language Models for Hebrewcited by 5
  26. 2023Impact of Tokenization on LLaMa Russian Adaptationcited by 5
  27. 2024Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreementcited by 5
  28. 2024Verbing Weirds Language (Models): Evaluation of English Zero-Derivation in Five LLMscited by 5
  29. 2022Morphological Processing of Low-Resource Languages: Where We Are and What's Nextcited by 4
  30. 2024A Morphology-Based Investigation of Positional Encodingscited by 4
  31. 2022What do Large Language Models Learn beyond Language?cited by 3
  32. 2022Morphology Without Borders: Clause-Level Morphologycited by 2
  33. 2022How Adults Understand What Young Children Saycited by 2
  34. 2024Word Boundary Information Isn't Useful for Encoder Language Modelscited by 2
  35. 2025Incorporating Domain Knowledge into Materials Tokenizationcited by 2
  36. 2025KinyaColBERT: A Lexically Grounded Retrieval Model for Low-Resource Retrieval-Augmented Generationcited by 2
  37. 2026Word Recovery in Large Language Models Enables Character-Level Tokenization Robustnesscited by 2
  38. 2026On the Proper Treatment of Units in Surprisal Theorycited by 2

50 most cited of 63 All 63 (JSON)