mlsanityData to 7 Oct 2026

Knowledge of Pretrained Language Models on Surface Information of Tokens

arXiv 2402.09808 ↗First posted 15 Feb 2024cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Do pretrained language models have knowledge regarding the surface information of tokens? We examined the surface information stored in word or subword embeddings acquired by pretrained language models from the perspectives of token length, substrings, and token constitution. Additionally, we evaluated the ability of models to generate knowledge regarding token surfaces. We focused on 12 pretrained language models that were mainly trained on English and Japanese corpora. Experimental results demonstrate that pretrained language models have knowledge regarding token length and substrings but not token constitution. Additionally, the results imply that there is a bottleneck on the decoder side in terms of effectively utilizing acquired knowledge.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

32 of 43 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  12. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
20 more
  1. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  2. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  3. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  4. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  5. 2023GPT-NER: Named Entity Recognition via Large Language Modelscited by 158
  6. 2015Character-Aware Neural Language Modelscited by 129
  7. 2017Controllable Abstractive Summarizationcited by 110
  8. 2021Position Information in Transformers: An Overviewcited by 103
  9. 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
  10. 2020Fast WordPiece Tokenizationcited by 85
  11. 2020CharBERT: Character-aware Pre-trained Language Modelcited by 63
  12. 2019Positional Encoding to Control Output Sequence Lengthcited by 56
  13. 2018Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numberscited by 42
  14. 2022What do tokens know about their characters and how do they know it?cited by 36
  15. 2019Pre-trained Language Model for Biomedical Question Answeringcited by 31
  16. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  17. 2023Exploring the Potential of Large Language Models to Generate Formative Programming Feedbackcited by 28
  18. 2022Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Trainingcited by 16
  19. 2020Named Entity Recognition and Relation Extraction using Enhanced Table Filling by Contextualized Representationscited by 5
  20. 2023From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understandingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2025Meeseeks: A Feedback-Driven, Iterative Self-Correction Benchmark evaluating LLMs' Instruction Following Capabilitycited by 3
  2. 2025Spelling-out is not Straightforward: LLMs' Capability of Tokenization from Token to Characterscited by 3
  3. 2024Multi-Modal Multi-Granularity Tokenizer for Chu Bamboo Slip Scriptscited by 1
  4. 2025Decoding-Free Sampling Strategies for LLM Marginalizationcited by 1
  5. 2026How Do Language Models Acquire Character-Level Information?cited by 0