mlsanityData to 14 Oct 2026

An Information Extraction Study: Take In Mind the Tokenization!

arXiv 2303.15100 ↗First posted 27 Mar 2023cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Current research on the advantages and trade-offs of using characters, instead of tokenized text, as input for deep learning models, has evolved substantially. New token-free models remove the traditional tokenization step; however, their efficiency remains unclear. Moreover, the effect of tokenization is relatively unexplored in sequence tagging tasks. To this end, we investigate the impact of tokenization when extracting information from documents and present a comparative study and analysis of subword-based and character-based models. Specifically, we study Information Extraction (IE) from biomedical texts. The main outcome is twofold: tokenization patterns can introduce inductive bias that results in state-of-the-art performance, and the character-based models produce promising results; thus, transitioning to token-free IE models is feasible.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 23 of its 39 references that are in the map. First posted on arXiv 27 Mar 2023.

Builds on

23 of 39 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2018Deep contextualized word representationscited by 2,290
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  8. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  9. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  10. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  11. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  12. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 362
11 more
  1. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
  2. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 160
  3. 2019Span-based Joint Entity and Relation Extraction with Transformer Pre-trainingcited by 133
  4. 2020Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoderscited by 87
  5. 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
  6. 2018Joint entity recognition and relation extraction as a multi-head selection problemcited by 59
  7. 2021A Partition Filter Network for Joint Entity and Relation Extractioncited by 42
  8. 2021Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokenscited by 27
  9. 2021Subword Pooling Makes a Differencecited by 18
  10. 2021Imposing Relation Structure in Language-Model Embeddings Using Contrastive Learningcited by 9
  11. 2020Let's Stop Incorrect Comparisons in End-to-end Relation Extraction!cited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 2.

6 papers.

  1. 2023InstructUIE: Multi-task Instruction Tuning for Unified Information Extractioncited by 170
  2. 2024Deep Learning based Key Information Extraction from Business Documents: Systematic Literature Reviewcited by 6
  3. 2025Tokenization Disparities as Infrastructure Bias: How Subword Systems Create Inequities in LLM Access and Efficiencycited by 2
  4. 2024Enhancing Biomedical Knowledge Discovery for Diseases: An Open-Source Framework Applied on Rett Syndrome and Alzheimer's Diseasecited by 1
  5. 2024Evaluating the Predictive Features of Person-Centric Knowledge Graph Embeddings: Unfolding Ablation Studiescited by 0
  6. 2025Reduction of Supervision for Biomedical Knowledge Discoverycited by 0