mlsanityData to 7 Oct 2026

Integrating Approaches to Word Representation

arXiv 2109.04876 ↗First posted 10 Sep 2021cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

The problem of representing the atomic elements of language in modern neural learning systems is one of the central challenges of the field of natural language processing. I present a survey of the distributional, compositional, and relational approaches to addressing this task, and discuss various means of integrating them into systems, with special emphasis on the word level and the out-of-vocabulary phenomenon.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 46 of its 93 references that are in the map. First posted on arXiv 10 Sep 2021.

Builds on

46 of 93 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  8. 2016node2vec: Scalable Feature Learning for Networkscited by 1,913
  9. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
  10. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  11. 2014Embedding Entities and Relations for Learning and Inference in Knowledge Basescited by 929
  12. 2017Convolutional 2D Knowledge Graph Embeddingscited by 729
34 more
  1. 2017Poincar'e Embeddings for Learning Hierarchical Representationscited by 674
  2. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  3. 2016Neural Architectures for Named Entity Recognitioncited by 548
  4. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  5. 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 262
  6. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  7. 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 170
  8. 2015Character-Aware Neural Language Modelscited by 129
  9. 2019Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphscited by 125
  10. 2016Hierarchical Multiscale Recurrent Neural Networkscited by 102
  11. 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
  12. 2020CharBERT: Character-aware Pre-trained Language Modelcited by 63
  13. 2015Traversing Knowledge Graphs in Vector Spacecited by 51
  14. 2018Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledgecited by 45
  15. 2016What to do about non-standard (or non-canonical) language in NLPcited by 35
  16. 2016Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Losscited by 30
  17. 2017Mimicking Word Embeddings using Subword RNNscited by 30
  18. 2015Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representationcited by 28
  19. 2016ProjE: Embedding Projection for Knowledge Graph Completioncited by 26
  20. 2017How Robust Are Character-Based Word Embeddings in Tagging and MT Against Wrod Scramlbing or Randdm Nouse?cited by 23
  21. 2020Mind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encodingcited by 22
  22. 2016Charagram: Embedding Words and Sentences via Character n-gramscited by 14
  23. 2016Hypernyms under Siege: Linguistically-motivated Artillery for Hypernymy Detectioncited by 13
  24. 2014Compositional Morphology for Word Representations and Language Modellingcited by 11
  25. 2020Will it Unblend?cited by 11
  26. 2020NYTWIT: A Dataset of Novel Words in the New York Timescited by 10
  27. 2018Generalizing Word Embeddings using Bag of Subwordscited by 9
  28. 2016Morphological Priors for Probabilistic Neural Word Embeddingscited by 6
  29. 2017The Interplay of Semantics and Morphology in Word Embeddingscited by 6
  30. 2018Predicting Semantic Relations using Global Graph Propertiescited by 4
  31. 2021Learning to Look Inside: Augmenting Token-Based Encoders with Character-Level Informationcited by 4
  32. 2017Context encoders as a simple but powerful extension of word2veccited by 3
  33. 2015Mapping Unseen Words to Task-Trained Embedding Spacescited by 1
  34. 2019Attending Form and Context to Generate Specialized Out-of-VocabularyWords Representationscited by 1

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3.

4 papers.

  1. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  2. 2022What do tokens know about their characters and how do they know it?cited by 36
  3. 2022Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Trainingcited by 16
  4. 2022Linguistically inspired roadmap for building biologically reliable protein language modelscited by 7