# On the Impact of Knowledge-based Linguistic Annotations in the Quality of Scientific Embeddings

arXiv: https://arxiv.org/abs/2104.06200 · HTML: https://mlsanity.com/paper/2104.06200/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 53 references that are in the map.
- First posted on arXiv 13 Apr 2021.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [A Survey on Knowledge Graphs: Representation, Acquisition and Applications](https://mlsanity.com/paper/2002.00388/) (2020)
- [Holographic Embeddings of Knowledge Graphs](https://mlsanity.com/paper/1510.04935/) (2015)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [Construction of the Literature Graph in Semantic Scholar](https://mlsanity.com/paper/1805.02262/) (2018)
- [BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages](https://mlsanity.com/paper/1710.02187/) (2017)
- [SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity](https://mlsanity.com/paper/1608.00869/) (2016)
- [Embedding Words and Senses Together via Joint Knowledge-Enhanced Training](https://mlsanity.com/paper/1612.02703/) (2016)
- [Swivel: Improving Embeddings by Noticing What's Missing](https://mlsanity.com/paper/1602.02215/) (2016)

## Abstract (arXiv)

In essence, embedding algorithms work by optimizing the distance between a word and its usual context in order to generate an embedding space that encodes the distributional representation of words. In addition to single words or word pieces, other features which result from the linguistic analysis of text, including lexical, grammatical and semantic information, can be used to improve the quality of embedding spaces. However, until now we did not have a precise understanding of the impact that such individual annotations and their possible combinations may have in the quality of the embeddings. In this paper, we conduct a comprehensive study on the use of explicit linguistic annotations to generate embeddings from a scientific corpus and quantify their impact in the resulting representations. Our results show how the effect of such annotations in the embeddings varies depending on the evaluation task. In general, we observe that learning embeddings using linguistic annotations contributes to achieve better evaluation results.