# Word2rate: training and evaluating multiple word embeddings as statistical transitions

arXiv: https://arxiv.org/abs/2104.08173 · HTML: https://mlsanity.com/paper/2104.08173/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 12 references that are in the map.
- First posted on arXiv 16 Apr 2021.

## Builds on (most cited first, 6 of 6)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [Context-Dependent Word Representation for Neural Machine Translation](https://mlsanity.com/paper/1607.00578/) (2016)
- [CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model](https://mlsanity.com/paper/1902.06423/) (2019)

## Built on by (most cited first, 1 of 1)

- [General Cross-Architecture Distillation of Pretrained Language Models into Matrix Embeddings](https://mlsanity.com/paper/2109.08449/) (2021)

## Abstract (arXiv)

Using pretrained word embeddings has been shown to be a very effective way in improving the performance of natural language processing tasks. In fact almost any natural language tasks that can be thought of has been improved by these pretrained embeddings. These tasks range from sentiment analysis, translation, sequence prediction amongst many others. One of the most successful word embeddings is the Word2vec CBOW model proposed by Mikolov trained by the negative sampling technique. Mai et al. modifies this objective to train CMOW embeddings that are sensitive to word order. We used a modified version of the negative sampling objective for our context words, modelling the context embeddings as a Taylor series of rate matrices. We show that different modes of the Taylor series produce different types of embeddings. We compare these embeddings to their similar counterparts like CBOW and CMOW and show that they achieve comparable performance. We also introduce a novel left-right context split objective that improves performance for tasks sensitive to word order. Our Word2rate model is grounded in a statistical foundation using rate matrices while being competitive in variety of language tasks.