# MorphTE: Injecting Morphology in Tensorized Embeddings

arXiv: https://arxiv.org/abs/2210.15379 · HTML: https://mlsanity.com/paper/2210.15379/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 27 Oct 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models](https://mlsanity.com/paper/2101.11714/) (2021)
- [Compressing Word Embeddings via Deep Compositional Code Learning](https://mlsanity.com/paper/1711.01068/) (2017)
- [GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking](https://mlsanity.com/paper/1806.06950/) (2018)
- [Simple and Effective Text Matching with Richer Alignment Features](https://mlsanity.com/paper/1908.00300/) (2019)
- [Online Embedding Compression for Text Classification using Low Rank Matrix Factorization](https://mlsanity.com/paper/1811.00641/) (2018)
- [KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation](https://mlsanity.com/paper/2109.06243/) (2021)
- [word2ket: Space-efficient Word Embeddings inspired by Quantum Entanglement](https://mlsanity.com/paper/1911.04975/) (2019)
- [Near-lossless Binarization of Word Embeddings](https://mlsanity.com/paper/1803.09065/) (2018)
- [Compositional Morphology for Word Representations and Language Modelling](https://mlsanity.com/paper/1405.4273/) (2014)
- [Morphological Word Embeddings](https://mlsanity.com/paper/1907.02423/) (2019)
- [Morphological Priors for Probabilistic Neural Word Embeddings](https://mlsanity.com/paper/1608.01056/) (2016)
- [The Interplay of Semantics and Morphology in Word Embeddings](https://mlsanity.com/paper/1704.01938/) (2017)
- [Slim Embedding Layers for Recurrent Neural Language Models](https://mlsanity.com/paper/1711.09873/) (2017)
- [Rank and run-time aware compression of NLP Applications](https://mlsanity.com/paper/2010.03193/) (2020)

## Built on by (most cited first, 5 of 5)

- [Tensor Networks Meet Neural Networks: A Survey and Future Perspectives](https://mlsanity.com/paper/2302.09019/) (2023)
- [Quantum Natural Language Processing: A Comprehensive Review of Models, Methods, and Applications](https://mlsanity.com/paper/2504.09909/) (2025)
- [Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic](https://mlsanity.com/paper/2510.17001/) (2025)
- [Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability](https://mlsanity.com/paper/2608.30505/) (2026)
- [More Than Words: Compositional Tokenization for Efficient Language Models](https://mlsanity.com/paper/2610.05597/) (2026)

## Abstract (arXiv)

In the era of deep learning, word embeddings are essential when dealing with text tasks. However, storing and accessing these embeddings requires a large amount of space. This is not conducive to the deployment of these models on resource-limited devices. Combining the powerful compression capability of tensor products, we propose a word embedding compression method with morphological augmentation, Morphologically-enhanced Tensorized Embeddings (MorphTE). A word consists of one or more morphemes, the smallest units that bear meaning or have a grammatical function. MorphTE represents a word embedding as an entangled form of its morpheme vectors via the tensor product, which injects prior semantic and grammatical knowledge into the learning of embeddings. Furthermore, the dimensionality of the morpheme vector and the number of morphemes are much smaller than those of words, which greatly reduces the parameters of the word embeddings. We conduct experiments on tasks such as machine translation and question answering. Experimental results on four translation datasets of different languages show that MorphTE can compress word embedding parameters by about 20 times without performance loss and significantly outperforms related embedding compression methods.