# SEE: Sememe Entanglement Encoding for Transformer-bases Models Compression

arXiv: https://arxiv.org/abs/2412.12204 · HTML: https://mlsanity.com/paper/2412.12204/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 19 references that are in the map.
- First posted on arXiv 15 Dec 2024.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Lite Transformer with Long-Short Range Attention](https://mlsanity.com/paper/2004.11886/) (2020)
- [A Tensorized Transformer for Language Modeling](https://mlsanity.com/paper/1906.09777/) (2019)
- [OpenHowNet: An Open Sememe-based Lexical Knowledge Base](https://mlsanity.com/paper/1901.09957/) (2019)
- [The University of Edinburgh's Neural MT Systems for WMT17](https://mlsanity.com/paper/1708.00726/) (2017)
- [word2ket: Space-efficient Word Embeddings inspired by Quantum Entanglement](https://mlsanity.com/paper/1911.04975/) (2019)
- [Rank and run-time aware compression of NLP Applications](https://mlsanity.com/paper/2010.03193/) (2020)

## Abstract (arXiv)

Transformer-based large language models exhibit groundbreaking capabilities, but their storage and computational costs are prohibitively high, limiting their application in resource-constrained scenarios. An effective approach is to eliminate redundant model parameters and computational costs while incorporating efficient expert-derived knowledge structures to achieve a balance between compression and performance. Therefore, we propose the textitSememe Entanglement Encoding (SEE) algorithm. Guided by expert prior knowledge, the model is compressed through the low-rank approximation idea. In Entanglement Embedding, basic semantic units such as sememes are represented as low-dimensional vectors, and then reconstructed into high-dimensional word embeddings through the combination of generalized quantum entanglement. We adapt the Sememe Entanglement Encoding algorithm to transformer-based models of different magnitudes. Experimental results indicate that our approach achieves stable performance while compressing model parameters and computational costs.