# Language Modeling with Sparse Product of Sememe Experts

arXiv: https://arxiv.org/abs/1810.12387 · HTML: https://mlsanity.com/paper/1810.12387/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2018.

## Built on by (most cited first, 6 of 6)

- [LET: Linguistic Knowledge Enhanced Graph Transformer for Chinese Short Text Matching](https://mlsanity.com/paper/2102.12671/) (2021)
- [TKDP: Threefold Knowledge-enriched Deep Prompt Tuning for Few-shot Named Entity Recognition](https://mlsanity.com/paper/2306.03974/) (2023)
- [Automatic Construction of Sememe Knowledge Bases via Dictionaries](https://mlsanity.com/paper/2105.12585/) (2021)
- [Sememe Prediction for BabelNet Synsets using Multilingual and Multimodal Information](https://mlsanity.com/paper/2203.07426/) (2022)
- [The Analysis about Building Cross-lingual Sememe Knowledge Base Based on Deep Clustering Network](https://mlsanity.com/paper/2208.05462/) (2022)
- [SememeASR: Boosting Performance of End-to-End Speech Recognition against Domain and Long-Tailed Data Shift with Sememe Semantic Knowledge](https://mlsanity.com/paper/2309.01437/) (2023)

## Abstract (arXiv)

Most language modeling methods rely on large-scale data to statistically learn the sequential patterns of words. In this paper, we argue that words are atomic language units but not necessarily atomic semantic units. Inspired by HowNet, we use sememes, the minimum semantic units in human languages, to represent the implicit semantics behind words for language modeling, named Sememe-Driven Language Model (SDLM). More specifically, to predict the next word, SDLM first estimates the sememe distribution gave textual context. Afterward, it regards each sememe as a distinct semantic expert, and these experts jointly identify the most probable senses and the corresponding word. In this way, SDLM enables language models to work beyond word-level manipulation to fine-grained sememe-level semantics and offers us more powerful tools to fine-tune language models and improve the interpretability as well as the robustness of language models. Experiments on language modeling and the downstream application of headline gener- ation demonstrate the significant effect of SDLM. Source code and data used in the experiments can be accessed at https:// github.com/thunlp/SDLM-pytorch.