# SememeLM: A Sememe Knowledge Enhanced Method for Long-tail Relation Representation

arXiv: https://arxiv.org/abs/2406.10297 · HTML: https://mlsanity.com/paper/2406.10297/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Composition-based Multi-Relational Graph Convolutional Networks](https://mlsanity.com/paper/1911.03082/) (2019)
- [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [Inducing Relational Knowledge from BERT](https://mlsanity.com/paper/1911.12753/) (2019)
- [BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?](https://mlsanity.com/paper/2105.04949/) (2021)
- [E-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoning](https://mlsanity.com/paper/2203.08480/) (2022)
- [LET: Linguistic Knowledge Enhanced Graph Transformer for Chinese Short Text Matching](https://mlsanity.com/paper/2102.12671/) (2021)
- [Distilling Relation Embeddings from Pre-trained Language Models](https://mlsanity.com/paper/2110.15705/) (2021)
- [Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples](https://mlsanity.com/paper/2111.10962/) (2021)
- [Distantly-Supervised Long-Tailed Relation Extraction Using Constraint Graphs](https://mlsanity.com/paper/2105.11225/) (2021)
- [Learning Relation Prototype from Unlabeled Texts for Long-tail Relation Extraction](https://mlsanity.com/paper/2011.13574/) (2020)
- [Rescue Implicit and Long-tail Cases: Nearest Neighbor Relation Extraction](https://mlsanity.com/paper/2210.11800/) (2022)
- [Towards Building a Multilingual Sememe Knowledge Base: Predicting Sememes for BabelNet Synsets](https://mlsanity.com/paper/1912.01795/) (2019)
- [KEML: A Knowledge-Enriched Meta-Learning Framework for Lexical Relation Classification](https://mlsanity.com/paper/2002.10903/) (2020)
- [Measuring the Feasibility of Analogical Transfer using Complexity](https://mlsanity.com/paper/2206.11753/) (2022)

## Abstract (arXiv)

Recognizing relations between two words is a fundamental task with the broad applications. Different from extracting relations from text, it is difficult to identify relations among words without their contexts. Especially for long-tail relations, it becomes more difficult due to inadequate semantic features. Existing approaches based on language models (LMs) utilize rich knowledge of LMs to enhance the semantic features of relations. However, they capture uncommon relations while overlooking less frequent but meaningful ones since knowledge of LMs seriously relies on trained data where often represents common relations. On the other hand, long-tail relations are often uncommon in training data. It is interesting but not trivial to use external knowledge to enrich LMs due to collecting corpus containing long-tail relationships is hardly feasible. In this paper, we propose a sememe knowledge enhanced method (SememeLM) to enhance the representation of long-tail relations, in which sememes can break the contextual constraints between wors. Firstly, we present a sememe relation graph and propose a graph encoding method. Moreover, since external knowledge base possibly consisting of massive irrelevant knowledge, the noise is introduced. We propose a consistency alignment module, which aligns the introduced knowledge with LMs, reduces the noise and integrates the knowledge into the language model. Finally, we conducted experiments on word analogy datasets, which evaluates the ability to distinguish relation representations subtle differences, including long-tail relations. Extensive experiments show that our approach outperforms some state-of-the-art methods.