# ZS-BERT: Towards Zero-Shot Relation Extraction with Attribute Representation Learning

arXiv: https://arxiv.org/abs/2104.04697 · HTML: https://mlsanity.com/paper/2104.04697/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 10 Apr 2021.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [Zero-Shot Learning by Convex Combination of Semantic Embeddings](https://mlsanity.com/paper/1312.5650/) (2013)
- [Semantic Autoencoder for Zero-Shot Learning](https://mlsanity.com/paper/1704.08345/) (2017)
- [FewRel 2.0: Towards More Challenging Few-Shot Relation Classification](https://mlsanity.com/paper/1910.07124/) (2019)
- [Enriching Pre-trained Language Model with Entity Information for Relation Classification](https://mlsanity.com/paper/1905.08284/) (2019)
- [Reasoning about Entailment with Neural Attention](https://mlsanity.com/paper/1509.06664/) (2015)
- [Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptions](https://mlsanity.com/paper/1506.00511/) (2015)
- [Question Answering on Freebase via Relation Extraction and Textual Evidence](https://mlsanity.com/paper/1603.00957/) (2016)
- [Zero-Shot Learning by Generating Pseudo Feature Representations](https://mlsanity.com/paper/1703.06389/) (2017)

## Built on by (most cited first, 30 of 46)

- [InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction](https://mlsanity.com/paper/2304.08085/) (2023)
- [Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extraction](https://mlsanity.com/paper/2109.03659/) (2021)
- [RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extraction](https://mlsanity.com/paper/2203.09101/) (2022)
- [Universal Information Extraction as Unified Semantic Matching](https://mlsanity.com/paper/2301.03282/) (2023)
- [Revisiting Large Language Models as Zero-shot Relation Extractors](https://mlsanity.com/paper/2310.05028/) (2023)
- [YAYI-UIE: A Chat-Enhanced Instruction Tuning Framework for Universal Information Extraction](https://mlsanity.com/paper/2312.15548/) (2023)
- [Unified Semantic Typing with Meaningful Label Inference](https://mlsanity.com/paper/2205.01826/) (2022)
- [SeqGPT: An Out-of-the-box Large Language Model for Open Domain Sequence Understanding](https://mlsanity.com/paper/2308.10529/) (2023)
- [UniRel: Unified Representation and Interaction for Joint Relational Triple Extraction](https://mlsanity.com/paper/2211.09039/) (2022)
- [Consistency Guided Knowledge Retrieval and Denoising in LLMs for Zero-shot Document-level Relation Triplet Extraction](https://mlsanity.com/paper/2401.13598/) (2024)
- [RE-Matching: A Fine-Grained Semantic Matching Method for Zero-Shot Relation Extraction](https://mlsanity.com/paper/2306.04954/) (2023)
- [Meta In-Context Learning Makes Large Language Models Better Zero and Few-Shot Relation Extractors](https://mlsanity.com/paper/2404.17807/) (2024)
- [Mirror: A Universal Framework for Various Information Extraction Tasks](https://mlsanity.com/paper/2311.05419/) (2023)
- [GLiREL -- Generalist Model for Zero-Shot Relation Extraction](https://mlsanity.com/paper/2501.03172/) (2025)
- [Information Extraction in Low-Resource Scenarios: Survey and Perspective](https://mlsanity.com/paper/2202.08063/) (2022)
- [Unleashing the Power of Large Language Models in Zero-shot Relation Extraction via Self-Prompting](https://mlsanity.com/paper/2410.01154/) (2024)
- [InstructIE: A Bilingual Instruction-based Information Extraction Dataset](https://mlsanity.com/paper/2305.11527/) (2023)
- [CreoleVal: Multilingual Multitask Benchmarks for Creoles](https://mlsanity.com/paper/2310.19567/) (2023)
- [Grasping the Essentials: Tailoring Large Language Models for Zero-Shot Relation Extraction](https://mlsanity.com/paper/2402.11142/) (2024)
- [Prompt-based Zero-shot Relation Extraction with Semantic Knowledge Augmentation](https://mlsanity.com/paper/2112.04539/) (2021)
- [Pre-training to Match for Unified Low-shot Relation Extraction](https://mlsanity.com/paper/2203.12274/) (2022)
- [Zero-shot Triplet Extraction by Template Infilling](https://mlsanity.com/paper/2212.10708/) (2022)
- [UniEX: An Effective and Efficient Framework for Unified Information Extraction via a Span-extractive Perspective](https://mlsanity.com/paper/2305.10306/) (2023)
- [Benchmarking Knowledge-driven Zero-shot Learning](https://mlsanity.com/paper/2106.15047/) (2021)
- [Synergistic Anchored Contrastive Pre-training for Few-Shot Relation Extraction](https://mlsanity.com/paper/2312.12021/) (2023)
- [IEPile: Unearthing Large-Scale Schema-Based Information Extraction Corpus](https://mlsanity.com/paper/2402.14710/) (2024)
- [Linguistic representations for fewer-shot relation extraction across domains](https://mlsanity.com/paper/2307.03823/) (2023)
- [A Condensed Transition Graph Framework for Zero-shot Link Prediction with Large Language Models](https://mlsanity.com/paper/2402.10779/) (2024)
- [RUIE: Retrieval-based Unified Information Extraction using Large Language Model](https://mlsanity.com/paper/2409.11673/) (2024)
- [KnowCoder-X: Boosting Multilingual Information Extraction via Code](https://mlsanity.com/paper/2411.04794/) (2024)

## Abstract (arXiv)

While relation extraction is an essential task in knowledge acquisition and representation, and new-generated relations are common in the real world, less effort is made to predict unseen relations that cannot be observed at the training stage. In this paper, we formulate the zero-shot relation extraction problem by incorporating the text description of seen and unseen relations. We propose a novel multi-task learning model, zero-shot BERT (ZS-BERT), to directly predict unseen relations without hand-crafted attribute labeling and multiple pairwise classifications. Given training instances consisting of input sentences and the descriptions of their relations, ZS-BERT learns two functions that project sentences and relation descriptions into an embedding space by jointly minimizing the distances between them and classifying seen relations. By generating the embeddings of unseen relations and new-coming sentences based on such two functions, we use nearest neighbor search to obtain the prediction of unseen relations. Experiments conducted on two well-known datasets exhibit that ZS-BERT can outperform existing methods by at least 13.54% improvement on F1 score.