# Improving Relation Extraction by Pre-trained Language Representations

arXiv: https://arxiv.org/abs/1906.03088 · HTML: https://mlsanity.com/paper/1906.03088/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2019.

## Built on by (most cited first, 24 of 24)

- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [Re-TACRED: Addressing Shortcomings of the TACRED Dataset](https://mlsanity.com/paper/2104.08398/) (2021)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [A Survey on Extraction of Causal Relations from Natural Language Text](https://mlsanity.com/paper/2101.06426/) (2021)
- [Discovering Drug-Target Interaction Knowledge from Biomedical Literature](https://mlsanity.com/paper/2109.13187/) (2021)
- [NEREL: A Russian Dataset with Nested Named Entities, Relations and Events](https://mlsanity.com/paper/2108.13112/) (2021)
- [On the Evolution of Knowledge Graphs: A Survey and Perspective](https://mlsanity.com/paper/2310.04835/) (2023)
- [How Good are LLMs at Relation Extraction under Low-Resource Scenario? Comprehensive Evaluation](https://mlsanity.com/paper/2406.11162/) (2024)
- [Less is More: Rethinking State-of-the-art Continual Relation Extraction Models with a Frustratingly Easy but Effective Approach](https://mlsanity.com/paper/2209.00243/) (2022)
- [Confidence estimation of classification based on the distribution of the neural network output layer](https://mlsanity.com/paper/2210.07745/) (2022)
- [A Generative Model for Relation Extraction and Classification](https://mlsanity.com/paper/2202.13229/) (2022)
- [Automatic Knowledge Graph Construction for Judicial Cases](https://mlsanity.com/paper/2404.09416/) (2024)
- [RuSentNE-2023: Evaluating Entity-Oriented Sentiment Analysis on Russian News Texts](https://mlsanity.com/paper/2305.17679/) (2023)
- [Guided Distant Supervision for Multilingual Relation Extraction Data: Adapting to a New Language](https://mlsanity.com/paper/2403.17143/) (2024)
- [MobIE: A German Dataset for Named Entity Recognition, Entity Linking and Relation Extraction in the Mobility Domain](https://mlsanity.com/paper/2108.06955/) (2021)
- [KnowMAN: Weakly Supervised Multinomial Adversarial Networks](https://mlsanity.com/paper/2109.07994/) (2021)
- [Enriched Attention for Robust Relation Extraction](https://mlsanity.com/paper/2104.10899/) (2021)
- [Modeling Multi-Granularity Hierarchical Features for Relation Extraction](https://mlsanity.com/paper/2204.04437/) (2022)
- [Augmenting Task-Oriented Dialogue Systems with Relation Extraction](https://mlsanity.com/paper/2210.13344/) (2022)
- [Graph-Augmented Relation Extraction Model with LLMs-Generated Support Document](https://mlsanity.com/paper/2410.23452/) (2024)
- [Budget Sensitive Reannotation of Noisy Relation Classification Data Using Label Hierarchy](https://mlsanity.com/paper/2112.13320/) (2021)
- [FREDA: Flexible Relation Extraction Data Annotation](https://mlsanity.com/paper/2204.07150/) (2022)
- [Noise in Relation Classification Dataset TACRED: Characterization and Reduction](https://mlsanity.com/paper/2311.12298/) (2023)
- [Multi-layer Sequence Labeling-based Joint Biomedical Event Extraction](https://mlsanity.com/paper/2408.05545/) (2024)

## Abstract (arXiv)

Current state-of-the-art relation extraction methods typically rely on a set of lexical, syntactic, and semantic features, explicitly computed in a pre-processing step. Training feature extraction models requires additional annotated language resources, which severely restricts the applicability and portability of relation extraction to novel languages. Similarly, pre-processing introduces an additional source of error. To address these limitations, we introduce TRE, a Transformer for Relation Extraction, extending the OpenAI Generative Pre-trained Transformer [Radford et al., 2018]. Unlike previous relation extraction models, TRE uses pre-trained deep language representations instead of explicit linguistic features to inform the relation classification and combines it with the self-attentive Transformer architecture to effectively model long-range dependencies between entity mentions. TRE allows us to learn implicit linguistic features solely from plain text corpora by unsupervised pre-training, before fine-tuning the learned language representations on the relation extraction task. TRE obtains a new state-of-the-art result on the TACRED and SemEval 2010 Task 8 datasets, achieving a test F1 of 67.4 and 87.1, respectively. Furthermore, we observe a significant increase in sample efficiency. With only 20% of the training examples, TRE matches the performance of our baselines and our model trained from scratch on 100% of the TACRED dataset. We open-source our trained models, experiments, and source code.