# ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning

arXiv: https://arxiv.org/abs/2012.15022 · HTML: https://mlsanity.com/paper/2012.15022/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Dec 2020.

## Built on by (most cited first, 30 of 72)

- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Unified Structure Generation for Universal Information Extraction](https://mlsanity.com/paper/2203.12277/) (2022)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [A Survey of Knowledge Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2211.05994/) (2022)
- [Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling](https://mlsanity.com/paper/2306.11489/) (2023)
- [Knowledge Inheritance for Pre-trained Language Models](https://mlsanity.com/paper/2105.13880/) (2021)
- [Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey](https://mlsanity.com/paper/2110.08455/) (2021)
- [Graph Machine Learning in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2404.14928/) (2024)
- [Relational World Knowledge Representation in Contextual Language Models: A Review](https://mlsanity.com/paper/2104.05837/) (2021)
- [Learning from the Dictionary: Heterogeneous Knowledge Guided Fine-tuning for Chinese Spell Checking](https://mlsanity.com/paper/2210.10320/) (2022)
- [Contrastive Learning with Hard Negative Entities for Entity Set Expansion](https://mlsanity.com/paper/2204.07789/) (2022)
- [The Life Cycle of Knowledge in Big Language Models: A Survey](https://mlsanity.com/paper/2303.07616/) (2023)
- [MERIt: Meta-Path Guided Contrastive Learning for Logical Reasoning](https://mlsanity.com/paper/2203.00357/) (2022)
- [Knowledge-Augmented Language Model Verification](https://mlsanity.com/paper/2310.12836/) (2023)
- [KALA: Knowledge-Augmented Language Model Adaptation](https://mlsanity.com/paper/2204.10555/) (2022)
- [YAYI-UIE: A Chat-Enhanced Instruction Tuning Framework for Universal Information Extraction](https://mlsanity.com/paper/2312.15548/) (2023)
- [InfuserKI: Enhancing Large Language Models with Knowledge Graphs via Infuser-Guided Knowledge Integration](https://mlsanity.com/paper/2402.11441/) (2024)
- [Visually-Augmented Language Modeling](https://mlsanity.com/paper/2205.10178/) (2022)
- [SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives](https://mlsanity.com/paper/2305.13192/) (2023)
- [Multi-level Contrastive Learning for Script-based Character Understanding](https://mlsanity.com/paper/2310.13231/) (2023)
- [VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding](https://mlsanity.com/paper/2211.00430/) (2022)
- [DictBERT: Dictionary Description Knowledge Enhanced Language Model Pre-training via Contrastive Learning](https://mlsanity.com/paper/2208.00635/) (2022)
- [A survey on knowledge-enhanced multimodal learning](https://mlsanity.com/paper/2211.12328/) (2022)
- [Learning Multi-view Molecular Representations with Structured and Unstructured Knowledge](https://mlsanity.com/paper/2406.09841/) (2024)
- [Structure Inducing Pre-Training](https://mlsanity.com/paper/2103.10334/) (2021)
- [An Effective Deployment of Contrastive Learning in Multi-label Text Classification](https://mlsanity.com/paper/2212.00552/) (2022)
- [HiCLRE: A Hierarchical Contrastive Learning Framework for Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2202.13352/) (2022)
- [A Multi-level Supervised Contrastive Learning Framework for Low-Resource Natural Language Inference](https://mlsanity.com/paper/2205.15550/) (2022)
- [Exploring Self-supervised Logic-enhanced Training for Large Language Models](https://mlsanity.com/paper/2305.13718/) (2023)
- [Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction](https://mlsanity.com/paper/2205.03521/) (2022)

## Abstract (arXiv)

Pre-trained Language Models (PLMs) have shown superior performance on various downstream Natural Language Processing (NLP) tasks. However, conventional pre-training objectives do not explicitly model relational facts in text, which are crucial for textual understanding. To address this issue, we propose a novel contrastive learning framework ERICA to obtain a deep understanding of the entities and their relations in text. Specifically, we define two novel pre-training tasks to better understand entities and relations: (1) the entity discrimination task to distinguish which tail entity can be inferred by the given head entity and relation; (2) the relation discrimination task to distinguish whether two relations are close or not semantically, which involves complex relational reasoning. Experimental results demonstrate that ERICA can improve typical PLMs (BERT and RoBERTa) on several language understanding tasks, including relation extraction, entity typing and question answering, especially under low-resource settings.