mlsanityData to 7 Oct 2026

ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning

arXiv 2012.15022 ↗First posted 30 Dec 2020cs.CL cs.AIcited by 72

Not on the map. Why?

Abstract (arXiv)

Pre-trained Language Models (PLMs) have shown superior performance on various downstream Natural Language Processing (NLP) tasks. However, conventional pre-training objectives do not explicitly model relational facts in text, which are crucial for textual understanding. To address this issue, we propose a novel contrastive learning framework ERICA to obtain a deep understanding of the entities and their relations in text. Specifically, we define two novel pre-training tasks to better understand entities and relations: (1) the entity discrimination task to distinguish which tail entity can be inferred by the given head entity and relation; (2) the relation discrimination task to distinguish whether two relations are close or not semantically, which involves complex relational reasoning. Experimental results demonstrate that ERICA can improve typical PLMs (BERT and RoBERTa) on several language understanding tasks, including relation extraction, entity typing and question answering, especially under low-resource settings.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 7; 2022: 30; 2023: 21; 2024: 12; 2025: 2.

72 papers.

Sort:

By area:

  1. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  2. 2022Unified Structure Generation for Universal Information Extractioncited by 220
  3. 2022A Review on Language Models as Knowledge Basescited by 156
  4. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  5. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  6. 2021Knowledge Inheritance for Pre-trained Language Modelscited by 38
  7. 2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
  8. 2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
  9. 2021Relational World Knowledge Representation in Contextual Language Models: A Reviewcited by 31
  10. 2022Learning from the Dictionary: Heterogeneous Knowledge Guided Fine-tuning for Chinese Spell Checkingcited by 31
  11. 2022Contrastive Learning with Hard Negative Entities for Entity Set Expansioncited by 26
  12. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
38 more
  1. 2022MERIt: Meta-Path Guided Contrastive Learning for Logical Reasoningcited by 22
  2. 2022KALA: Knowledge-Augmented Language Model Adaptationcited by 21
  3. 2023Knowledge-Augmented Language Model Verificationcited by 21
  4. 2023YAYI-UIE: A Chat-Enhanced Instruction Tuning Framework for Universal Information Extractioncited by 19
  5. 2024InfuserKI: Enhancing Large Language Models with Knowledge Graphs via Infuser-Guided Knowledge Integrationcited by 19
  6. 2022Visually-Augmented Language Modelingcited by 18
  7. 2023SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectivescited by 15
  8. 2023Multi-level Contrastive Learning for Script-based Character Understandingcited by 15
  9. 2022DictBERT: Dictionary Description Knowledge Enhanced Language Model Pre-training via Contrastive Learningcited by 11
  10. 2022VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understandingcited by 11
  11. 2022A survey on knowledge-enhanced multimodal learningcited by 11
  12. 2024Learning Multi-view Molecular Representations with Structured and Unstructured Knowledgecited by 11
  13. 2021Structure Inducing Pre-Trainingcited by 10
  14. 2022An Effective Deployment of Contrastive Learning in Multi-label Text Classificationcited by 9
  15. 2022HiCLRE: A Hierarchical Contrastive Learning Framework for Distantly Supervised Relation Extractioncited by 8
  16. 2022A Multi-level Supervised Contrastive Learning Framework for Low-Resource Natural Language Inferencecited by 8
  17. 2023Exploring Self-supervised Logic-enhanced Training for Large Language Modelscited by 8
  18. 2022Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extractioncited by 7
  19. 2022Fine-grained Contrastive Learning for Relation Extractioncited by 7
  20. 2024Enhancing Idiomatic Representation in Multiple Languages via an Adaptive Contrastive Triplet Losscited by 7
  21. 2023Pre-training Language Models for Comparative Reasoningcited by 6
  22. 2023Plug-and-Play Knowledge Injection for Pre-trained Language Modelscited by 6
  23. 2022Instilling Type Knowledge in Language Models via Multi-Task QAcited by 5
  24. 2022Event-Centric Question Answering via Contrastive Learning and Invertible Event Transformationcited by 5
  25. 2022Query-based Instance Discrimination Network for Relational Triple Extractioncited by 5
  26. 2023Bridging the Language Gap: Knowledge Injected Multilingual Question Answeringcited by 5
  27. 2023Synergistic Anchored Contrastive Pre-training for Few-Shot Relation Extractioncited by 5
  28. 2025Rote Learning Considered Useful: Generalizing over Memorized Data in LLMscited by 5
  29. 2022Pre-training Language Models with Deterministic Factual Knowledgecited by 4
  30. 2023Unifying Structure Reasoning and Language Model Pre-training for Complex Reasoningcited by 4
  31. 2023Towards Integration of Discriminability and Robustness for Document-Level Relation Extractioncited by 4
  32. 2023Controlled Text Generation with Hidden Representation Transformationscited by 4
  33. 2023DistillCSE: Distilled Contrastive Learning for Sentence Embeddingscited by 4
  34. 2021Fine-Grained Chemical Entity Typing with Multimodal Knowledge Representationcited by 3
  35. 2021A Survey of Knowledge Enhanced Pre-trained Modelscited by 3
  36. 2022Improving Long Tailed Document-Level Relation Extraction via Easy Relation Augmentation and Contrastive Learningcited by 3
  37. 2022An Overview of Distant Supervision for Relation Extraction with a Focus on Denoising and Pre-training Methodscited by 3
  38. 2022A Survey on Knowledge-Enhanced Pre-trained Language Modelscited by 3

50 most cited of 72 All 72 (JSON)