# CLEVE: Contrastive Pre-training for Event Extraction

arXiv: https://arxiv.org/abs/2105.14485 · HTML: https://mlsanity.com/paper/2105.14485/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 82 references that are in the map.
- First posted on arXiv 30 May 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Deep Graph Contrastive Representation Learning](https://mlsanity.com/paper/2006.04131/) (2020)
- [GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training](https://mlsanity.com/paper/2006.09963/) (2020)
- [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019)
- [Entity, Relation, and Event Extraction with Contextualized Span Representations](https://mlsanity.com/paper/1909.03546/) (2019)
- [MAVEN: A Massive General Domain Event Detection Dataset](https://mlsanity.com/paper/2004.13590/) (2020)
- [A Mutual Information Maximization Perspective of Language Representation Learning](https://mlsanity.com/paper/1910.08350/) (2019)
- [Learning from Context or Names? An Empirical Study on Neural Relation Extraction](https://mlsanity.com/paper/2010.01923/) (2020)
- [Zero-Shot Transfer Learning for Event Extraction](https://mlsanity.com/paper/1707.01066/) (2017)
- [Conditional Negative Sampling for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2010.02037/) (2020)
- [Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models](https://mlsanity.com/paper/2005.10389/) (2020)
- [Train No Evil: Selective Masking for Task-Guided Pre-Training](https://mlsanity.com/paper/2004.09733/) (2020)
- [Improving AMR Parsing with Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2010.01771/) (2020)
- [Event Detection: Gate Diversity and Syntactic Importance Scoresfor Graph Convolution Neural Networks](https://mlsanity.com/paper/2010.14123/) (2020)
- [Graph Transformer Networks with Syntactic and Semantic Structures for Event Argument Extraction](https://mlsanity.com/paper/2010.13391/) (2020)
- [Joint Learning Templates and Slots for Event Schema Induction](https://mlsanity.com/paper/1603.01333/) (2016)
- [QAInfomax: Learning Robust Question Answering System by Mutual Information Maximization](https://mlsanity.com/paper/1909.00215/) (2019)
- [Unsupervised Natural Language Inference via Decoupled Multimodal Contrastive Learning](https://mlsanity.com/paper/2010.08200/) (2020)

## Built on by (most cited first, 30 of 36)

- [Unified Structure Generation for Universal Information Extraction](https://mlsanity.com/paper/2203.12277/) (2022)
- [KoLA: Carefully Benchmarking World Knowledge of Large Language Models](https://mlsanity.com/paper/2306.09296/) (2023)
- [Prompt for Extraction? PAIE: Prompting Argument Interaction for Event Argument Extraction](https://mlsanity.com/paper/2202.12109/) (2022)
- [ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification](https://mlsanity.com/paper/2203.02225/) (2022)
- [LasUIE: Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Model](https://mlsanity.com/paper/2304.06248/) (2023)
- [EventBERT: A Pre-Trained Model for Event Correlation Reasoning](https://mlsanity.com/paper/2110.06533/) (2021)
- [When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks](https://mlsanity.com/paper/2311.08993/) (2023)
- [LEVEN: A Large-Scale Chinese Legal Event Detection Dataset](https://mlsanity.com/paper/2203.08556/) (2022)
- [TextEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction](https://mlsanity.com/paper/2311.09562/) (2023)
- [Pushing the Limits of ChatGPT on NLP Tasks](https://mlsanity.com/paper/2306.09719/) (2023)
- [Query and Extract: Refining Event Extraction as Type-oriented Binary Decoding](https://mlsanity.com/paper/2110.07476/) (2021)
- [The Devil is in the Details: On the Pitfalls of Event Extraction Evaluation](https://mlsanity.com/paper/2306.06918/) (2023)
- [YAYI-UIE: A Chat-Enhanced Instruction Tuning Framework for Universal Information Extraction](https://mlsanity.com/paper/2312.15548/) (2023)
- [Unified Semantic Typing with Meaningful Label Inference](https://mlsanity.com/paper/2205.01826/) (2022)
- [A Dataset for Hyper-Relational Extraction and a Cube-Filling Approach](https://mlsanity.com/paper/2211.10018/) (2022)
- [A Survey on Deep Learning Event Extraction: Approaches and Applications](https://mlsanity.com/paper/2107.02126/) (2021)
- [Document-level Causal Relation Extraction with Knowledge-guided Binary Question Answering](https://mlsanity.com/paper/2410.04752/) (2024)
- [Open-Vocabulary Argument Role Prediction for Event Extraction](https://mlsanity.com/paper/2211.01577/) (2022)
- [Training Multimedia Event Extraction With Generated Images and Captions](https://mlsanity.com/paper/2306.08966/) (2023)
- [OmniEvent: A Comprehensive, Fair, and Easy-to-Use Toolkit for Event Understanding](https://mlsanity.com/paper/2309.14258/) (2023)
- [Information Extraction in Low-Resource Scenarios: Survey and Perspective](https://mlsanity.com/paper/2202.08063/) (2022)
- [Enhancing Document-level Event Argument Extraction with Contextual Clues and Role Relevance](https://mlsanity.com/paper/2310.05991/) (2023)
- [The Art of Prompting: Event Detection based on Type Specific Prompts](https://mlsanity.com/paper/2204.07241/) (2022)
- [An AMR-based Link Prediction Approach for Document-level Event Argument Extraction](https://mlsanity.com/paper/2305.19162/) (2023)
- [MAVEN-Arg: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation](https://mlsanity.com/paper/2311.09105/) (2023)
- [CMNEE: A Large-Scale Document-Level Event Extraction Dataset based on Open-Source Chinese Military News](https://mlsanity.com/paper/2404.12242/) (2024)
- [Continual Event Extraction with Semantic Confusion Rectification](https://mlsanity.com/paper/2310.15470/) (2023)
- [Event Extraction in Large Language Model](https://mlsanity.com/paper/2512.19537/) (2025)
- [Improve Event Extraction via Self-Training with Gradient Guidance](https://mlsanity.com/paper/2205.12490/) (2022)
- [HCL-TAT: A Hybrid Contrastive Learning Method for Few-shot Event Detection with Task-Adaptive Threshold](https://mlsanity.com/paper/2210.08806/) (2022)

## Abstract (arXiv)

Event extraction (EE) has considerably benefited from pre-trained language models (PLMs) by fine-tuning. However, existing pre-training methods have not involved modeling event characteristics, resulting in the developed EE models cannot take full advantage of large-scale unsupervised data. To this end, we propose CLEVE, a contrastive pre-training framework for EE to better learn event knowledge from large unsupervised data and their semantic structures (e.g. AMR) obtained with automatic parsers. CLEVE contains a text encoder to learn event semantics and a graph encoder to learn event structures respectively. Specifically, the text encoder learns event semantic representations by self-supervised contrastive learning to represent the words of the same events closer than those unrelated words; the graph encoder learns event structure representations by graph contrastive pre-training on parsed event-related semantic structures. The two complementary representations then work together to improve both the conventional supervised EE and the unsupervised "liberal" EE, which requires jointly extracting events and discovering event schemata without any annotated data. Experiments on ACE 2005 and MAVEN datasets show that CLEVE achieves significant improvements, especially in the challenging unsupervised setting. The source code and pre-trained checkpoints can be obtained from https://github.com/THU-KEG/CLEVE.