# ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification

arXiv: https://arxiv.org/abs/2203.02225 · HTML: https://mlsanity.com/paper/2203.02225/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 74 references that are in the map.
- First posted on arXiv 4 Mar 2022.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Abductive Commonsense Reasoning](https://mlsanity.com/paper/1908.05739/) (2019)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs](https://mlsanity.com/paper/2010.05953/) (2020)
- [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018)
- [Strategies for Structuring Story Generation](https://mlsanity.com/paper/1902.01109/) (2019)
- [A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation](https://mlsanity.com/paper/2001.05139/) (2020)
- [UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark](https://mlsanity.com/paper/2103.13009/) (2021)
- [ASER: A Large-scale Eventuality Knowledge Graph](https://mlsanity.com/paper/1905.00270/) (2019)
- [Counterfactual Story Reasoning and Generation](https://mlsanity.com/paper/1909.04076/) (2019)
- [Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning](https://mlsanity.com/paper/2010.05906/) (2020)

## Built on by (most cited first, 30 of 45)

- [Large Language Models for Generative Information Extraction: A Survey](https://mlsanity.com/paper/2312.17617/) (2023)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation](https://mlsanity.com/paper/2305.07375/) (2023)
- [Generative Knowledge Graph Construction: A Review](https://mlsanity.com/paper/2210.12714/) (2022)
- [Multimodal Event Transformer for Image-guided Story Ending Generation](https://mlsanity.com/paper/2301.11357/) (2023)
- [Improving Cross-modal Alignment for Text-Guided Image Inpainting](https://mlsanity.com/paper/2301.11362/) (2023)
- [Style-Aware Contrastive Learning for Multi-Style Image Captioning](https://mlsanity.com/paper/2301.11367/) (2023)
- [Rumor Detection with a novel graph neural network approach](https://mlsanity.com/paper/2403.16206/) (2024)
- [Image Captioning in news report scenario](https://mlsanity.com/paper/2403.16209/) (2024)
- [SubeventWriter: Iterative Sub-event Sequence Generation with Coherence Controller](https://mlsanity.com/paper/2210.06694/) (2022)
- [A Generative Approach for Script Event Prediction via Contrastive Fine-tuning](https://mlsanity.com/paper/2212.03496/) (2022)
- [Optimizing Search Advertising Strategies: Integrating Reinforcement Learning with Generalized Second-Price Auctions for Enhanced Ad Ranking and Bidding](https://mlsanity.com/paper/2405.13381/) (2024)
- [AgentRE: An Agent-Based Framework for Navigating Complex Information Landscapes in Relation Extraction](https://mlsanity.com/paper/2409.01854/) (2024)
- [Educational Personalized Learning Path Planning with Large Language Models](https://mlsanity.com/paper/2407.11773/) (2024)
- [Topic-Selective Graph Network for Topic-Focused Summarization](https://mlsanity.com/paper/2302.13106/) (2023)
- [Unsupervised Knowledge Graph Construction and Event-centric Knowledge Infusion for Scientific NLI](https://mlsanity.com/paper/2210.15248/) (2022)
- [CORECODE: A Common Sense Annotated Dialogue Dataset with Benchmark Tasks for Chinese Large Language Models](https://mlsanity.com/paper/2312.12853/) (2023)
- [End-to-End Chart Summarization via Visual Chain-of-Thought in Vision-Language Models](https://mlsanity.com/paper/2502.17589/) (2025)
- [Interpretable and Robust Dialogue State Tracking via Natural Language Summarization with LLMs](https://mlsanity.com/paper/2503.08857/) (2025)
- [Harnessing RLHF for Robust Unanswerability Recognition and Trustworthy Response Generation in LLMs](https://mlsanity.com/paper/2507.16951/) (2025)
- [Optimizing Psychological Counseling with Instruction-Tuned Large Language Models](https://mlsanity.com/paper/2406.13617/) (2024)
- [Improving Visual Storytelling with Multimodal Large Language Models](https://mlsanity.com/paper/2407.02586/) (2024)
- [Large Vision-Language Models for Remote Sensing Visual Question Answering](https://mlsanity.com/paper/2411.10857/) (2024)
- [Enhancing Scientific Figure Captioning Through Cross-modal Learning](https://mlsanity.com/paper/2406.17047/) (2024)
- [Fault Diagnosis in Power Grids with Large Language Model](https://mlsanity.com/paper/2407.08836/) (2024)
- [Enhancing Agricultural Machinery Management through Advanced LLM Integration](https://mlsanity.com/paper/2407.20588/) (2024)
- [Leveraging Language Models for Emotion and Behavior Analysis in Education](https://mlsanity.com/paper/2408.06874/) (2024)
- [Coal Mining Question Answering with LLMs](https://mlsanity.com/paper/2410.02959/) (2024)
- [CIMR: Contextualized Iterative Multimodal Reasoning for Robust Instruction Following in LVLMs](https://mlsanity.com/paper/2507.22074/) (2025)
- [CAP-LLM: Context-Augmented Personalized Large Language Models for News Headline Generation](https://mlsanity.com/paper/2508.03935/) (2025)

## Abstract (arXiv)

Generating new events given context with correlated ones plays a crucial role in many event-centric reasoning tasks. Existing works either limit their scope to specific scenarios or overlook event-level correlations. In this paper, we propose to pre-train a general Correlation-aware context-to-Event Transformer (ClarET) for event-centric reasoning. To achieve this, we propose three novel event-centric objectives, i.e., whole event recovering, contrastive event-correlation encoding and prompt-based event locating, which highlight event-level correlations with effective training. The proposed ClarET is applicable to a wide range of event-centric reasoning scenarios, considering its versatility of (i) event-correlation types (e.g., causal, temporal, contrast), (ii) application formulations (i.e., generation and classification), and (iii) reasoning types (e.g., abductive, counterfactual and ending reasoning). Empirical fine-tuning results, as well as zero- and few-shot learning, on 9 benchmarks (5 generation and 4 classification tasks covering 4 reasoning types with diverse event correlations), verify its effectiveness and generalization ability.