# PromptCL: Improving Event Representation via Prompt Template and Contrastive Learning

arXiv: https://arxiv.org/abs/2404.17877 · HTML: https://mlsanity.com/paper/2404.17877/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 27 Apr 2024.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Should You Mask 15% in Masked Language Modeling?](https://mlsanity.com/paper/2202.08005/) (2022)
- [OntoED: Low-resource Event Detection with Ontology Embedding](https://mlsanity.com/paper/2105.10922/) (2021)
- [Event Representation Learning Enhanced with External Commonsense Knowledge](https://mlsanity.com/paper/1909.05190/) (2019)
- [GraphPlan: Story Generation by Planning with Event Graph](https://mlsanity.com/paper/2102.02977/) (2021)
- [Event Representations with Tensor-based Compositions](https://mlsanity.com/paper/1711.07611/) (2017)
- [Improving Event Representation via Simultaneous Weakly Supervised Contrastive Learning and Clustering](https://mlsanity.com/paper/2203.07633/) (2022)
- [A Study of Entanglement in a Categorical Framework of Natural Language](https://mlsanity.com/paper/1405.2874/) (2014)
- [Event Representation with Sequential, Semi-Supervised Discrete Variables](https://mlsanity.com/paper/2010.04361/) (2020)

## Abstract (arXiv)

The representation of events in text plays a significant role in various NLP tasks. Recent research demonstrates that contrastive learning has the ability to improve event comprehension capabilities of Pre-trained Language Models (PLMs) and enhance the performance of event representation learning. However, the efficacy of event representation learning based on contrastive learning and PLMs is limited by the short length of event texts. The length of event texts differs significantly from the text length used in the pre-training of PLMs. As a result, there is inconsistency in the distribution of text length between pre-training and event representation learning, which may undermine the learning process of event representation based on PLMs. In this study, we present PromptCL, a novel framework for event representation learning that effectively elicits the capabilities of PLMs to comprehensively capture the semantics of short event texts. PromptCL utilizes a Prompt template borrowed from prompt learning to expand the input text during Contrastive Learning. This helps in enhancing the event representation learning by providing a structured outline of the event components. Moreover, we propose Subject-Predicate-Object (SPO) word order and Event-oriented Masked Language Modeling (EventMLM) to train PLMs to understand the relationships between event components. Our experimental results demonstrate that PromptCL outperforms state-of-the-art baselines on event related tasks. Additionally, we conduct a thorough analysis and demonstrate that using a prompt results in improved generalization capabilities for event representations. Our code will be available at https://github.com/YuboFeng2023/PromptCL.