# Keywords and Instances: A Hierarchical Contrastive Learning Framework Unifying Hybrid Granularities for Text Generation

arXiv: https://arxiv.org/abs/2205.13346 · HTML: https://mlsanity.com/paper/2205.13346/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 26 May 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders](https://mlsanity.com/paper/1703.10960/) (2017)
- [A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection](https://mlsanity.com/paper/2106.09022/) (2021)
- [A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues](https://mlsanity.com/paper/1605.06069/) (2016)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [Generalization in Generation: A closer look at Exposure Bias](https://mlsanity.com/paper/1910.00292/) (2019)
- [Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detection](https://mlsanity.com/paper/2101.03778/) (2021)
- [Contrastive Learning with Adversarial Perturbations for Conditional Text Generation](https://mlsanity.com/paper/2012.07280/) (2020)
- [DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder](https://mlsanity.com/paper/1805.12352/) (2018)
- [A Sequence-to-Sequence Approach to Dialogue State Tracking](https://mlsanity.com/paper/2011.09553/) (2020)
- [A Hierarchical Neural Autoencoder for Paragraphs and Documents](https://mlsanity.com/paper/1506.01057/) (2015)
- [Guiding Variational Response Generator to Exploit Persona](https://mlsanity.com/paper/1911.02390/) (2019)
- [Group-wise Contrastive Learning for Neural Dialogue Generation](https://mlsanity.com/paper/2009.07543/) (2020)
- [The Style-Content Duality of Attractiveness: Learning to Write Eye-Catching Headlines via Disentanglement](https://mlsanity.com/paper/2012.07419/) (2020)
- [Adversarial Mahalanobis Distance-based Attentive Song Recommender for Automatic Playlist Continuation](https://mlsanity.com/paper/1906.03450/) (2019)

## Built on by (most cited first, 7 of 7)

- [HiCo: Hierarchical Contrastive Learning for Ultrasound Video Model Pretraining](https://mlsanity.com/paper/2210.04477/) (2022)
- [EZInterviewer: To Improve Job Interview Performance with Mock Interview Generator](https://mlsanity.com/paper/2301.00972/) (2023)
- [Multi-Intent Attribute-Aware Text Matching in Searching](https://mlsanity.com/paper/2402.07788/) (2024)
- [Entropy Guided Extrapolative Decoding to Improve Factuality in Large Language Models](https://mlsanity.com/paper/2404.09338/) (2024)
- [Optimizing Non-Autoregressive Transformers with Contrastive Learning](https://mlsanity.com/paper/2305.13667/) (2023)
- [A Note on Bias to Complete](https://mlsanity.com/paper/2402.11710/) (2024)
- [Contrastive Learning for Inference in Dialogue](https://mlsanity.com/paper/2310.12467/) (2023)

## Abstract (arXiv)

Contrastive learning has achieved impressive success in generation tasks to militate the "exposure bias" problem and discriminatively exploit the different quality of references. Existing works mostly focus on contrastive learning on the instance-level without discriminating the contribution of each word, while keywords are the gist of the text and dominant the constrained mapping relationships. Hence, in this work, we propose a hierarchical contrastive learning mechanism, which can unify hybrid granularities semantic meaning in the input text. Concretely, we first propose a keyword graph via contrastive correlations of positive-negative pairs to iteratively polish the keyword representations. Then, we construct intra-contrasts within instance-level and keyword-level, where we assume words are sampled nodes from a sentence distribution. Finally, to bridge the gap between independent contrast levels and tackle the common contrast vanishing problem, we propose an inter-contrast mechanism that measures the discrepancy between contrastive keyword nodes respectively to the instance distribution. Experiments demonstrate that our model outperforms competitive baselines on paraphrasing, dialogue generation, and storytelling tasks.